Fase 8 (1/N): delete the worker-side remote sync protocol
Deletes frontend.worker.sync/* wholesale (apply_txs, asset_db_listener,
assets, auth, client_op, const, crypt, download, handle_message,
large_title, log_and_state, presence, temp_sqlite, transport, upload,
util), worker/sync.cljs, worker/handler/sync.cljs, and
worker/ui_request.cljs -- the remote CRDT sync protocol and its E2EE
password-request plumbing, both made obsolete by the server-client
daemon architecture built in Fases 1-7. Deviates from the plan's KEEP
list on sync/presence.cljs and sync/transport.cljs: neither has any
caller left once the rest of the cluster is gone (confirmed by grep),
because Fase 4's WS resume-by-revision work already built its own
self-contained module (db_worker_node_ws.cljs) instead of adopting
these as the plan assumed it would.
This touched far more than a mechanical deletion:
- frontend.worker.sync.client-op wasn't sync-only. Its client_ops
SQLite table is also the idempotency marker for db_core.cljs's
built-in-property repair migration and the forward/inverse-outliner-
ops relay undo/redo reads back through history-action-ops-by-tx-id.
Trimmed (not deleted) into frontend.worker.local-tx-log, keeping only
the tx-entry/local-tx functions those two callers need and dropping
the genuinely sync-only surface (pending-upload counts, checksums,
graph-uuid, sync-conflict tracking, asset-op queues).
- frontend.worker.sync.apply-txs's apply-history-action! looked
sync-specific (it lived next to the remote-tx-apply/rebase machinery)
but turned out to be the *only* implementation ever registered for
worker-undo-redo/*apply-history-action! -- undo_redo.cljs calls it on
every db-transact undo/redo, local edits included, not just
remote-origin ones. Deleting it outright would have silently broken
undo/redo app-wide the next time anyone hit Cmd-Z. Ported the
general-purpose slice (replaying a recorded entry's forward/inverse
outliner ops against the current db state, plus the raw-tx-data
fallback for edits that bypass the structured outliner-op layer) into
a new frontend.worker.undo-redo.replay, along with persist-local-tx!
(derive history ops on every local commit, stash them in
local-tx-log, hand off to gen-undo-ops!) -- dropping only the
remote-rebase-specific rest (CRDT conflict resolution, upload-queue
bookkeeping). Wired into db_listener.cljs as a renamed
:persist-local-tx handler (was :db-sync, a name that stopped meaning
anything once sync was gone).
Caught by running the full existing undo_redo_test.cljs suite (30
deftests written for the old mechanism) against the new module after
fixing its test harness to use the real db_listener.cljs dispatch
instead of a hand-rolled listener -- all 29 tests port over passing
with no behavior change, which is the actual evidence this preserves
undo/redo correctness rather than just compiling.
- db_core.cljs's own 8 :thread-api/db-sync-* handlers (remote-graph-
download import, close-db/invalidate-search-db/recreate-lock
sync-specific variants) are deleted along with their sync-crypt/
sync-download calls; :thread-api/unsafe-unlink-db, :close-db, and
:release-access-handles keep their real (non-sync) bodies.
db_worker_node.cljs's non-repo-methods set drops from 22 to 7
entries (the 15 removed were all e2ee/db-sync/ui-request keywords
backed by deleted files).
- worker/handler/export.cljs's :thread-api/recompute-checksum-diagnostics
(and worker/db/validate.cljs's backing recompute-checksum-diagnostics,
which pulled in logseq-db-sync.checksum) compared a local checksum
against a remote one that no longer exists -- deleted outright, not
ported, since there's nothing left to diff against.
- worker/handler/block.cljs's :block.temp/sync-conflicts metadata
(backed by the deleted client-op/get-all-sync-conflicts) is gone;
its client-side consumer (a retry-upload/clear-conflicts UI in
components/block.cljs) is dead code now and still needs a follow-up
pass -- flagging here since worker-side cleanup alone can't remove
it without breaking that UI's thread-api calls at runtime.
Verified: app/electron/db-worker-node/mobile all compile clean. Full
worker test suite green (db_core_test 131, db_listener_test 11,
db_worker_test 8, db_worker_node_test 59, db_worker_node_lock_test 10,
local_tx_log_test 4, undo_redo_test 29, subs_test 50, subs_loader_test
2, config_test 6 -- 0 failures across all of them). The entire
src/test/frontend/worker/sync/ test directory (7 files) and
db_sync_test.cljs/db_sync_sim_test.cljs/node_sync_test.cljs (all
testing only-deleted namespaces) are deleted wholesale rather than
ported.
Known follow-up, not yet done: the client-side (frontend.handler.*,
frontend.components.*) RTC/E2EE/db-sync UI and its dead thread-api call
sites (components/block.cljs, components/repo.cljs,
components/settings.cljs, handler/user.cljs, handler/assets.cljs,
db/restore.cljs, handler/common/developer.cljs, and the
handler/db-based/{sync,rtc-flows,rtc-background-tasks}.cljs +
handler/events/rtc.cljs + components/rtc/* + db/rtc/* files themselves)
-- a separate commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -6,7 +6,6 @@
|
||||
[frontend.worker.db.migrate :as db-migrate]
|
||||
[frontend.worker.shared-service :as shared-service]
|
||||
[logseq.db :as ldb]
|
||||
[logseq.db-sync.checksum :as sync-checksum]
|
||||
[logseq.db.frontend.class :as db-class]
|
||||
[logseq.db.frontend.validate :as db-validate]))
|
||||
|
||||
@@ -326,13 +325,3 @@
|
||||
(merge {:errors errors
|
||||
:invalid-entity-ids invalid-entity-ids}
|
||||
counts)))
|
||||
|
||||
(defn recompute-checksum-diagnostics
|
||||
[_repo conn {:keys [local-checksum remote-checksum] :as _sync-diagnostics}]
|
||||
(let [{:keys [checksum attrs blocks e2ee?]} (sync-checksum/recompute-checksum-diagnostics @conn)]
|
||||
{:recomputed-checksum checksum
|
||||
:local-checksum local-checksum
|
||||
:remote-checksum remote-checksum
|
||||
:e2ee? e2ee?
|
||||
:checksum-attrs attrs
|
||||
:blocks blocks}))
|
||||
|
||||
@@ -24,7 +24,6 @@
|
||||
[frontend.worker.handler.query]
|
||||
[frontend.worker.handler.render-resource.engine]
|
||||
[frontend.worker.handler.search :as search-handler]
|
||||
[frontend.worker.handler.sync]
|
||||
[frontend.worker.handler.transaction :as transaction-handler]
|
||||
[frontend.worker.handler.undo-redo]
|
||||
[frontend.worker.handler.view]
|
||||
@@ -34,10 +33,7 @@
|
||||
[frontend.worker.search :as search]
|
||||
[frontend.worker.shared-service :as shared-service]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync :as db-sync]
|
||||
[frontend.worker.sync.client-op :as client-op]
|
||||
[frontend.worker.sync.crypt :as sync-crypt]
|
||||
[frontend.worker.sync.download :as sync-download]
|
||||
[frontend.worker.local-tx-log :as local-tx-log]
|
||||
[frontend.worker.thread-atom]
|
||||
[frontend.worker.undo-redo :as worker-undo-redo]
|
||||
[goog.functions :as gfun]
|
||||
@@ -364,7 +360,6 @@
|
||||
(checkpoint-db! repo db)
|
||||
(checkpoint-db! repo search)
|
||||
(checkpoint-db! repo client-ops)
|
||||
(sync-download/close-import-state-for-repo! repo)
|
||||
(when-let [timer (get @*client-ops-cleanup-timers repo)]
|
||||
(js/clearInterval timer))
|
||||
(swap! *client-ops-cleanup-timers dissoc repo)
|
||||
@@ -375,7 +370,7 @@
|
||||
(swap! *vector-indexes dissoc repo)
|
||||
(swap! *datascript-conns dissoc repo)
|
||||
(swap! *client-ops-conns dissoc repo)
|
||||
(swap! client-op/*repo->pending-local-tx-count dissoc repo)
|
||||
(swap! local-tx-log/*repo->pending-local-tx-count dissoc repo)
|
||||
(search-handler/clear-search-index-builds! repo)
|
||||
(when db (.close db))
|
||||
(when search (.close search))
|
||||
@@ -396,27 +391,6 @@
|
||||
(let [{:keys [db search client-ops]} (get @*sqlite-conns repo)]
|
||||
(close-db-aux! repo db search client-ops)))
|
||||
|
||||
(defn- <invalidate-search-db!
|
||||
[repo]
|
||||
(if-let [search-db (worker-state/get-sqlite-conn repo :search)]
|
||||
(do
|
||||
(search/truncate-table! search-db)
|
||||
(search/truncate-vector-index! (worker-state/get-vector-index repo))
|
||||
(p/resolved nil))
|
||||
(when-not @*publishing?
|
||||
(p/let [pool (<get-opfs-pool repo)
|
||||
search-path (resolve-db-path repo pool (str "search" repo-path))
|
||||
search-db (platform/sqlite-open (platform/current)
|
||||
{:sqlite @*sqlite
|
||||
:pool pool
|
||||
:path search-path
|
||||
:mode "c"})]
|
||||
(try
|
||||
(search/truncate-table! search-db)
|
||||
(finally
|
||||
(.close search-db)))
|
||||
nil))))
|
||||
|
||||
(defn- vector-index-path
|
||||
[repo pool]
|
||||
(resolve-db-path repo pool "search/vector"))
|
||||
@@ -476,7 +450,7 @@
|
||||
(defn- run-client-ops-cleanup!
|
||||
[repo]
|
||||
(let [protected-tx-ids (worker-undo-redo/referenced-history-tx-ids repo)]
|
||||
(client-op/cleanup-finished-history-ops! repo protected-tx-ids)
|
||||
(local-tx-log/cleanup-finished-history-ops! repo protected-tx-ids)
|
||||
nil))
|
||||
|
||||
(defn- ensure-client-ops-cleanup-timer!
|
||||
@@ -490,11 +464,6 @@
|
||||
(swap! *client-ops-cleanup-timers assoc repo timer))
|
||||
nil))
|
||||
|
||||
(defn- handle-migrate-result-local-txs!
|
||||
[repo migrate-result]
|
||||
(doseq [tx-report (:upgrade-result-coll migrate-result)]
|
||||
(db-sync/handle-local-tx! repo tx-report)))
|
||||
|
||||
(def ^:private built-in-sync-repair-tx-id
|
||||
#uuid "00000000-0000-4000-8000-652665286528")
|
||||
|
||||
@@ -551,9 +520,9 @@
|
||||
|
||||
(defn- enqueue-built-in-sync-repair!
|
||||
[repo]
|
||||
(when-not (client-op/get-local-tx-entry repo built-in-sync-repair-tx-id)
|
||||
(when-not (local-tx-log/get-local-tx-entry repo built-in-sync-repair-tx-id)
|
||||
(let [{:keys [should-inc-pending?]}
|
||||
(client-op/upsert-local-tx-entry!
|
||||
(local-tx-log/upsert-local-tx-entry!
|
||||
repo
|
||||
{:tx-id built-in-sync-repair-tx-id
|
||||
:created-at 0
|
||||
@@ -567,7 +536,7 @@
|
||||
:normalized-tx-data (built-in-sync-repair-tx-data)
|
||||
:reversed-tx-data []})]
|
||||
(when should-inc-pending?
|
||||
(client-op/adjust-pending-local-tx-count! repo 1)))))
|
||||
(local-tx-log/adjust-pending-local-tx-count! repo 1)))))
|
||||
|
||||
(defn- maybe-enqueue-built-in-sync-repair!
|
||||
[repo conn migrate-result initial-data-exists?]
|
||||
@@ -607,8 +576,8 @@
|
||||
(debug-transit-raw->datoms debug-transit-raw)))]
|
||||
(when creating-remote-graph?
|
||||
(when (and (worker-state/get-sqlite-conn repo :client-ops)
|
||||
(nil? (client-op/get-local-tx repo)))
|
||||
(client-op/update-local-tx repo 0)))
|
||||
(nil? (local-tx-log/get-local-tx repo)))
|
||||
(local-tx-log/update-local-tx repo 0)))
|
||||
(when-not (worker-state/get-sqlite-conn repo)
|
||||
(p/let [[db search-db client-ops-db vector-index] (get-dbs repo)
|
||||
dbs (cond-> [db search-db]
|
||||
@@ -656,30 +625,26 @@
|
||||
(swap! *datascript-conns assoc repo conn)
|
||||
(swap! *client-ops-conns assoc repo client-ops-conn)
|
||||
(when-not @*publishing?
|
||||
(client-op/ensure-sqlite-schema! client-ops-db))
|
||||
(local-tx-log/ensure-sqlite-schema! client-ops-db))
|
||||
(when creating-remote-graph?
|
||||
(when (nil? (client-op/get-local-tx repo))
|
||||
(client-op/update-local-tx repo 0)))
|
||||
(when (nil? (local-tx-log/get-local-tx repo))
|
||||
(local-tx-log/update-local-tx repo 0)))
|
||||
(ensure-client-ops-cleanup-timer! repo)
|
||||
(let [initial-tx-report (when-not (or initial-data-exists?
|
||||
(seq datoms)
|
||||
sync-download-graph?)
|
||||
(let [config (resolve-initial-config config)
|
||||
initial-data (sqlite-create-graph/build-db-initial-data
|
||||
config (select-keys opts [:import-type :graph-git-sha :creating-remote-graph?]))]
|
||||
(bootstrap-transact! conn initial-data)))]
|
||||
(do
|
||||
(when-not (or initial-data-exists?
|
||||
(seq datoms)
|
||||
sync-download-graph?)
|
||||
(let [config (resolve-initial-config config)
|
||||
initial-data (sqlite-create-graph/build-db-initial-data
|
||||
config (select-keys opts [:import-type :graph-git-sha :creating-remote-graph?]))]
|
||||
(bootstrap-transact! conn initial-data)))
|
||||
(when-not sync-download-graph?
|
||||
(let [migrate-result (db-migrate/migrate conn)]
|
||||
(if migrate-result
|
||||
(handle-migrate-result-local-txs! repo migrate-result)
|
||||
(maybe-enqueue-built-in-sync-repair! repo conn migrate-result initial-data-exists?)))
|
||||
(maybe-enqueue-built-in-sync-repair! repo conn migrate-result initial-data-exists?))
|
||||
(transaction-handler/maybe-run-recycle-gc! conn))
|
||||
|
||||
(ensure-canonical-revisions! conn)
|
||||
|
||||
(when initial-tx-report
|
||||
(db-sync/handle-local-tx! repo initial-tx-report))
|
||||
|
||||
(db-listener/listen-db-changes! repo conn)
|
||||
|
||||
nil))))))
|
||||
@@ -713,19 +678,7 @@
|
||||
[]
|
||||
(init-sqlite-module!))
|
||||
|
||||
(defn- db-sync-dbs-open?
|
||||
[repo]
|
||||
(and (some? (worker-state/get-datascript-conn repo))
|
||||
(some? (worker-state/get-client-ops-conn repo))))
|
||||
|
||||
(declare start-db!)
|
||||
(def-thread-api :thread-api/db-sync-start
|
||||
[repo]
|
||||
(if (db-sync-dbs-open? repo)
|
||||
(db-sync/start! repo)
|
||||
(p/do!
|
||||
(start-db! repo {:close-other-db? false})
|
||||
(db-sync/start! repo))))
|
||||
|
||||
;; [graph service]
|
||||
(defonce *service (atom []))
|
||||
@@ -772,54 +725,17 @@
|
||||
(def-thread-api :thread-api/unsafe-unlink-db
|
||||
[repo]
|
||||
(p/let [pool (<get-opfs-pool repo)
|
||||
_ (sync-crypt/cancel-ui-requests! {:reason :unsafe-unlink-db
|
||||
:repo repo})
|
||||
_ (close-db! repo)
|
||||
_result (remove-vfs! pool)]
|
||||
nil))
|
||||
|
||||
(def-thread-api :thread-api/close-db
|
||||
[repo]
|
||||
(sync-crypt/cancel-ui-requests! {:reason :close-db
|
||||
:repo repo})
|
||||
(close-db! repo)
|
||||
nil)
|
||||
|
||||
(def-thread-api :thread-api/db-sync-close-db
|
||||
[repo]
|
||||
(sync-crypt/cancel-ui-requests! {:reason :db-sync-close-db
|
||||
:repo repo})
|
||||
(close-db! repo))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-invalidate-search-db
|
||||
[repo]
|
||||
(<invalidate-search-db! repo))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-recreate-lock
|
||||
[repo]
|
||||
(if-let [recreate-lock-fn (get-in (platform/current) [:env :recreate-lock-fn])]
|
||||
(recreate-lock-fn repo)
|
||||
nil))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-rehydrate-large-titles
|
||||
[repo graph-id]
|
||||
(db-sync/rehydrate-large-titles-from-db! repo graph-id))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-import-prepare
|
||||
[repo reset? graph-id graph-e2ee? & [total-datoms]]
|
||||
(sync-download/prepare-import! repo reset? graph-id graph-e2ee? total-datoms))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-import-rows-chunk
|
||||
[rows graph-id import-id]
|
||||
(sync-download/import-rows-chunk! rows graph-id import-id))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-import-finalize
|
||||
[repo graph-id remote-tx import-id]
|
||||
(sync-download/finalize-import! repo graph-id remote-tx import-id))
|
||||
|
||||
(def-thread-api :thread-api/release-access-handles
|
||||
[repo]
|
||||
(sync-download/close-import-state-for-repo! repo)
|
||||
(when-let [^js pool (get-storage-pool repo)]
|
||||
(when (exists? (.-pauseVfs pool))
|
||||
(.pauseVfs pool))
|
||||
@@ -929,12 +845,9 @@
|
||||
(set (map
|
||||
common-util/keyword->string
|
||||
[:sync-db-changes
|
||||
:sync-conflicts-updated
|
||||
:notification
|
||||
:log
|
||||
:add-repo
|
||||
:rtc-log
|
||||
:rtc-sync-state])))
|
||||
:add-repo])))
|
||||
|
||||
(defn- <init-service!
|
||||
[graph start-opts]
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
[frontend.worker.search :as search]
|
||||
[frontend.worker.shared-service :as shared-service]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync :as db-sync]
|
||||
[frontend.worker.undo-redo.replay :as undo-redo-replay]
|
||||
[lambdaisland.glogi :as log]
|
||||
[promesa.core :as p]))
|
||||
|
||||
@@ -154,9 +154,9 @@
|
||||
(prn :tx-data tx-data)
|
||||
(prn :tx-meta tx-meta)))
|
||||
|
||||
(defmethod listen-db-changes :db-sync
|
||||
(defmethod listen-db-changes :persist-local-tx
|
||||
[_ {:keys [repo]} tx-report]
|
||||
(db-sync/handle-local-tx! repo tx-report))
|
||||
(undo-redo-replay/persist-local-tx! repo tx-report))
|
||||
|
||||
(defmethod listen-db-changes :markdown-mirror
|
||||
[_ {:keys [repo]} tx-report]
|
||||
@@ -229,9 +229,6 @@
|
||||
[repo conn persist-local-tx-handler deferred-handlers
|
||||
sync-db-to-main-thread? {:keys [tx-data tx-meta] :as tx-report}]
|
||||
(let [started-at (perf-time-ms)]
|
||||
(run-post-commit!
|
||||
repo tx-meta :update-checksum
|
||||
#(db-sync/update-local-sync-checksum! repo tx-report))
|
||||
(let [checksum-at (perf-time-ms)
|
||||
opt {:repo repo}
|
||||
handler-timings (atom [])]
|
||||
@@ -239,7 +236,7 @@
|
||||
(run-post-commit!
|
||||
repo tx-meta :persist-local-tx
|
||||
#(invoke-listener-handler! handler-timings
|
||||
:db-sync
|
||||
:persist-local-tx
|
||||
persist-local-tx-handler
|
||||
opt
|
||||
tx-report)))
|
||||
@@ -268,8 +265,8 @@
|
||||
(let [handlers (if (seq handler-keys)
|
||||
(select-keys (methods listen-db-changes) handler-keys)
|
||||
(methods listen-db-changes))
|
||||
persist-local-tx-handler (get handlers :db-sync)
|
||||
deferred-handlers (dissoc handlers :db-sync)
|
||||
persist-local-tx-handler (get handlers :persist-local-tx)
|
||||
deferred-handlers (dissoc handlers :persist-local-tx)
|
||||
sync-db-to-main-thread?
|
||||
(or (nil? handler-keys)
|
||||
(contains? (set handler-keys) :sync-db-to-main-thread))]
|
||||
|
||||
@@ -246,28 +246,12 @@
|
||||
|
||||
(def ^:private non-repo-methods
|
||||
#{:thread-api/init
|
||||
:thread-api/set-db-sync-config
|
||||
:thread-api/get-db-sync-config
|
||||
:thread-api/db-sync-stop
|
||||
:thread-api/db-sync-list-remote-graphs
|
||||
:thread-api/db-sync-update-presence
|
||||
:thread-api/db-sync-ensure-user-rsa-keys
|
||||
:thread-api/list-db
|
||||
:thread-api/get-version
|
||||
:thread-api/set-context
|
||||
:thread-api/sync-app-state
|
||||
:thread-api/update-thread-atom
|
||||
:thread-api/mobile-logs
|
||||
:thread-api/get-user-rsa-key-pair
|
||||
:thread-api/init-user-rsa-key-pair
|
||||
:thread-api/reset-user-rsa-key-pair
|
||||
:thread-api/change-e2ee-password
|
||||
:thread-api/get-e2ee-password
|
||||
:thread-api/save-e2ee-password
|
||||
:thread-api/verify-and-save-e2ee-password
|
||||
:thread-api/resolve-ui-request
|
||||
:thread-api/reject-ui-request
|
||||
:thread-api/cancel-ui-requests})
|
||||
:thread-api/mobile-logs})
|
||||
|
||||
(defn- repo-arg
|
||||
[args]
|
||||
|
||||
@@ -11,7 +11,6 @@
|
||||
[frontend.worker.handler.search :as search-handler]
|
||||
[frontend.worker.plain-value :as worker-plain]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync.client-op :as client-op]
|
||||
[logseq.common.util :as common-util]
|
||||
[logseq.db :as ldb]
|
||||
[logseq.db.common.initial-data :as common-initial-data]
|
||||
@@ -440,7 +439,7 @@
|
||||
block (conj block)))
|
||||
|
||||
(defn- assoc-block-metadata
|
||||
[db conflicts-by-block commented-block-uuids now-ms render-data? block]
|
||||
[db commented-block-uuids now-ms render-data? block]
|
||||
(let [block-id (:db/id block)
|
||||
block-uuid (:block/uuid block)]
|
||||
(cond-> (assoc block
|
||||
@@ -448,23 +447,21 @@
|
||||
:block.temp/breadcrumb
|
||||
(block-breadcrumb/block-breadcrumb db block)
|
||||
:block.temp/comment-thread-present?
|
||||
(contains? commented-block-uuids (str block-uuid))
|
||||
:block.temp/sync-conflicts
|
||||
(vec (get conflicts-by-block block-uuid)))
|
||||
(contains? commented-block-uuids (str block-uuid)))
|
||||
render-data?
|
||||
(assoc :block.temp/task-spent-time
|
||||
(or (query-handler/task-spent-time db block-id now-ms) [])))))
|
||||
|
||||
(defn- assoc-result-block-metadata
|
||||
[result db conflicts-by-block commented-block-uuids now-ms render-data? root-render-data?]
|
||||
[result db commented-block-uuids now-ms render-data? root-render-data?]
|
||||
(cond-> result
|
||||
(:block result)
|
||||
(update :block #(assoc-block-metadata db conflicts-by-block commented-block-uuids
|
||||
(update :block #(assoc-block-metadata db commented-block-uuids
|
||||
now-ms (or render-data? root-render-data?) %))
|
||||
|
||||
(:children result)
|
||||
(update :children (fn [children]
|
||||
(mapv #(assoc-block-metadata db conflicts-by-block commented-block-uuids
|
||||
(mapv #(assoc-block-metadata db commented-block-uuids
|
||||
now-ms render-data? %)
|
||||
children)))))
|
||||
|
||||
@@ -485,14 +482,12 @@
|
||||
(mapv :block/uuid)
|
||||
(comments-handler/get-comment-thread-block-uuids db)
|
||||
set))
|
||||
conflicts-by-block (when (seq metadata-blocks)
|
||||
(client-op/get-all-sync-conflicts repo))
|
||||
now-ms (common-util/time-ms)]
|
||||
(->> (mapv (fn [request result]
|
||||
(cond-> result
|
||||
(get-in request [:opts :block-metadata?])
|
||||
(assoc-result-block-metadata
|
||||
db conflicts-by-block commented-block-uuids now-ms
|
||||
db commented-block-uuids now-ms
|
||||
(true? (get-in request [:opts :render-data?]))
|
||||
(true? (get-in request [:opts :root-render-data?])))))
|
||||
requests
|
||||
|
||||
@@ -5,8 +5,6 @@
|
||||
[frontend.worker.export :as worker-export]
|
||||
[frontend.worker.platform :as platform]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync :as db-sync]
|
||||
[frontend.worker.sync.client-op :as client-op]
|
||||
[logseq.db :as ldb]
|
||||
[logseq.db.sqlite.export :as sqlite-export]
|
||||
[logseq.publishing.html :as publish-html]))
|
||||
@@ -41,23 +39,6 @@
|
||||
(when-let [conn (worker-state/get-datascript-conn repo)]
|
||||
(worker-db-validate/validate-db conn opts)))
|
||||
|
||||
(defn- checksum-diagnostics
|
||||
[repo]
|
||||
{:local-checksum (client-op/get-local-checksum repo)
|
||||
:remote-checksum (get @db-sync/*repo->latest-remote-checksum repo)})
|
||||
|
||||
(def-thread-api :thread-api/recompute-checksum-diagnostics
|
||||
[repo]
|
||||
(when-let [conn (worker-state/get-datascript-conn repo)]
|
||||
(let [result (worker-db-validate/recompute-checksum-diagnostics repo conn (checksum-diagnostics repo))
|
||||
recomputed-checksum (:recomputed-checksum result)]
|
||||
(when (and (some? recomputed-checksum)
|
||||
(worker-state/get-client-ops-conn repo))
|
||||
(client-op/update-local-checksum repo recomputed-checksum))
|
||||
(cond-> result
|
||||
(some? recomputed-checksum)
|
||||
(assoc :local-checksum recomputed-checksum)))))
|
||||
|
||||
;; Returns an export-edn map for given repo. When there's an unexpected error, a map
|
||||
;; with key :export-edn-error is returned.
|
||||
(def-thread-api :thread-api/export-edn
|
||||
|
||||
@@ -1,104 +0,0 @@
|
||||
(ns frontend.worker.handler.sync
|
||||
"Sync operations for the db worker."
|
||||
(:require [frontend.common.thread-api :refer [def-thread-api]]
|
||||
[frontend.worker.shared-service :as shared-service]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync :as db-sync]
|
||||
[frontend.worker.sync.client-op :as client-op]
|
||||
[frontend.worker.sync.crypt :as sync-crypt]
|
||||
[frontend.worker.sync.download :as sync-download]))
|
||||
|
||||
(def-thread-api :thread-api/set-db-sync-config
|
||||
[config]
|
||||
(reset! worker-state/*db-sync-config (worker-state/non-auth-db-sync-config config))
|
||||
nil)
|
||||
|
||||
(def-thread-api :thread-api/get-db-sync-config
|
||||
[]
|
||||
(worker-state/non-auth-db-sync-config @worker-state/*db-sync-config))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-status
|
||||
[repo]
|
||||
(db-sync/status repo))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-stop
|
||||
[]
|
||||
(db-sync/stop!))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-update-presence
|
||||
[editing-block-uuid]
|
||||
(db-sync/update-presence! editing-block-uuid))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-request-asset-download
|
||||
[repo asset-uuid]
|
||||
(db-sync/request-asset-download! repo asset-uuid))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-download-missing-assets
|
||||
[repo graph-id]
|
||||
(db-sync/download-missing-assets! repo graph-id))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-retry-asset-upload
|
||||
[repo]
|
||||
(db-sync/retry-asset-upload! repo))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-grant-graph-access
|
||||
[repo graph-id target-email]
|
||||
(sync-crypt/<grant-graph-access! repo graph-id target-email))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-ensure-user-rsa-keys
|
||||
[& [opts]]
|
||||
(sync-crypt/ensure-user-rsa-keys! opts))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-list-remote-graphs
|
||||
[]
|
||||
(db-sync/list-remote-graphs!))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-upload-graph
|
||||
[repo]
|
||||
(db-sync/upload-graph! repo))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-create-remote-graph
|
||||
[repo graph-e2ee? graph-ready-for-use?]
|
||||
(db-sync/create-remote-graph! repo {:graph-e2ee? graph-e2ee?
|
||||
:graph-ready-for-use? graph-ready-for-use?}))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-stop-upload
|
||||
[repo]
|
||||
(db-sync/stop-upload! repo))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-resume-upload
|
||||
[repo]
|
||||
(db-sync/resume-upload! repo))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-upload-stopped?
|
||||
[repo]
|
||||
(db-sync/upload-stopped? repo))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-get-all-block-conflicts
|
||||
[repo]
|
||||
(let [conflicts (client-op/get-all-sync-conflicts repo)]
|
||||
(when-not (map? conflicts)
|
||||
(throw (ex-info "Expected sync conflicts grouped by block"
|
||||
{:repo repo
|
||||
:conflicts conflicts})))
|
||||
(into {}
|
||||
(map (fn [[block-uuid block-conflicts]]
|
||||
(when-not (uuid? block-uuid)
|
||||
(throw (ex-info "Expected sync conflict block UUID"
|
||||
{:repo repo
|
||||
:block-uuid block-uuid})))
|
||||
[(str block-uuid) block-conflicts]))
|
||||
conflicts)))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-clear-block-conflicts
|
||||
[repo block-uuid]
|
||||
(client-op/clear-sync-conflicts! repo block-uuid)
|
||||
(shared-service/broadcast-to-clients!
|
||||
:sync-conflicts-updated
|
||||
{:repo repo
|
||||
:block-uuid block-uuid
|
||||
:conflicts []}))
|
||||
|
||||
(def-thread-api :thread-api/db-sync-download-graph-by-id
|
||||
[repo graph-id graph-e2ee?]
|
||||
(sync-download/download-graph-by-id! repo graph-id graph-e2ee?))
|
||||
347
src/main/frontend/worker/local_tx_log.cljs
Normal file
347
src/main/frontend/worker/local_tx_log.cljs
Normal file
@@ -0,0 +1,347 @@
|
||||
(ns frontend.worker.local-tx-log
|
||||
"SQLite-backed local tx ledger, keyed by tx-id. Originally part of the
|
||||
now-deleted remote sync client (frontend.worker.sync.client-op, see Fase
|
||||
8 in docs/agent-guide); the sync-specific surface (pending-upload
|
||||
counts/checksums/graph-uuid/conflict tracking/asset-op queues) is gone,
|
||||
but two callers still need this table: db_core.cljs's built-in-property
|
||||
repair uses it as an idempotency marker (has this repo already gotten the
|
||||
repair tx?) via get-local-tx-entry/upsert-local-tx-entry!, and
|
||||
undo_redo.cljs reads a tx's recorded forward/inverse outliner ops back out
|
||||
via history-action-ops-by-tx-id so a repair tx can be undone/redone like
|
||||
any other edit."
|
||||
(:require [frontend.worker.state :as worker-state]
|
||||
[goog.object :as gobj]
|
||||
[logseq.db.sqlite.util :as sqlite-util]))
|
||||
|
||||
(defonce *repo->pending-local-tx-count (atom {}))
|
||||
|
||||
(def ^:private sqlite-schema-ready-key "__logseq_client_ops_schema_ready_v2")
|
||||
(def ^:private sqlite-mode-key "__logseq_client_ops_sqlite_mode")
|
||||
(def ^:private sync-meta-table-sql
|
||||
"create table if not exists sync_meta (key text primary key, value text)")
|
||||
(def ^:private client-ops-table-sql
|
||||
(str "create table if not exists client_ops ("
|
||||
"id integer primary key autoincrement,"
|
||||
"kind text not null,"
|
||||
"created_at integer not null,"
|
||||
"tx_id text unique,"
|
||||
"pending integer not null default 0,"
|
||||
"failed integer not null default 0,"
|
||||
"outliner_op text,"
|
||||
"undo_redo text,"
|
||||
"forward_outliner_ops text,"
|
||||
"inverse_outliner_ops text,"
|
||||
"inferred_outliner_ops integer,"
|
||||
"normalized_tx_data text,"
|
||||
"reversed_tx_data text,"
|
||||
"asset_uuid text,"
|
||||
"asset_op text,"
|
||||
"asset_t integer,"
|
||||
"asset_value text"
|
||||
")"))
|
||||
(def ^:private pending-index-sql
|
||||
"create index if not exists idx_client_ops_pending_created on client_ops(kind, pending, created_at, id)")
|
||||
|
||||
(defn- client-ops-store
|
||||
[repo]
|
||||
(worker-state/get-client-ops-conn repo))
|
||||
|
||||
(declare ensure-sqlite-schema!)
|
||||
|
||||
(defn- detect-sqlite-mode
|
||||
[^js db]
|
||||
(or (gobj/get db sqlite-mode-key)
|
||||
(let [mode
|
||||
(cond
|
||||
(not db) nil
|
||||
(fn? (gobj/get db "prepare"))
|
||||
(try
|
||||
(let [^js stmt (.prepare db "select 1")]
|
||||
(try
|
||||
(if (fn? (gobj/get stmt "run"))
|
||||
:better-sqlite
|
||||
:sqlite-wasm)
|
||||
(finally
|
||||
(when (fn? (gobj/get stmt "finalize"))
|
||||
(.finalize stmt)))))
|
||||
(catch :default _
|
||||
:sqlite-wasm))
|
||||
(fn? (gobj/get db "exec")) :sqlite-wasm
|
||||
:else nil)]
|
||||
(when mode
|
||||
(try
|
||||
(gobj/set db sqlite-mode-key mode)
|
||||
(catch :default _
|
||||
nil)))
|
||||
mode)))
|
||||
|
||||
(defn- sqlite-db?
|
||||
[conn]
|
||||
(some? (detect-sqlite-mode conn)))
|
||||
|
||||
(defn- sqlite-store-or-throw
|
||||
[repo]
|
||||
(when-let [store (client-ops-store repo)]
|
||||
(if (sqlite-db? store)
|
||||
(do
|
||||
(ensure-sqlite-schema! store)
|
||||
store)
|
||||
(throw (ex-info "Legacy DataScript client-op storage is unsupported. Please back up the graph and re-download it."
|
||||
{:type :db-sync/legacy-client-ops-storage
|
||||
:repo repo})))))
|
||||
|
||||
(defn- parse-uuid-str
|
||||
[v]
|
||||
(when (string? v)
|
||||
(try
|
||||
(uuid v)
|
||||
(catch :default _
|
||||
nil))))
|
||||
|
||||
(defn- kw->str
|
||||
[v]
|
||||
(cond
|
||||
(keyword? v) (name v)
|
||||
(string? v) v
|
||||
:else nil))
|
||||
|
||||
(defn- str->kw
|
||||
[v]
|
||||
(when (string? v)
|
||||
(keyword v)))
|
||||
|
||||
(defn- bool->int [v] (if v 1 0))
|
||||
(defn- int->bool [v] (not (or (nil? v) (= 0 v) (= false v))))
|
||||
|
||||
(defn- normalize-op-entries
|
||||
[ops]
|
||||
(let [ops' (some-> ops seq vec)]
|
||||
(cond
|
||||
(nil? ops')
|
||||
nil
|
||||
|
||||
(and (keyword? (first ops'))
|
||||
(vector? (second ops')))
|
||||
[ops']
|
||||
|
||||
:else
|
||||
ops')))
|
||||
|
||||
(defn- sqlite-run!
|
||||
[^js db sql params]
|
||||
(case (detect-sqlite-mode db)
|
||||
:better-sqlite
|
||||
(let [^js stmt (.prepare db sql)
|
||||
run-fn (gobj/get stmt "run")]
|
||||
(if (seq params)
|
||||
(.apply run-fn stmt (to-array params))
|
||||
(.call run-fn stmt)))
|
||||
|
||||
:sqlite-wasm
|
||||
(.exec db #js {:sql sql
|
||||
:bind (into-array params)})
|
||||
|
||||
nil))
|
||||
|
||||
(defn- sqlite-rows
|
||||
[^js db sql params]
|
||||
(case (detect-sqlite-mode db)
|
||||
:better-sqlite
|
||||
(let [^js stmt (.prepare db sql)
|
||||
all-fn (gobj/get stmt "all")]
|
||||
(vec (if (seq params)
|
||||
(.apply all-fn stmt (to-array params))
|
||||
(.call all-fn stmt))))
|
||||
|
||||
:sqlite-wasm
|
||||
(let [^js result (.exec db #js {:sql sql
|
||||
:bind (into-array params)
|
||||
:rowMode "object"
|
||||
:returnValue "resultRows"})]
|
||||
(cond
|
||||
(nil? result) []
|
||||
(array? result) (vec result)
|
||||
(fn? (gobj/get result "toArray")) (vec (.toArray result))
|
||||
:else []))
|
||||
|
||||
[]))
|
||||
|
||||
(defn- sqlite-row
|
||||
[db sql params]
|
||||
(first (sqlite-rows db sql params)))
|
||||
|
||||
(defn- sqlite-with-tx!
|
||||
[^js db f]
|
||||
(case (detect-sqlite-mode db)
|
||||
:better-sqlite
|
||||
(let [tx-fn (.transaction db (fn [] (f db)))]
|
||||
(tx-fn))
|
||||
|
||||
:sqlite-wasm
|
||||
(if (fn? (gobj/get db "transaction"))
|
||||
(.transaction db (fn [tx] (f tx)))
|
||||
(f db))
|
||||
|
||||
(f db)))
|
||||
|
||||
(defn ensure-sqlite-schema!
|
||||
[db]
|
||||
(when (sqlite-db? db)
|
||||
(when-not (true? (gobj/get db sqlite-schema-ready-key))
|
||||
(sqlite-with-tx!
|
||||
db
|
||||
(fn [tx]
|
||||
(sqlite-run! tx sync-meta-table-sql [])
|
||||
(sqlite-run! tx client-ops-table-sql [])
|
||||
(sqlite-run! tx pending-index-sql [])))
|
||||
(try
|
||||
(gobj/set db sqlite-schema-ready-key true)
|
||||
(catch :default _
|
||||
nil)))))
|
||||
|
||||
(defn- sqlite-get-meta
|
||||
[db k]
|
||||
(some-> (sqlite-row db "select value from sync_meta where key = ?" [(name k)])
|
||||
(aget "value")))
|
||||
|
||||
(defn- sqlite-set-meta!
|
||||
[db k v]
|
||||
(sqlite-run! db
|
||||
(str "insert into sync_meta (key, value) values (?, ?)"
|
||||
" on conflict(key) do update set value = excluded.value")
|
||||
[(name k) (str v)]))
|
||||
|
||||
(defn get-local-tx
|
||||
[repo]
|
||||
(when-let [store (sqlite-store-or-throw repo)]
|
||||
(when-let [result (sqlite-get-meta store :local-tx)]
|
||||
(js/parseInt result 10))))
|
||||
|
||||
(defn update-local-tx
|
||||
[repo t]
|
||||
{:pre [(and (integer? t) (>= t 0))]}
|
||||
(let [store (sqlite-store-or-throw repo)
|
||||
prev-t (get-local-tx repo)]
|
||||
(assert (some? store) repo)
|
||||
(when (and prev-t (< t prev-t))
|
||||
(throw (ex-info "local-tx should be monotonically increasing"
|
||||
{:repo repo
|
||||
:prev-t prev-t
|
||||
:new-t t})))
|
||||
(sqlite-set-meta! store :local-tx t)))
|
||||
|
||||
(defn- row->pending-local-tx
|
||||
[row]
|
||||
(let [tx-id (parse-uuid-str (aget row "tx_id"))]
|
||||
(when tx-id
|
||||
{:tx-id tx-id
|
||||
:outliner-op (str->kw (aget row "outliner_op"))
|
||||
:forward-outliner-ops (or (normalize-op-entries
|
||||
(sqlite-util/read-transit-str (aget row "forward_outliner_ops")))
|
||||
[])
|
||||
:inverse-outliner-ops (or (normalize-op-entries
|
||||
(sqlite-util/read-transit-str (aget row "inverse_outliner_ops")))
|
||||
[])
|
||||
:inferred-outliner-ops? (int->bool (aget row "inferred_outliner_ops"))
|
||||
:db-sync/undo-redo (str->kw (aget row "undo_redo"))
|
||||
:tx (sqlite-util/read-transit-str (aget row "normalized_tx_data"))
|
||||
:reversed-tx (sqlite-util/read-transit-str (aget row "reversed_tx_data"))})))
|
||||
|
||||
(defn upsert-local-tx-entry!
|
||||
[repo {:keys [tx-id created-at pending? failed? outliner-op undo-redo
|
||||
forward-outliner-ops inverse-outliner-ops inferred-outliner-ops?
|
||||
normalized-tx-data reversed-tx-data]
|
||||
:or {pending? true failed? false}}]
|
||||
{:pre [(some? tx-id)]}
|
||||
(let [store (sqlite-store-or-throw repo)]
|
||||
(assert (some? store) repo)
|
||||
(let [tx-id-str (str tx-id)
|
||||
existing (sqlite-row store
|
||||
"select pending, created_at from client_ops where kind = 'tx' and tx_id = ?"
|
||||
[tx-id-str])
|
||||
should-inc-pending? (not= 1 (some-> existing (aget "pending")))
|
||||
forward-outliner-ops' (or (normalize-op-entries forward-outliner-ops) [])
|
||||
inverse-outliner-ops' (or (normalize-op-entries inverse-outliner-ops) [])
|
||||
created-at' (or (some-> existing (aget "created_at"))
|
||||
created-at
|
||||
(.now js/Date))]
|
||||
(sqlite-run! store
|
||||
(str "insert into client_ops ("
|
||||
"kind, created_at, tx_id, pending, failed, outliner_op, undo_redo, "
|
||||
"forward_outliner_ops, inverse_outliner_ops, inferred_outliner_ops, "
|
||||
"normalized_tx_data, reversed_tx_data"
|
||||
") values ('tx', ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
|
||||
" on conflict(tx_id) do update set "
|
||||
"created_at = excluded.created_at,"
|
||||
"pending = excluded.pending,"
|
||||
"failed = excluded.failed,"
|
||||
"outliner_op = excluded.outliner_op,"
|
||||
"undo_redo = excluded.undo_redo,"
|
||||
"forward_outliner_ops = excluded.forward_outliner_ops,"
|
||||
"inverse_outliner_ops = excluded.inverse_outliner_ops,"
|
||||
"inferred_outliner_ops = excluded.inferred_outliner_ops,"
|
||||
"normalized_tx_data = excluded.normalized_tx_data,"
|
||||
"reversed_tx_data = excluded.reversed_tx_data")
|
||||
[created-at'
|
||||
tx-id-str
|
||||
(bool->int pending?)
|
||||
(bool->int failed?)
|
||||
(kw->str outliner-op)
|
||||
(kw->str undo-redo)
|
||||
(sqlite-util/write-transit-str forward-outliner-ops')
|
||||
(sqlite-util/write-transit-str inverse-outliner-ops')
|
||||
(bool->int inferred-outliner-ops?)
|
||||
(sqlite-util/write-transit-str (or normalized-tx-data []))
|
||||
(sqlite-util/write-transit-str (or reversed-tx-data []))])
|
||||
{:tx-id tx-id
|
||||
:created-at created-at'
|
||||
:should-inc-pending? should-inc-pending?})))
|
||||
|
||||
(defn get-local-tx-entry
|
||||
[repo tx-id]
|
||||
(when (uuid? tx-id)
|
||||
(when-let [store (sqlite-store-or-throw repo)]
|
||||
(some-> (sqlite-row store
|
||||
(str "select tx_id, outliner_op, undo_redo, "
|
||||
"forward_outliner_ops, inverse_outliner_ops, inferred_outliner_ops, "
|
||||
"normalized_tx_data, reversed_tx_data "
|
||||
"from client_ops where kind = 'tx' and tx_id = ? limit 1")
|
||||
[(str tx-id)])
|
||||
(row->pending-local-tx)))))
|
||||
|
||||
(defn adjust-pending-local-tx-count!
|
||||
[repo delta]
|
||||
(swap! *repo->pending-local-tx-count
|
||||
(fn [m]
|
||||
(let [base (or (get m repo) 0)
|
||||
next (max 0 (+ base delta))]
|
||||
(assoc m repo next)))))
|
||||
|
||||
(defn history-action-ops-by-tx-id
|
||||
[repo tx-id]
|
||||
(when-let [entry (get-local-tx-entry repo tx-id)]
|
||||
{:db-sync/forward-outliner-ops (some-> (:forward-outliner-ops entry) seq vec)
|
||||
:db-sync/inverse-outliner-ops (some-> (:inverse-outliner-ops entry) seq vec)}))
|
||||
|
||||
(defn cleanup-finished-history-ops!
|
||||
[repo protected-tx-ids]
|
||||
(if-let [store (sqlite-store-or-throw repo)]
|
||||
(let [protected-tx-ids (set protected-tx-ids)
|
||||
tx-id-rows (sqlite-rows store
|
||||
(str "select tx_id from client_ops "
|
||||
"where kind = 'tx' and pending = 0 and tx_id is not null")
|
||||
[])
|
||||
removable-tx-ids (->> tx-id-rows
|
||||
(keep (fn [row]
|
||||
(let [tx-id (parse-uuid-str (aget row "tx_id"))]
|
||||
(when (and (uuid? tx-id)
|
||||
(not (contains? protected-tx-ids tx-id)))
|
||||
tx-id))))
|
||||
vec)]
|
||||
(when (seq removable-tx-ids)
|
||||
(doseq [tx-id removable-tx-ids]
|
||||
(sqlite-run! store
|
||||
"delete from client_ops where kind = 'tx' and tx_id = ?"
|
||||
[(str tx-id)])))
|
||||
(count removable-tx-ids))
|
||||
0))
|
||||
@@ -1,465 +0,0 @@
|
||||
(ns frontend.worker.sync
|
||||
"Sync client"
|
||||
(:require
|
||||
[frontend.worker.platform :as platform]
|
||||
[frontend.worker.shared-service :as shared-service]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync.apply-txs :as sync-apply]
|
||||
[frontend.worker.sync.assets :as sync-assets]
|
||||
[frontend.worker.sync.auth :as sync-auth]
|
||||
[frontend.worker.sync.client-op :as client-op]
|
||||
[frontend.worker.sync.handle-message :as sync-handle-message]
|
||||
[frontend.worker.sync.presence :as sync-presence]
|
||||
[frontend.worker.sync.transport :as sync-transport]
|
||||
[frontend.worker.sync.upload :as sync-upload]
|
||||
[frontend.worker.sync.util :as sync-util]
|
||||
[lambdaisland.glogi :as log]
|
||||
[logseq.common.util :as common-util]
|
||||
[logseq.db-sync.checksum :as sync-checksum]
|
||||
[promesa.core :as p]
|
||||
[logseq.common.config :as common-config]))
|
||||
|
||||
(def ^:private reconnect-base-delay-ms 1000)
|
||||
(def ^:private reconnect-max-delay-ms 30000)
|
||||
(def ^:private reconnect-jitter-ms 250)
|
||||
(def ^:private ws-stale-kill-interval-ms 60000)
|
||||
(def ^:private ws-stale-timeout-ms 600000)
|
||||
(def fail-fast sync-util/fail-fast)
|
||||
|
||||
(defonce *repo->latest-remote-tx sync-apply/*repo->latest-remote-tx)
|
||||
(defonce *repo->latest-remote-checksum sync-apply/*repo->latest-remote-checksum)
|
||||
(defonce *start-inflight-target (atom nil))
|
||||
|
||||
(defn- current-client
|
||||
[repo]
|
||||
(sync-presence/current-client worker-state/*db-sync-client repo))
|
||||
|
||||
(defn status
|
||||
[repo]
|
||||
(sync-presence/sync-counts
|
||||
{:get-datascript-conn worker-state/get-datascript-conn
|
||||
:get-client-ops-conn worker-state/get-client-ops-conn
|
||||
:get-pending-local-tx-count client-op/get-pending-local-tx-count
|
||||
:get-unpushed-asset-ops-count client-op/get-unpushed-asset-ops-count
|
||||
:get-missing-asset-upload-files sync-assets/get-missing-asset-upload-files
|
||||
:get-local-tx client-op/get-local-tx
|
||||
:get-local-checksum client-op/get-local-checksum
|
||||
:get-graph-uuid client-op/get-graph-uuid
|
||||
:latest-remote-tx @*repo->latest-remote-tx
|
||||
:latest-remote-checksum @*repo->latest-remote-checksum}
|
||||
repo))
|
||||
|
||||
(defn update-local-sync-checksum!
|
||||
[repo tx-report]
|
||||
(when (worker-state/get-client-ops-conn repo)
|
||||
(let [current-checksum (client-op/get-local-checksum repo)
|
||||
new-checksum (sync-checksum/update-checksum current-checksum tx-report)]
|
||||
(when (and (exists? js/process)
|
||||
(= "1" (aget (.-env js/process) "LOGSEQ_CHECKSUM_ASSERT")))
|
||||
(let [recomputed-checksum (sync-checksum/recompute-checksum (:db-after tx-report))]
|
||||
(when-not (= new-checksum recomputed-checksum)
|
||||
(let [{:keys [tx-meta tx-data]} tx-report]
|
||||
(log/error :db-sync/checksum-incremental-drift
|
||||
{:repo repo
|
||||
:current-checksum current-checksum
|
||||
:incremental-checksum new-checksum
|
||||
:recomputed-checksum recomputed-checksum
|
||||
:tx-meta tx-meta
|
||||
:tx-count (count tx-data)
|
||||
:tx-sample (take 30 tx-data)})
|
||||
(throw (ex-info "Incremental checksum drift"
|
||||
{:repo repo
|
||||
:current-checksum current-checksum
|
||||
:incremental-checksum new-checksum
|
||||
:recomputed-checksum recomputed-checksum
|
||||
:tx-meta tx-meta
|
||||
:tx-count (count tx-data)}))))))
|
||||
(client-op/update-local-checksum repo new-checksum))))
|
||||
|
||||
(defn- broadcast-rtc-state!
|
||||
[client]
|
||||
(when client
|
||||
(shared-service/broadcast-to-clients!
|
||||
:rtc-sync-state
|
||||
(sync-presence/rtc-state-payload status client))))
|
||||
|
||||
(defn- set-ws-state!
|
||||
[client ws-state]
|
||||
(sync-presence/set-ws-state! broadcast-rtc-state! client ws-state))
|
||||
|
||||
(defn- update-online-users!
|
||||
[client users]
|
||||
(sync-presence/update-online-users! broadcast-rtc-state! client users))
|
||||
|
||||
(defn- clear-inflight!
|
||||
[client]
|
||||
(when-let [*inflight (:inflight client)]
|
||||
(reset! *inflight [])))
|
||||
|
||||
(defn- ws-base-url
|
||||
[]
|
||||
(sync-auth/ws-base-url @worker-state/*db-sync-config))
|
||||
|
||||
(def auth-token sync-util/auth-token)
|
||||
|
||||
(def id-token-expired? sync-auth/id-token-expired?)
|
||||
|
||||
(def <resolve-ws-token sync-auth/<resolve-ws-token)
|
||||
|
||||
(defn- ensure-client-graph-uuid!
|
||||
[repo graph-id]
|
||||
(when (seq graph-id)
|
||||
(client-op/update-graph-uuid repo graph-id)))
|
||||
|
||||
(defn- client-op-ready?
|
||||
[repo]
|
||||
(and (some? (worker-state/get-client-ops-conn repo))
|
||||
(integer? (client-op/get-local-tx repo))))
|
||||
|
||||
(defn- reconnect-delay-ms
|
||||
[attempt]
|
||||
(sync-transport/reconnect-delay-ms
|
||||
attempt
|
||||
{:base-delay-ms reconnect-base-delay-ms
|
||||
:max-delay-ms reconnect-max-delay-ms
|
||||
:jitter-ms reconnect-jitter-ms}))
|
||||
|
||||
(defn- clear-reconnect-timer!
|
||||
[reconnect]
|
||||
(when-let [timer (:timer @reconnect)]
|
||||
(js/clearTimeout timer)
|
||||
(swap! reconnect assoc :timer nil)))
|
||||
|
||||
(defn- reset-reconnect!
|
||||
[client]
|
||||
(when-let [reconnect (:reconnect client)]
|
||||
(clear-reconnect-timer! reconnect)
|
||||
(swap! reconnect assoc :attempt 0)))
|
||||
|
||||
(defn- clear-stale-ws-loop-timer!
|
||||
[client]
|
||||
(when-let [*timer (:stale-kill-timer client)]
|
||||
(when-let [timer @*timer]
|
||||
(js/clearInterval timer)
|
||||
(reset! *timer nil))))
|
||||
|
||||
(defn- touch-last-ws-message!
|
||||
[client]
|
||||
(when-let [*ts (:last-ws-message-ts client)]
|
||||
(reset! *ts (common-util/time-ms))))
|
||||
|
||||
(defn- ready-state
|
||||
[ws]
|
||||
(sync-transport/ready-state ws))
|
||||
|
||||
(defn- ws-open?
|
||||
[ws]
|
||||
(sync-transport/ws-open? ws))
|
||||
|
||||
(defn- send!
|
||||
[ws message]
|
||||
(sync-transport/send! sync-transport/coerce-ws-client-message ws message))
|
||||
|
||||
(defn- enqueue-receive-message!
|
||||
[client task]
|
||||
(if-let [queue (:receive-queue client)]
|
||||
(swap! queue
|
||||
(fn [prev]
|
||||
(-> (or prev (p/resolved nil))
|
||||
;; Keep queue alive even if one message handler fails.
|
||||
(p/catch (fn [_] nil))
|
||||
(p/then (fn [_] (task)))
|
||||
(p/catch (fn [error]
|
||||
(sync-util/set-last-sync-error! client error)
|
||||
(log/error :db-sync/ws-handle-message-failed
|
||||
{:repo (:repo client)
|
||||
:error error}))))))
|
||||
(task)))
|
||||
|
||||
(defn update-presence!
|
||||
[editing-block-uuid]
|
||||
(when-let [client @worker-state/*db-sync-client]
|
||||
(when-let [ws (:ws client)]
|
||||
(send! ws {:type "presence"
|
||||
:editing-block-uuid editing-block-uuid}))))
|
||||
|
||||
(defn- enqueue-asset-task!
|
||||
[client task]
|
||||
(when-let [queue (:asset-queue client)]
|
||||
(swap! queue
|
||||
(fn [prev]
|
||||
(p/then prev (fn [_] (task)))))))
|
||||
|
||||
(defn- ensure-client-state!
|
||||
[repo]
|
||||
{:repo repo
|
||||
:send-queue (atom (p/resolved nil))
|
||||
:receive-queue (atom (p/resolved nil))
|
||||
:asset-queue (atom (p/resolved nil))
|
||||
:pending-pull-since (atom nil)
|
||||
:inflight (atom [])
|
||||
:upload-request (atom nil)
|
||||
:last-sync-error (atom nil)
|
||||
:reconnect (atom {:attempt 0 :timer nil})
|
||||
:stale-kill-timer (atom nil)
|
||||
:last-ws-message-ts (atom (common-util/time-ms))
|
||||
:online-users (atom [])
|
||||
:ws-state (atom :closed)})
|
||||
|
||||
(declare connect!)
|
||||
|
||||
(defn- schedule-reconnect!
|
||||
[repo client url reason]
|
||||
(when-let [reconnect (:reconnect client)]
|
||||
(let [{:keys [attempt timer]} @reconnect]
|
||||
(when (nil? timer)
|
||||
(let [delay (reconnect-delay-ms attempt)
|
||||
timeout-id (js/setTimeout
|
||||
(fn []
|
||||
(log/info :db-sync/ws-reconnect {:repo repo
|
||||
:db-sync-client-exists? (some? @worker-state/*db-sync-client)})
|
||||
(swap! reconnect assoc :timer nil)
|
||||
(when-let [current @worker-state/*db-sync-client]
|
||||
(when (and (= (:repo current) repo)
|
||||
(= (:graph-id current) (:graph-id client)))
|
||||
(-> (p/let [token (<resolve-ws-token)
|
||||
updated (connect! repo current url token)]
|
||||
(reset! worker-state/*db-sync-client updated))
|
||||
(p/catch (fn [error]
|
||||
(log/error :db-sync/ws-reconnect-failed {:repo repo :error error})
|
||||
(schedule-reconnect! repo current url :connect-failed)))))))
|
||||
delay)]
|
||||
(swap! reconnect assoc :timer timeout-id :attempt (inc attempt))
|
||||
(log/info :db-sync/ws-reconnect-scheduled
|
||||
{:repo repo :delay delay :attempt attempt :reason reason}))))))
|
||||
|
||||
(defn- attach-ws-handlers!
|
||||
[repo client ws url]
|
||||
(set! (.-onmessage ws)
|
||||
(fn [event]
|
||||
(touch-last-ws-message! client)
|
||||
(enqueue-receive-message! client
|
||||
(fn []
|
||||
(sync-handle-message/handle-message! repo client (.-data event))))))
|
||||
(set! (.-onerror ws) (fn [error] (log/error :db-sync/ws-error error)))
|
||||
(set! (.-onclose ws)
|
||||
(fn [_]
|
||||
(log/info :db-sync/ws-closed {:repo repo})
|
||||
(clear-stale-ws-loop-timer! client)
|
||||
(clear-inflight! client)
|
||||
(update-online-users! client [])
|
||||
(set-ws-state! client :closed)
|
||||
(schedule-reconnect! repo client url :close))))
|
||||
|
||||
(defn- detach-ws-handlers!
|
||||
[ws]
|
||||
(set! (.-onopen ws) nil)
|
||||
(set! (.-onmessage ws) nil)
|
||||
(set! (.-onerror ws) nil)
|
||||
(set! (.-onclose ws) nil))
|
||||
|
||||
(defn- close-stale-ws-loop
|
||||
[client ws url]
|
||||
(let [repo (:repo client)
|
||||
graph-id (:graph-id client)]
|
||||
(clear-stale-ws-loop-timer! client)
|
||||
(when-let [*timer (:stale-kill-timer client)]
|
||||
(let [timer (js/setInterval
|
||||
(fn []
|
||||
(when-let [current @worker-state/*db-sync-client]
|
||||
(when (and (= repo (:repo current))
|
||||
(= graph-id (:graph-id current))
|
||||
(identical? ws (:ws current)))
|
||||
(cond
|
||||
(ws-open? ws)
|
||||
(let [now (common-util/time-ms)
|
||||
last-ts (or (some-> (:last-ws-message-ts current) deref) now)
|
||||
stale-ms (- now last-ts)]
|
||||
(when (>= stale-ms ws-stale-timeout-ms)
|
||||
(log/warn :db-sync/ws-stale-timeout {:repo repo :stale-ms stale-ms})
|
||||
(try (.close ws) (catch :default _ nil))))
|
||||
|
||||
(contains? #{2 3} (ready-state ws))
|
||||
(do
|
||||
(log/warn :db-sync/ws-stale-closed {:repo repo :ready-state (ready-state ws)})
|
||||
(clear-stale-ws-loop-timer! current)
|
||||
(clear-inflight! current)
|
||||
(update-online-users! current [])
|
||||
(set-ws-state! current :closed)
|
||||
(schedule-reconnect! repo current url :stale-closed))))))
|
||||
ws-stale-kill-interval-ms)]
|
||||
(reset! *timer timer))))
|
||||
client)
|
||||
|
||||
(defn- stop-client!
|
||||
[client]
|
||||
(clear-stale-ws-loop-timer! client)
|
||||
(sync-apply/clear-upload-response-timeout! client)
|
||||
(when-let [reconnect (:reconnect client)]
|
||||
(clear-reconnect-timer! reconnect))
|
||||
(when-let [ws (:ws client)]
|
||||
(detach-ws-handlers! ws)
|
||||
(update-online-users! client [])
|
||||
(set-ws-state! client :closed)
|
||||
(try (.close ws) (catch :default _ nil))))
|
||||
|
||||
(defn- active-client-for?
|
||||
[client repo graph-id]
|
||||
(when (and client (= repo (:repo client)) (= graph-id (:graph-id client)))
|
||||
(let [ws (:ws client)
|
||||
ws-ready-state (when ws (ready-state ws))]
|
||||
(contains? #{0 1} ws-ready-state))))
|
||||
|
||||
(defn- connect!
|
||||
[repo client url token]
|
||||
(when (:ws client)
|
||||
(stop-client! client))
|
||||
(log/info :db-sync/connect! {:repo repo
|
||||
:token-exists? (some? (or token (auth-token)))})
|
||||
(when-let [token' (or token (auth-token))]
|
||||
(let [ws (platform/websocket-connect (platform/current) (sync-transport/append-token url token'))
|
||||
updated (assoc client :ws ws)]
|
||||
(attach-ws-handlers! repo updated ws url)
|
||||
(set! (.-onopen ws)
|
||||
(fn [_]
|
||||
(reset-reconnect! updated)
|
||||
(touch-last-ws-message! updated)
|
||||
(set-ws-state! updated :open)
|
||||
(sync-util/clear-last-sync-error! updated)
|
||||
(send! ws {:type "hello" :client repo})
|
||||
(sync-assets/enqueue-asset-sync!
|
||||
repo updated
|
||||
{:enqueue-asset-task-f enqueue-asset-task!
|
||||
:current-client-f current-client
|
||||
:broadcast-rtc-state!-f broadcast-rtc-state!
|
||||
:fail-fast-f fail-fast})))
|
||||
(close-stale-ws-loop updated ws url))))
|
||||
|
||||
(defn stop!
|
||||
[]
|
||||
(when-let [client @worker-state/*db-sync-client]
|
||||
(stop-client! client)
|
||||
(reset! worker-state/*db-sync-client nil))
|
||||
(p/resolved nil))
|
||||
|
||||
(declare list-remote-graphs!)
|
||||
|
||||
(defn- <resolve-start-graph-id
|
||||
[repo]
|
||||
(if-let [graph-id (sync-util/get-graph-id repo)]
|
||||
(p/resolved graph-id)
|
||||
(let [target-graph-name (some-> repo common-config/strip-leading-db-version-prefix)]
|
||||
(if-not (seq target-graph-name)
|
||||
(p/resolved nil)
|
||||
(p/let [remote-graphs (list-remote-graphs!)
|
||||
remote-graph-id (some (fn [{:keys [graph-name graph-id]}]
|
||||
(when (= target-graph-name graph-name)
|
||||
graph-id))
|
||||
remote-graphs)]
|
||||
(when (seq remote-graph-id)
|
||||
(ensure-client-graph-uuid! repo remote-graph-id)
|
||||
remote-graph-id))))))
|
||||
|
||||
(defn start!
|
||||
[repo]
|
||||
(let [base (ws-base-url)
|
||||
graph-id (sync-util/get-graph-id repo)
|
||||
start-target [repo graph-id]
|
||||
inflight-target @*start-inflight-target
|
||||
current @worker-state/*db-sync-client]
|
||||
(if-not (and (string? base) (seq base))
|
||||
(do
|
||||
(log/info :db-sync/start-skipped {:repo repo :graph-id graph-id :base base})
|
||||
(p/resolved nil))
|
||||
(p/let [graph-id (<resolve-start-graph-id repo)]
|
||||
(cond
|
||||
(not (seq graph-id))
|
||||
(do
|
||||
(log/info :db-sync/start-skipped {:repo repo :graph-id graph-id :base base})
|
||||
(p/resolved nil))
|
||||
|
||||
(not (client-op-ready? repo))
|
||||
(do
|
||||
(log/info :db-sync/start-skipped {:repo repo :graph-id graph-id :base base :reason :client-op-not-ready})
|
||||
(p/resolved nil))
|
||||
|
||||
(= start-target inflight-target)
|
||||
(p/resolved nil)
|
||||
|
||||
(active-client-for? current repo graph-id)
|
||||
(do
|
||||
(broadcast-rtc-state! current)
|
||||
(sync-apply/enqueue-flush-pending! repo current)
|
||||
(p/resolved nil))
|
||||
|
||||
:else
|
||||
(do
|
||||
(reset! *start-inflight-target start-target)
|
||||
(->
|
||||
(p/do!
|
||||
(stop!)
|
||||
(p/let [client (ensure-client-state! repo)
|
||||
url (sync-transport/format-ws-url base graph-id)
|
||||
_ (ensure-client-graph-uuid! repo graph-id)
|
||||
connected (assoc client :graph-id graph-id)
|
||||
token (<resolve-ws-token)
|
||||
connected (connect! repo connected url token)]
|
||||
(reset! worker-state/*db-sync-client connected)
|
||||
nil))
|
||||
(p/finally
|
||||
(fn []
|
||||
(when (= start-target @*start-inflight-target)
|
||||
(reset! *start-inflight-target nil)))))))))))
|
||||
|
||||
(defn enqueue-local-tx!
|
||||
[repo tx-report]
|
||||
(sync-apply/enqueue-local-tx! repo tx-report))
|
||||
|
||||
(defn handle-local-tx!
|
||||
[repo tx-report]
|
||||
(sync-apply/handle-local-tx! repo tx-report))
|
||||
|
||||
(defn request-asset-download!
|
||||
[repo asset-uuid]
|
||||
(sync-apply/request-asset-download! repo asset-uuid))
|
||||
|
||||
(defn download-missing-assets!
|
||||
[repo graph-id]
|
||||
(sync-assets/download-missing-remote-assets! repo graph-id))
|
||||
|
||||
(defn retry-asset-upload!
|
||||
[repo]
|
||||
(when-let [client (current-client repo)]
|
||||
(sync-assets/enqueue-asset-sync!
|
||||
repo client
|
||||
{:enqueue-asset-task-f enqueue-asset-task!
|
||||
:current-client-f current-client
|
||||
:broadcast-rtc-state!-f broadcast-rtc-state!
|
||||
:fail-fast-f fail-fast}))
|
||||
(p/resolved nil))
|
||||
|
||||
(defn rehydrate-large-titles-from-db!
|
||||
[repo graph-id]
|
||||
(sync-apply/rehydrate-large-titles-from-db! repo graph-id))
|
||||
|
||||
(defn upload-graph!
|
||||
[repo]
|
||||
(sync-upload/upload-graph! repo))
|
||||
|
||||
(defn create-remote-graph!
|
||||
[repo opts]
|
||||
(sync-upload/create-remote-graph! repo opts))
|
||||
|
||||
(def list-remote-graphs! sync-upload/list-remote-graphs!)
|
||||
|
||||
(defn stop-upload!
|
||||
[repo]
|
||||
(sync-apply/set-upload-stopped! repo true))
|
||||
|
||||
(defn resume-upload!
|
||||
[repo]
|
||||
(sync-apply/set-upload-stopped! repo false))
|
||||
|
||||
(defn upload-stopped?
|
||||
[repo]
|
||||
(sync-apply/upload-stopped? repo))
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,25 +0,0 @@
|
||||
(ns frontend.worker.sync.asset-db-listener
|
||||
"Listen asset-block changes in db, generate asset-sync operations"
|
||||
(:require [datascript.core :as d]
|
||||
[frontend.worker.sync.client-op :as client-op]
|
||||
[logseq.db :as ldb]))
|
||||
|
||||
(defn- asset-checksum?
|
||||
[a]
|
||||
(= :logseq.property.asset/checksum a))
|
||||
|
||||
(defn- datom=>op
|
||||
[db-after [e _a _v t _]]
|
||||
(let [ent-after (d/entity db-after e)]
|
||||
(when (ldb/asset? ent-after)
|
||||
[:update-asset t {:block-uuid (:block/uuid ent-after)}])))
|
||||
|
||||
(defn generate-asset-ops
|
||||
[repo {:keys [tx-data _tx-meta _db-before db-after]}]
|
||||
(when (client-op/rtc-db-graph? repo)
|
||||
(let [related-datoms (filter
|
||||
(fn [datom]
|
||||
(and (asset-checksum? (:a datom)) (:added datom)))
|
||||
tx-data)]
|
||||
(when-let [ops (not-empty (map #(datom=>op db-after %) related-datoms))]
|
||||
(client-op/add-asset-ops repo ops)))))
|
||||
@@ -1,507 +0,0 @@
|
||||
(ns frontend.worker.sync.assets
|
||||
"Asset sync helpers for db sync."
|
||||
(:require
|
||||
[datascript.core :as d]
|
||||
[frontend.common.crypt :as crypt]
|
||||
[frontend.worker.platform :as platform]
|
||||
[frontend.worker.shared-service :as shared-service]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync.auth :as sync-auth]
|
||||
[frontend.worker.sync.client-op :as client-op]
|
||||
[frontend.worker.sync.crypt :as sync-crypt]
|
||||
[frontend.worker.sync.large-title :as sync-large-title]
|
||||
[lambdaisland.glogi :as log]
|
||||
[logseq.common.config :as common-config]
|
||||
[logseq.common.util :as common-util]
|
||||
[logseq.db :as ldb]
|
||||
[promesa.core :as p]))
|
||||
|
||||
(def max-asset-size (* 100 1024 1024))
|
||||
|
||||
(defonce *repo->missing-asset-upload-files
|
||||
(atom {}))
|
||||
|
||||
(def ^:private remote-asset-download-parallelism 10)
|
||||
|
||||
(defn graph-aes-key
|
||||
[repo graph-id fail-fast-f]
|
||||
(if (sync-crypt/graph-e2ee? repo)
|
||||
(p/let [aes-key (sync-crypt/<ensure-graph-aes-key repo graph-id)
|
||||
_ (when (nil? aes-key)
|
||||
(fail-fast-f :db-sync/missing-field {:repo repo :field :aes-key}))]
|
||||
aes-key)
|
||||
(p/resolved nil)))
|
||||
|
||||
(defn- asset-file-name
|
||||
[asset-uuid asset-type]
|
||||
(str asset-uuid "." asset-type))
|
||||
|
||||
(defn- ->uint8
|
||||
[payload]
|
||||
(cond
|
||||
(instance? js/Uint8Array payload)
|
||||
payload
|
||||
|
||||
(instance? js/ArrayBuffer payload)
|
||||
(js/Uint8Array. payload)
|
||||
|
||||
(and (exists? js/ArrayBuffer)
|
||||
(.isView js/ArrayBuffer payload))
|
||||
(js/Uint8Array. (.-buffer payload) (.-byteOffset payload) (.-byteLength payload))
|
||||
|
||||
(array? payload)
|
||||
(js/Uint8Array. payload)
|
||||
|
||||
(sequential? payload)
|
||||
(js/Uint8Array. (clj->js payload))
|
||||
|
||||
(and (object? payload)
|
||||
(= "Buffer" (aget payload "type"))
|
||||
(array? (aget payload "data")))
|
||||
(js/Uint8Array. (aget payload "data"))
|
||||
|
||||
:else
|
||||
(throw (ex-info "unsupported binary payload"
|
||||
{:payload-type (str (type payload))}))))
|
||||
|
||||
(defn- payload-size
|
||||
[payload]
|
||||
(cond
|
||||
(string? payload) (count payload)
|
||||
(some? (.-byteLength payload)) (.-byteLength payload)
|
||||
(some? (.-length payload)) (.-length payload)
|
||||
:else 0))
|
||||
|
||||
(defn- notify-asset-progress!
|
||||
[repo asset-id direction loaded total]
|
||||
(shared-service/broadcast-to-clients!
|
||||
:rtc-asset-upload-download-progress
|
||||
{:repo repo
|
||||
:asset-id asset-id
|
||||
:progress {:direction direction
|
||||
:loaded loaded
|
||||
:total total}}))
|
||||
|
||||
(defn- missing-asset-upload-file
|
||||
[asset-id asset-type]
|
||||
{:asset-id asset-id
|
||||
:asset-type asset-type
|
||||
:file (str common-config/local-assets-dir "/" (asset-file-name asset-id asset-type))})
|
||||
|
||||
(defn- mark-missing-asset-upload-file!
|
||||
[repo asset-id asset-type]
|
||||
(swap! *repo->missing-asset-upload-files
|
||||
update repo
|
||||
(fn [files]
|
||||
(assoc (or files {})
|
||||
asset-id
|
||||
(missing-asset-upload-file asset-id asset-type))))
|
||||
nil)
|
||||
|
||||
(defn- clear-missing-asset-upload-file!
|
||||
[repo asset-id]
|
||||
(swap! *repo->missing-asset-upload-files
|
||||
(fn [repo->files]
|
||||
(let [files (dissoc (get repo->files repo) asset-id)]
|
||||
(if (seq files)
|
||||
(assoc repo->files repo files)
|
||||
(dissoc repo->files repo)))))
|
||||
nil)
|
||||
|
||||
(defn get-missing-asset-upload-files
|
||||
[repo]
|
||||
(->> (vals (get @*repo->missing-asset-upload-files repo))
|
||||
(sort-by :file)
|
||||
vec))
|
||||
|
||||
(defn clear-missing-asset-upload-files!
|
||||
[repo]
|
||||
(swap! *repo->missing-asset-upload-files dissoc repo)
|
||||
nil)
|
||||
|
||||
(defn- mark-asset-write-finish!
|
||||
[repo asset-id]
|
||||
(shared-service/broadcast-to-clients!
|
||||
:asset-file-write-finish
|
||||
{:repo repo
|
||||
:asset-id asset-id
|
||||
:ts (common-util/time-ms)}))
|
||||
|
||||
(defn- <read-asset-bytes
|
||||
[repo asset-id asset-type]
|
||||
(platform/asset-read-bytes! (platform/current)
|
||||
repo
|
||||
(asset-file-name asset-id asset-type)))
|
||||
|
||||
(defn- <write-asset-bytes!
|
||||
[repo asset-id asset-type payload]
|
||||
(p/let [_ (platform/asset-write-bytes! (platform/current)
|
||||
repo
|
||||
(asset-file-name asset-id asset-type)
|
||||
payload)]
|
||||
(mark-asset-write-finish! repo asset-id)
|
||||
nil))
|
||||
|
||||
(defn upload-remote-asset!
|
||||
[repo graph-id asset-uuid asset-type checksum]
|
||||
(let [base (sync-auth/http-base-url @worker-state/*db-sync-config)]
|
||||
(if (and (seq base) (seq graph-id) (seq asset-type) (seq checksum))
|
||||
(-> (p/let [aes-key (graph-aes-key
|
||||
repo graph-id
|
||||
(fn [tag data]
|
||||
(throw (ex-info (name tag) data))) )
|
||||
asset-id (str asset-uuid)
|
||||
put-url (sync-large-title/asset-url base graph-id asset-id asset-type)
|
||||
asset-bytes (->
|
||||
(<read-asset-bytes repo asset-id asset-type)
|
||||
(p/catch (fn [e]
|
||||
(log/error :read-asset-failed e)
|
||||
(mark-missing-asset-upload-file! repo asset-id asset-type)
|
||||
(throw (ex-info "read-asset failed"
|
||||
{:type :rtc.exception/read-asset-failed}
|
||||
e)))))
|
||||
_ (clear-missing-asset-upload-file! repo asset-id)
|
||||
asset-bytes (if aes-key (->uint8 asset-bytes) asset-bytes)
|
||||
payload (if (not aes-key)
|
||||
asset-bytes
|
||||
(p/let [encrypted-bytes (crypt/<encrypt-uint8array aes-key asset-bytes)]
|
||||
(ldb/write-transit-str encrypted-bytes)))
|
||||
total (payload-size payload)
|
||||
_ (notify-asset-progress! repo asset-id :upload 0 total)
|
||||
headers (merge (sync-auth/auth-headers (worker-state/get-id-token))
|
||||
{"x-amz-meta-checksum" checksum
|
||||
"x-amz-meta-type" asset-type})
|
||||
^js resp (js/fetch put-url
|
||||
(clj->js {:method "PUT"
|
||||
:headers headers
|
||||
:body payload}))
|
||||
status (.-status resp)
|
||||
_ (notify-asset-progress! repo asset-id :upload total total)]
|
||||
(when-not (.-ok resp)
|
||||
(throw (ex-info "upload-asset failed"
|
||||
{:type :rtc.exception/upload-asset-failed
|
||||
:data {:status status}})))
|
||||
nil)
|
||||
(p/catch
|
||||
(fn [e]
|
||||
(if (contains? #{:rtc.exception/read-asset-failed
|
||||
:rtc.exception/upload-asset-failed}
|
||||
(:type (ex-data e)))
|
||||
(p/rejected e)
|
||||
(p/rejected (ex-info "upload-asset failed"
|
||||
{:type :rtc.exception/upload-asset-failed}
|
||||
e))))))
|
||||
(p/rejected (ex-info "missing asset upload info"
|
||||
{:repo repo
|
||||
:asset-uuid asset-uuid
|
||||
:asset-type asset-type
|
||||
:checksum checksum
|
||||
:base base
|
||||
:graph-id graph-id})))))
|
||||
|
||||
(defn- drop-asset-op!
|
||||
[repo asset-uuid reason data current-client-f broadcast-rtc-state!-f]
|
||||
(log/warn :db-sync/drop-asset-op
|
||||
(merge data
|
||||
{:repo repo
|
||||
:asset-uuid asset-uuid
|
||||
:reason reason}))
|
||||
(clear-missing-asset-upload-file! repo (str asset-uuid))
|
||||
(client-op/remove-asset-op repo asset-uuid)
|
||||
(when-let [client (current-client-f repo)]
|
||||
(broadcast-rtc-state!-f client))
|
||||
(p/resolved nil))
|
||||
|
||||
(defn process-asset-op!
|
||||
[repo graph-id asset-op {:keys [current-client-f broadcast-rtc-state!-f fail-fast-f]}]
|
||||
(let [asset-uuid (:block/uuid asset-op)
|
||||
op-type (cond
|
||||
(contains? asset-op :update-asset) :update-asset
|
||||
(contains? asset-op :remove-asset) :remove-asset
|
||||
:else :unknown)]
|
||||
(when-not asset-uuid
|
||||
(fail-fast-f :db-sync/missing-field {:repo repo :field :asset-uuid :op op-type}))
|
||||
(cond
|
||||
(contains? asset-op :update-asset)
|
||||
(if-let [conn (worker-state/get-datascript-conn repo)]
|
||||
(let [ent (d/entity @conn [:block/uuid asset-uuid])
|
||||
asset-type (:logseq.property.asset/type ent)
|
||||
checksum (:logseq.property.asset/checksum ent)
|
||||
size (:logseq.property.asset/size ent 0)]
|
||||
(cond
|
||||
(nil? ent)
|
||||
(drop-asset-op! repo asset-uuid :missing-asset-entity {:op :update-asset}
|
||||
current-client-f broadcast-rtc-state!-f)
|
||||
|
||||
(not (seq asset-type))
|
||||
(drop-asset-op! repo asset-uuid :missing-asset-type {:op :update-asset}
|
||||
current-client-f broadcast-rtc-state!-f)
|
||||
|
||||
(not (seq checksum))
|
||||
(drop-asset-op! repo asset-uuid :missing-checksum
|
||||
{:op :update-asset
|
||||
:asset-type asset-type}
|
||||
current-client-f broadcast-rtc-state!-f)
|
||||
|
||||
(> size max-asset-size)
|
||||
(do
|
||||
(log/info :db-sync/asset-too-large {:repo repo
|
||||
:asset-uuid asset-uuid
|
||||
:size size})
|
||||
(clear-missing-asset-upload-file! repo (str asset-uuid))
|
||||
(client-op/remove-asset-op repo asset-uuid)
|
||||
(when-let [client (current-client-f repo)]
|
||||
(broadcast-rtc-state!-f client))
|
||||
(p/resolved nil))
|
||||
|
||||
:else
|
||||
(-> (upload-remote-asset! repo graph-id asset-uuid asset-type checksum)
|
||||
(p/then (fn [_]
|
||||
(when (d/entity @conn [:block/uuid asset-uuid])
|
||||
(ldb/transact!
|
||||
conn
|
||||
[{:block/uuid asset-uuid
|
||||
:logseq.property.asset/remote-metadata {:checksum checksum :type asset-type}}]
|
||||
{:persist-op? true}))
|
||||
(client-op/remove-asset-op repo asset-uuid)
|
||||
(when-let [client (current-client-f repo)]
|
||||
(broadcast-rtc-state!-f client))))
|
||||
(p/catch (fn [e]
|
||||
(case (:type (ex-data e))
|
||||
:rtc.exception/read-asset-failed
|
||||
(when-let [client (current-client-f repo)]
|
||||
(broadcast-rtc-state!-f client))
|
||||
|
||||
:rtc.exception/upload-asset-failed
|
||||
nil
|
||||
|
||||
(log/error :db-sync/asset-upload-failed
|
||||
{:repo repo
|
||||
:asset-uuid asset-uuid
|
||||
:error e})))))))
|
||||
(fail-fast-f :db-sync/missing-db {:repo repo :op :process-asset-op}))
|
||||
|
||||
(contains? asset-op :remove-asset)
|
||||
(do
|
||||
(clear-missing-asset-upload-file! repo (str asset-uuid))
|
||||
(-> (client-op/remove-asset-op repo asset-uuid)
|
||||
(p/then (fn [_]
|
||||
(when-let [client (current-client-f repo)]
|
||||
(broadcast-rtc-state!-f client))))
|
||||
(p/catch (fn [e]
|
||||
(log/error :db-sync/asset-delete-failed
|
||||
{:repo repo
|
||||
:asset-uuid asset-uuid
|
||||
:error e})))))
|
||||
|
||||
:else
|
||||
(p/resolved nil))))
|
||||
|
||||
(defn process-asset-ops!
|
||||
[repo client {:keys [current-client-f broadcast-rtc-state!-f fail-fast-f]}]
|
||||
(let [graph-id (:graph-id client)
|
||||
asset-ops (not-empty (client-op/get-all-asset-ops repo))
|
||||
parallelism 10]
|
||||
(if (and (seq graph-id) asset-ops)
|
||||
(let [queue (atom (vec asset-ops))
|
||||
pop-queue! (fn []
|
||||
(let [selected (atom nil)]
|
||||
(swap! queue
|
||||
(fn [q]
|
||||
(if (seq q)
|
||||
(do
|
||||
(reset! selected (first q))
|
||||
(subvec q 1))
|
||||
q)))
|
||||
@selected))
|
||||
worker (fn worker []
|
||||
(if-let [asset-op (pop-queue!)]
|
||||
(-> (p/let [_ (process-asset-op! repo graph-id asset-op
|
||||
{:current-client-f current-client-f
|
||||
:broadcast-rtc-state!-f broadcast-rtc-state!-f
|
||||
:fail-fast-f fail-fast-f})]
|
||||
(worker))
|
||||
(p/catch (fn [e]
|
||||
(log/error :db-sync/process-asset-op-loop-failed
|
||||
{:repo repo
|
||||
:asset-op asset-op
|
||||
:error e}))))
|
||||
(p/resolved nil)))]
|
||||
(p/all (mapv (fn [_] (worker))
|
||||
(range (min parallelism (count asset-ops))))))
|
||||
(p/resolved nil))))
|
||||
|
||||
(defn enqueue-asset-sync!
|
||||
[repo client {:keys [enqueue-asset-task-f current-client-f broadcast-rtc-state!-f fail-fast-f]}]
|
||||
(enqueue-asset-task-f
|
||||
client
|
||||
#(process-asset-ops!
|
||||
repo client
|
||||
{:current-client-f current-client-f
|
||||
:broadcast-rtc-state!-f broadcast-rtc-state!-f
|
||||
:fail-fast-f fail-fast-f})))
|
||||
|
||||
(defn- parse-content-length
|
||||
[^js resp]
|
||||
(when-let [content-length (some-> (.-headers resp) (.get "content-length"))]
|
||||
(let [length (js/parseInt content-length 10)]
|
||||
(when (not (js/isNaN length))
|
||||
length))))
|
||||
|
||||
(defn download-remote-asset!
|
||||
[repo graph-id asset-uuid asset-type]
|
||||
(let [base (sync-auth/http-base-url @worker-state/*db-sync-config)]
|
||||
(if (and (seq base) (seq graph-id) (seq asset-type))
|
||||
(-> (p/let [aes-key (graph-aes-key
|
||||
repo graph-id
|
||||
(fn [tag data]
|
||||
(throw (ex-info (name tag) data))))
|
||||
asset-id (str asset-uuid)
|
||||
get-url (sync-large-title/asset-url base graph-id asset-id asset-type)
|
||||
headers (sync-auth/auth-headers (worker-state/get-id-token))
|
||||
request-opts (cond-> {:method "GET"}
|
||||
(seq headers) (assoc :headers headers))
|
||||
^js resp (js/fetch get-url
|
||||
(clj->js request-opts))
|
||||
status (.-status resp)
|
||||
_ (when-not (.-ok resp)
|
||||
(throw (ex-info "download asset failed"
|
||||
{:type :rtc.exception/download-asset-failed
|
||||
:data {:status status}})))
|
||||
total (or (parse-content-length resp) 0)
|
||||
_ (notify-asset-progress! repo asset-id :download 0 total)
|
||||
body (.arrayBuffer resp)
|
||||
body-size (.-byteLength body)
|
||||
total' (if (pos? total) total body-size)
|
||||
_ (notify-asset-progress! repo asset-id :download body-size total')
|
||||
asset-file
|
||||
(if (not aes-key)
|
||||
body
|
||||
(let [asset-file-untransited (ldb/read-transit-str (.decode (js/TextDecoder.) body))]
|
||||
(crypt/<decrypt-uint8array aes-key asset-file-untransited)))]
|
||||
(<write-asset-bytes! repo asset-id asset-type asset-file))
|
||||
(p/catch
|
||||
(fn [e]
|
||||
(if (= :rtc.exception/download-asset-failed (:type (ex-data e)))
|
||||
(p/rejected e)
|
||||
(p/rejected (ex-info "download asset failed"
|
||||
{:type :rtc.exception/download-asset-failed}
|
||||
e))))))
|
||||
(p/rejected (ex-info "missing asset download info"
|
||||
{:repo repo
|
||||
:asset-uuid asset-uuid
|
||||
:asset-type asset-type
|
||||
:base base
|
||||
:graph-id graph-id})))))
|
||||
|
||||
(defn log-request-asset-download-failed!
|
||||
[repo asset-uuid error]
|
||||
(log/error :db-sync/request-asset-download-failed
|
||||
{:repo repo
|
||||
:asset-uuid asset-uuid
|
||||
:error error}))
|
||||
|
||||
(defn request-asset-download!
|
||||
[repo asset-uuid {:keys [current-client-f enqueue-asset-task-f broadcast-rtc-state!-f]}]
|
||||
(when-let [client (current-client-f repo)]
|
||||
(when-let [graph-id (:graph-id client)]
|
||||
(enqueue-asset-task-f
|
||||
client
|
||||
#(when-let [conn (worker-state/get-datascript-conn repo)]
|
||||
(when-let [ent (d/entity @conn [:block/uuid asset-uuid])]
|
||||
(let [asset-type (:logseq.property.asset/type ent)
|
||||
asset-id (str asset-uuid)
|
||||
should-download? (and (seq asset-type)
|
||||
(:logseq.property.asset/remote-metadata ent))]
|
||||
(-> (p/let [meta (when should-download?
|
||||
(platform/asset-stat (platform/current)
|
||||
repo
|
||||
(asset-file-name asset-id asset-type)))
|
||||
missing-local? (and should-download? (nil? meta))
|
||||
_ (when missing-local?
|
||||
(download-remote-asset! repo graph-id asset-uuid asset-type))
|
||||
_ (when missing-local?
|
||||
(client-op/remove-asset-op repo asset-uuid))
|
||||
_ (when missing-local?
|
||||
(broadcast-rtc-state!-f client))]
|
||||
nil)
|
||||
(p/catch (fn [e]
|
||||
(log-request-asset-download-failed! repo asset-uuid e)
|
||||
(p/rejected e)))))))))))
|
||||
|
||||
(defn- remote-asset-download-candidates
|
||||
[db]
|
||||
(->> (d/q '[:find ?e ?asset-uuid ?asset-type
|
||||
:where
|
||||
[?asset-class :db/ident :logseq.class/Asset]
|
||||
[?e :block/tags ?asset-class]
|
||||
[?e :block/uuid ?asset-uuid]
|
||||
[?e :logseq.property.asset/type ?asset-type]
|
||||
[?e :logseq.property.asset/remote-metadata]]
|
||||
db)
|
||||
(keep (fn [[e asset-uuid asset-type]]
|
||||
(let [ent (d/entity db e)]
|
||||
(when-not (seq (some-> (:logseq.property.asset/external-url ent) str))
|
||||
{:asset-uuid asset-uuid
|
||||
:asset-type asset-type}))))
|
||||
(sort-by (comp str :asset-uuid))))
|
||||
|
||||
(defn download-remote-assets-if-missing!
|
||||
[repo graph-id candidates]
|
||||
(let [candidates (->> candidates
|
||||
(filter (fn [{:keys [asset-uuid asset-type]}]
|
||||
(and asset-uuid (seq asset-type))))
|
||||
distinct
|
||||
(sort-by (juxt (comp str :asset-uuid) :asset-type)))
|
||||
current-platform (platform/current)
|
||||
queue (atom (vec candidates))
|
||||
result (atom {:total (count candidates)
|
||||
:downloaded 0
|
||||
:skipped-existing 0})
|
||||
pop-queue! (fn []
|
||||
(let [selected (atom nil)]
|
||||
(swap! queue
|
||||
(fn [q]
|
||||
(if (seq q)
|
||||
(do
|
||||
(reset! selected (first q))
|
||||
(subvec q 1))
|
||||
q)))
|
||||
@selected))
|
||||
worker (fn worker []
|
||||
(if-let [{:keys [asset-uuid asset-type]} (pop-queue!)]
|
||||
(let [asset-id (str asset-uuid)]
|
||||
(p/let [meta (platform/asset-stat current-platform
|
||||
repo
|
||||
(asset-file-name asset-id asset-type))
|
||||
_ (if meta
|
||||
(do
|
||||
(swap! result update :skipped-existing inc)
|
||||
nil)
|
||||
(p/let [_ (download-remote-asset! repo graph-id asset-uuid asset-type)]
|
||||
(swap! result update :downloaded inc)))]
|
||||
(worker)))
|
||||
(p/resolved nil)))]
|
||||
(p/let [_ (p/all (mapv (fn [_] (worker))
|
||||
(range (min remote-asset-download-parallelism
|
||||
(count candidates)))))]
|
||||
@result)))
|
||||
|
||||
(defn remote-asset-download-candidates-in-tx
|
||||
[db tx-data]
|
||||
(->> tx-data
|
||||
(keep (fn [d]
|
||||
(when (and (= (:a d) :logseq.property.asset/remote-metadata) (:added d))
|
||||
(when-let [asset (d/entity db (:e d))]
|
||||
{:asset-uuid (:block/uuid asset)
|
||||
:asset-type (:logseq.property.asset/type asset)}))))
|
||||
distinct))
|
||||
|
||||
(defn download-missing-remote-assets!
|
||||
[repo graph-id]
|
||||
(if-let [conn (worker-state/get-datascript-conn repo)]
|
||||
(download-remote-assets-if-missing!
|
||||
repo graph-id (remote-asset-download-candidates @conn))
|
||||
(p/rejected (ex-info "datascript connection not found"
|
||||
{:repo repo
|
||||
:graph-id graph-id}))))
|
||||
@@ -1,109 +0,0 @@
|
||||
(ns frontend.worker.sync.auth
|
||||
"Auth and endpoint helpers for db sync."
|
||||
(:require [clojure.string :as string]
|
||||
[frontend.worker-common.util :as worker-util]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync.util :as sync-util]
|
||||
[logseq.common.util :as common-util]
|
||||
[promesa.core :as p]))
|
||||
|
||||
(defn ws-base-url
|
||||
[db-sync-config]
|
||||
(:ws-url db-sync-config))
|
||||
|
||||
(defn http-base-url
|
||||
[db-sync-config]
|
||||
(or (:http-base db-sync-config)
|
||||
(when-let [ws-url (ws-base-url db-sync-config)]
|
||||
(let [base (cond
|
||||
(string/starts-with? ws-url "wss://")
|
||||
(str "https://" (subs ws-url (count "wss://")))
|
||||
|
||||
(string/starts-with? ws-url "ws://")
|
||||
(str "http://" (subs ws-url (count "ws://")))
|
||||
|
||||
:else ws-url)]
|
||||
(string/replace base #"/sync/%s$" "")))))
|
||||
|
||||
(defn id-token-expired?
|
||||
[token]
|
||||
(if-not (string? token)
|
||||
true
|
||||
(try
|
||||
(let [exp-ms (some-> token worker-util/parse-jwt :exp (* 1000))]
|
||||
(or (not (number? exp-ms))
|
||||
(<= exp-ms (common-util/time-ms))))
|
||||
(catch :default _
|
||||
true))))
|
||||
|
||||
(defn oauth-token-url
|
||||
[state]
|
||||
(or (:auth/oauth-token-url state)
|
||||
(when-let [domain (not-empty (:auth/oauth-domain state))]
|
||||
(str "https://" domain "/oauth2/token"))))
|
||||
|
||||
(defn <refresh-id&access-token
|
||||
[]
|
||||
(let [state @worker-state/*state
|
||||
refresh-token (:auth/refresh-token state)
|
||||
token-url (oauth-token-url state)
|
||||
oauth-client-id (:auth/oauth-client-id state)]
|
||||
(when-not (seq refresh-token)
|
||||
(throw (ex-info "worker auth refresh requires refresh token"
|
||||
{:code :missing-refresh-token})))
|
||||
(when-not (seq token-url)
|
||||
(throw (ex-info "worker auth refresh requires oauth token url"
|
||||
{:code :missing-oauth-token-url})))
|
||||
(when-not (seq oauth-client-id)
|
||||
(throw (ex-info "worker auth refresh requires oauth client id"
|
||||
{:code :missing-oauth-client-id})))
|
||||
(let [form-data (js/URLSearchParams.)]
|
||||
(.set form-data "grant_type" "refresh_token")
|
||||
(.set form-data "client_id" oauth-client-id)
|
||||
(.set form-data "refresh_token" refresh-token)
|
||||
(p/let [resp (js/fetch token-url #js {:method "POST"
|
||||
:headers #js {"content-type" "application/x-www-form-urlencoded"}
|
||||
:body (.toString form-data)})
|
||||
text (.text resp)
|
||||
data (when (seq text)
|
||||
(js->clj (js/JSON.parse text) :keywordize-keys true))]
|
||||
(if (.-ok resp)
|
||||
{:id-token (:id_token data)
|
||||
:access-token (:access_token data)}
|
||||
(throw (ex-info "worker auth refresh failed"
|
||||
{:code :auth-refresh-failed
|
||||
:status (.-status resp)
|
||||
:token-url token-url
|
||||
:body data})))))))
|
||||
|
||||
(defn <resolve-ws-token
|
||||
[]
|
||||
(let [token (sync-util/auth-token)
|
||||
token-expired? (id-token-expired? token)]
|
||||
(if (and (not (sync-util/cli-node-owner?)) token-expired?)
|
||||
(p/let [{:keys [id-token access-token]} (<refresh-id&access-token)]
|
||||
(when-not (seq id-token)
|
||||
(throw (ex-info "worker auth refresh returned empty id-token"
|
||||
{:code :auth-refresh-empty-id-token})))
|
||||
(worker-state/set-new-state!
|
||||
(cond-> {:auth/id-token id-token}
|
||||
(seq access-token) (assoc :auth/access-token access-token)))
|
||||
id-token)
|
||||
(p/resolved token))))
|
||||
|
||||
(defn get-user-uuid
|
||||
[id-token]
|
||||
(some-> id-token
|
||||
worker-util/parse-jwt
|
||||
:sub))
|
||||
|
||||
(defn auth-headers
|
||||
[token]
|
||||
(when-let [token* token]
|
||||
{"authorization" (str "Bearer " token*)}))
|
||||
|
||||
(defn with-auth-headers
|
||||
[auth-headers-f opts]
|
||||
(if-let [auth (auth-headers-f)]
|
||||
(assoc opts :headers (merge (or (:headers opts) {}) auth))
|
||||
opts))
|
||||
@@ -1,670 +0,0 @@
|
||||
(ns frontend.worker.sync.client-op
|
||||
"Store client sync metadata and ops in sqlite tables.
|
||||
DataScript client-op storage is deprecated and unsupported."
|
||||
(:require [clojure.string :as string]
|
||||
[datascript.core :as d]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[goog.object :as gobj]
|
||||
[lambdaisland.glogi :as log]
|
||||
[logseq.db.sqlite.util :as sqlite-util]
|
||||
[malli.core :as ma]
|
||||
[malli.transform :as mt]))
|
||||
|
||||
(def op-schema
|
||||
[:multi {:dispatch first}
|
||||
[:update-asset
|
||||
[:catn
|
||||
[:op :keyword]
|
||||
[:t :int]
|
||||
[:value [:map
|
||||
[:block-uuid :uuid]]]]]
|
||||
[:remove-asset
|
||||
[:catn
|
||||
[:op :keyword]
|
||||
[:t :int]
|
||||
[:value [:map
|
||||
[:block-uuid :uuid]]]]]])
|
||||
|
||||
(def ops-schema [:sequential op-schema])
|
||||
(def ops-coercer (ma/coercer ops-schema mt/json-transformer nil
|
||||
#(do (log/error ::bad-ops (:value %))
|
||||
(ma/-fail! ::ops-schema (select-keys % [:value])))))
|
||||
|
||||
(defonce *repo->pending-local-tx-count (atom {}))
|
||||
|
||||
(def ^:private sqlite-schema-ready-key "__logseq_client_ops_schema_ready_v2")
|
||||
(def ^:private sqlite-mode-key "__logseq_client_ops_sqlite_mode")
|
||||
(def ^:private sync-meta-table-sql
|
||||
"create table if not exists sync_meta (key text primary key, value text)")
|
||||
(def ^:private client-ops-table-sql
|
||||
(str "create table if not exists client_ops ("
|
||||
"id integer primary key autoincrement,"
|
||||
"kind text not null,"
|
||||
"created_at integer not null,"
|
||||
"tx_id text unique,"
|
||||
"pending integer not null default 0,"
|
||||
"failed integer not null default 0,"
|
||||
"outliner_op text,"
|
||||
"undo_redo text,"
|
||||
"forward_outliner_ops text,"
|
||||
"inverse_outliner_ops text,"
|
||||
"inferred_outliner_ops integer,"
|
||||
"normalized_tx_data text,"
|
||||
"reversed_tx_data text,"
|
||||
"asset_uuid text,"
|
||||
"asset_op text,"
|
||||
"asset_t integer,"
|
||||
"asset_value text"
|
||||
")"))
|
||||
(def ^:private pending-index-sql
|
||||
"create index if not exists idx_client_ops_pending_created on client_ops(kind, pending, created_at, id)")
|
||||
(def ^:private asset-index-sql
|
||||
"create index if not exists idx_client_ops_asset_uuid on client_ops(kind, asset_uuid)")
|
||||
(def ^:private sync-conflicts-table-sql
|
||||
(str "create table if not exists sync_conflicts ("
|
||||
"id integer primary key autoincrement,"
|
||||
"block_uuid text not null,"
|
||||
"attr text not null,"
|
||||
"value text not null,"
|
||||
"remote_t integer,"
|
||||
"created_at integer not null,"
|
||||
"unique(block_uuid, attr, value)"
|
||||
")"))
|
||||
(def ^:private sync-conflicts-block-index-sql
|
||||
"create index if not exists idx_sync_conflicts_block_uuid on sync_conflicts(block_uuid, created_at)")
|
||||
|
||||
(defn- client-ops-store
|
||||
[repo]
|
||||
(worker-state/get-client-ops-conn repo))
|
||||
|
||||
(declare ensure-sqlite-schema!)
|
||||
|
||||
(defn- detect-sqlite-mode
|
||||
[^js db]
|
||||
(or (gobj/get db sqlite-mode-key)
|
||||
(let [mode
|
||||
(cond
|
||||
(not db) nil
|
||||
(fn? (gobj/get db "prepare"))
|
||||
(try
|
||||
(let [^js stmt (.prepare db "select 1")]
|
||||
(try
|
||||
(if (fn? (gobj/get stmt "run"))
|
||||
:better-sqlite
|
||||
:sqlite-wasm)
|
||||
(finally
|
||||
(when (fn? (gobj/get stmt "finalize"))
|
||||
(.finalize stmt)))))
|
||||
(catch :default _
|
||||
:sqlite-wasm))
|
||||
(fn? (gobj/get db "exec")) :sqlite-wasm
|
||||
:else nil)]
|
||||
(when mode
|
||||
(try
|
||||
(gobj/set db sqlite-mode-key mode)
|
||||
(catch :default _
|
||||
nil)))
|
||||
mode)))
|
||||
|
||||
(defn- sqlite-db?
|
||||
[conn]
|
||||
(some? (detect-sqlite-mode conn)))
|
||||
|
||||
(defn- sqlite-store-or-throw
|
||||
[repo]
|
||||
(when-let [store (client-ops-store repo)]
|
||||
(if (sqlite-db? store)
|
||||
(do
|
||||
(ensure-sqlite-schema! store)
|
||||
store)
|
||||
(throw (ex-info "Legacy DataScript client-op storage is unsupported. Please back up the graph and re-download it."
|
||||
{:type :db-sync/legacy-client-ops-storage
|
||||
:repo repo})))))
|
||||
|
||||
(defn- parse-uuid-str
|
||||
[v]
|
||||
(when (string? v)
|
||||
(try
|
||||
(uuid v)
|
||||
(catch :default _
|
||||
nil))))
|
||||
|
||||
(defn- kw->str
|
||||
[v]
|
||||
(cond
|
||||
(keyword? v) (name v)
|
||||
(string? v) v
|
||||
:else nil))
|
||||
|
||||
(defn- str->kw
|
||||
[v]
|
||||
(when (string? v)
|
||||
(keyword v)))
|
||||
|
||||
(defn- qualified-kw->str
|
||||
[v]
|
||||
(cond
|
||||
(qualified-keyword? v) (subs (str v) 1)
|
||||
(keyword? v) (name v)
|
||||
(string? v) v
|
||||
:else nil))
|
||||
|
||||
(defn- str->qualified-kw
|
||||
[v]
|
||||
(when (string? v)
|
||||
(keyword v)))
|
||||
|
||||
(defn- bool->int [v] (if v 1 0))
|
||||
(defn- int->bool [v] (not (or (nil? v) (= 0 v) (= false v))))
|
||||
|
||||
(defn- normalize-op-entries
|
||||
[ops]
|
||||
(let [ops' (some-> ops seq vec)]
|
||||
(cond
|
||||
(nil? ops')
|
||||
nil
|
||||
|
||||
(and (keyword? (first ops'))
|
||||
(vector? (second ops')))
|
||||
[ops']
|
||||
|
||||
:else
|
||||
ops')))
|
||||
|
||||
(defn- sqlite-run!
|
||||
[^js db sql params]
|
||||
(case (detect-sqlite-mode db)
|
||||
:better-sqlite
|
||||
(let [^js stmt (.prepare db sql)
|
||||
run-fn (gobj/get stmt "run")]
|
||||
(if (seq params)
|
||||
(.apply run-fn stmt (to-array params))
|
||||
(.call run-fn stmt)))
|
||||
|
||||
:sqlite-wasm
|
||||
(.exec db #js {:sql sql
|
||||
:bind (into-array params)})
|
||||
|
||||
nil))
|
||||
|
||||
(defn- sqlite-rows
|
||||
[^js db sql params]
|
||||
(case (detect-sqlite-mode db)
|
||||
:better-sqlite
|
||||
(let [^js stmt (.prepare db sql)
|
||||
all-fn (gobj/get stmt "all")]
|
||||
(vec (if (seq params)
|
||||
(.apply all-fn stmt (to-array params))
|
||||
(.call all-fn stmt))))
|
||||
|
||||
:sqlite-wasm
|
||||
(let [^js result (.exec db #js {:sql sql
|
||||
:bind (into-array params)
|
||||
:rowMode "object"
|
||||
:returnValue "resultRows"})]
|
||||
(cond
|
||||
(nil? result) []
|
||||
(array? result) (vec result)
|
||||
(fn? (gobj/get result "toArray")) (vec (.toArray result))
|
||||
:else []))
|
||||
|
||||
[]))
|
||||
|
||||
(defn- sqlite-row
|
||||
[db sql params]
|
||||
(first (sqlite-rows db sql params)))
|
||||
|
||||
(defn- sqlite-with-tx!
|
||||
[^js db f]
|
||||
(case (detect-sqlite-mode db)
|
||||
:better-sqlite
|
||||
(let [tx-fn (.transaction db (fn [] (f db)))]
|
||||
(tx-fn))
|
||||
|
||||
:sqlite-wasm
|
||||
(if (fn? (gobj/get db "transaction"))
|
||||
(.transaction db (fn [tx] (f tx)))
|
||||
(f db))
|
||||
|
||||
(f db)))
|
||||
|
||||
(defn ensure-sqlite-schema!
|
||||
[db]
|
||||
(when (sqlite-db? db)
|
||||
(when-not (true? (gobj/get db sqlite-schema-ready-key))
|
||||
(sqlite-with-tx!
|
||||
db
|
||||
(fn [tx]
|
||||
(sqlite-run! tx sync-meta-table-sql [])
|
||||
(sqlite-run! tx client-ops-table-sql [])
|
||||
(sqlite-run! tx sync-conflicts-table-sql [])
|
||||
(sqlite-run! tx pending-index-sql [])
|
||||
(sqlite-run! tx asset-index-sql [])
|
||||
(sqlite-run! tx sync-conflicts-block-index-sql [])))
|
||||
(try
|
||||
(gobj/set db sqlite-schema-ready-key true)
|
||||
(catch :default _
|
||||
nil)))))
|
||||
|
||||
(defn- sqlite-get-meta
|
||||
[db k]
|
||||
(some-> (sqlite-row db "select value from sync_meta where key = ?" [(name k)])
|
||||
(aget "value")))
|
||||
|
||||
(defn- sqlite-set-meta!
|
||||
[db k v]
|
||||
(sqlite-run! db
|
||||
(str "insert into sync_meta (key, value) values (?, ?)"
|
||||
" on conflict(key) do update set value = excluded.value")
|
||||
[(name k) (str v)]))
|
||||
|
||||
(defn update-graph-uuid
|
||||
[repo graph-uuid]
|
||||
{:pre [(some? graph-uuid)]}
|
||||
(when-let [store (sqlite-store-or-throw repo)]
|
||||
(sqlite-set-meta! store :graph-uuid graph-uuid)))
|
||||
|
||||
(defn get-graph-uuid
|
||||
[repo]
|
||||
(when-let [store (sqlite-store-or-throw repo)]
|
||||
(sqlite-get-meta store :graph-uuid)))
|
||||
|
||||
(defn get-local-tx
|
||||
[repo]
|
||||
(when-let [store (sqlite-store-or-throw repo)]
|
||||
(when-let [result (sqlite-get-meta store :local-tx)]
|
||||
(js/parseInt result 10))))
|
||||
|
||||
(defn update-local-tx
|
||||
[repo t]
|
||||
{:pre [(and (integer? t) (>= t 0))]}
|
||||
(let [store (sqlite-store-or-throw repo)
|
||||
prev-t (get-local-tx repo)]
|
||||
(assert (some? store) repo)
|
||||
(when (and prev-t (< t prev-t))
|
||||
(throw (ex-info "local-tx should be monotonically increasing"
|
||||
{:repo repo
|
||||
:prev-t prev-t
|
||||
:new-t t})))
|
||||
(sqlite-set-meta! store :local-tx t)))
|
||||
|
||||
(defn reset-local-tx
|
||||
"Should be used only when uploading a graph"
|
||||
[repo]
|
||||
(let [store (sqlite-store-or-throw repo)]
|
||||
(assert (some? store) repo)
|
||||
(sqlite-set-meta! store :local-tx 0)))
|
||||
|
||||
(defn update-local-checksum
|
||||
[repo checksum]
|
||||
{:pre [(some? checksum)]}
|
||||
(let [store (sqlite-store-or-throw repo)]
|
||||
(assert (some? store) repo)
|
||||
(sqlite-set-meta! store :db-sync/checksum checksum)))
|
||||
|
||||
(defn get-pending-local-tx-count
|
||||
[repo]
|
||||
(if-let [cached (get @*repo->pending-local-tx-count repo)]
|
||||
cached
|
||||
(let [count' (if-let [store (sqlite-store-or-throw repo)]
|
||||
(or (some-> (sqlite-row store
|
||||
"select count(*) as c from client_ops where kind = 'tx' and pending = 1"
|
||||
[])
|
||||
(aget "c"))
|
||||
0)
|
||||
0)]
|
||||
(swap! *repo->pending-local-tx-count assoc repo count')
|
||||
count')))
|
||||
|
||||
(defn adjust-pending-local-tx-count!
|
||||
[repo delta]
|
||||
(swap! *repo->pending-local-tx-count
|
||||
(fn [m]
|
||||
(let [base (or (get m repo) 0)
|
||||
next (max 0 (+ base delta))]
|
||||
(assoc m repo next)))))
|
||||
|
||||
(defn get-local-checksum
|
||||
[repo]
|
||||
(let [store (sqlite-store-or-throw repo)]
|
||||
(assert (some? store) repo)
|
||||
(sqlite-get-meta store :db-sync/checksum)))
|
||||
|
||||
(defn rtc-db-graph?
|
||||
"Is RTC enabled"
|
||||
[repo]
|
||||
(or (exists? js/process)
|
||||
(some? (get-graph-uuid repo))))
|
||||
|
||||
(defn- row->pending-local-tx
|
||||
[row]
|
||||
(let [tx-id (parse-uuid-str (aget row "tx_id"))]
|
||||
(when tx-id
|
||||
{:tx-id tx-id
|
||||
:outliner-op (str->kw (aget row "outliner_op"))
|
||||
:forward-outliner-ops (or (normalize-op-entries
|
||||
(sqlite-util/read-transit-str (aget row "forward_outliner_ops")))
|
||||
[])
|
||||
:inverse-outliner-ops (or (normalize-op-entries
|
||||
(sqlite-util/read-transit-str (aget row "inverse_outliner_ops")))
|
||||
[])
|
||||
:inferred-outliner-ops? (int->bool (aget row "inferred_outliner_ops"))
|
||||
:db-sync/undo-redo (str->kw (aget row "undo_redo"))
|
||||
:tx (sqlite-util/read-transit-str (aget row "normalized_tx_data"))
|
||||
:reversed-tx (sqlite-util/read-transit-str (aget row "reversed_tx_data"))})))
|
||||
|
||||
(defn upsert-local-tx-entry!
|
||||
[repo {:keys [tx-id created-at pending? failed? outliner-op undo-redo
|
||||
forward-outliner-ops inverse-outliner-ops inferred-outliner-ops?
|
||||
normalized-tx-data reversed-tx-data]
|
||||
:or {pending? true failed? false}}]
|
||||
{:pre [(some? tx-id)]}
|
||||
(let [store (sqlite-store-or-throw repo)]
|
||||
(assert (some? store) repo)
|
||||
(let [tx-id-str (str tx-id)
|
||||
existing (sqlite-row store
|
||||
"select pending, created_at from client_ops where kind = 'tx' and tx_id = ?"
|
||||
[tx-id-str])
|
||||
should-inc-pending? (not= 1 (some-> existing (aget "pending")))
|
||||
forward-outliner-ops' (or (normalize-op-entries forward-outliner-ops) [])
|
||||
inverse-outliner-ops' (or (normalize-op-entries inverse-outliner-ops) [])
|
||||
created-at' (or (some-> existing (aget "created_at"))
|
||||
created-at
|
||||
(.now js/Date))]
|
||||
(sqlite-run! store
|
||||
(str "insert into client_ops ("
|
||||
"kind, created_at, tx_id, pending, failed, outliner_op, undo_redo, "
|
||||
"forward_outliner_ops, inverse_outliner_ops, inferred_outliner_ops, "
|
||||
"normalized_tx_data, reversed_tx_data"
|
||||
") values ('tx', ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
|
||||
" on conflict(tx_id) do update set "
|
||||
"created_at = excluded.created_at,"
|
||||
"pending = excluded.pending,"
|
||||
"failed = excluded.failed,"
|
||||
"outliner_op = excluded.outliner_op,"
|
||||
"undo_redo = excluded.undo_redo,"
|
||||
"forward_outliner_ops = excluded.forward_outliner_ops,"
|
||||
"inverse_outliner_ops = excluded.inverse_outliner_ops,"
|
||||
"inferred_outliner_ops = excluded.inferred_outliner_ops,"
|
||||
"normalized_tx_data = excluded.normalized_tx_data,"
|
||||
"reversed_tx_data = excluded.reversed_tx_data")
|
||||
[created-at'
|
||||
tx-id-str
|
||||
(bool->int pending?)
|
||||
(bool->int failed?)
|
||||
(kw->str outliner-op)
|
||||
(kw->str undo-redo)
|
||||
(sqlite-util/write-transit-str forward-outliner-ops')
|
||||
(sqlite-util/write-transit-str inverse-outliner-ops')
|
||||
(bool->int inferred-outliner-ops?)
|
||||
(sqlite-util/write-transit-str (or normalized-tx-data []))
|
||||
(sqlite-util/write-transit-str (or reversed-tx-data []))])
|
||||
{:tx-id tx-id
|
||||
:created-at created-at'
|
||||
:should-inc-pending? should-inc-pending?})))
|
||||
|
||||
(defn get-local-tx-entry
|
||||
[repo tx-id]
|
||||
(when (uuid? tx-id)
|
||||
(when-let [store (sqlite-store-or-throw repo)]
|
||||
(some-> (sqlite-row store
|
||||
(str "select tx_id, outliner_op, undo_redo, "
|
||||
"forward_outliner_ops, inverse_outliner_ops, inferred_outliner_ops, "
|
||||
"normalized_tx_data, reversed_tx_data "
|
||||
"from client_ops where kind = 'tx' and tx_id = ? limit 1")
|
||||
[(str tx-id)])
|
||||
(row->pending-local-tx)))))
|
||||
|
||||
(defn get-pending-local-txs
|
||||
[repo & {:keys [limit]}]
|
||||
(when-let [store (sqlite-store-or-throw repo)]
|
||||
(let [sql (str "select tx_id, outliner_op, undo_redo, "
|
||||
"forward_outliner_ops, inverse_outliner_ops, inferred_outliner_ops, "
|
||||
"normalized_tx_data, reversed_tx_data "
|
||||
"from client_ops where kind = 'tx' and pending = 1 "
|
||||
"order by created_at asc, id asc"
|
||||
(when (number? limit) " limit ?"))
|
||||
rows (sqlite-rows store sql (if (number? limit) [limit] []))]
|
||||
(->> rows
|
||||
(keep row->pending-local-tx)
|
||||
vec))))
|
||||
|
||||
(defn add-sync-conflicts!
|
||||
[repo conflicts]
|
||||
(when-let [store (sqlite-store-or-throw repo)]
|
||||
(let [now (.now js/Date)]
|
||||
(doseq [{:keys [block-uuid attr value remote-t]} conflicts]
|
||||
(when (and (uuid? block-uuid)
|
||||
(qualified-keyword? attr)
|
||||
(string? value))
|
||||
(let [attr-str (qualified-kw->str attr)]
|
||||
(sqlite-run! store
|
||||
"delete from sync_conflicts where block_uuid = ? and attr = ?"
|
||||
[(str block-uuid) attr-str])
|
||||
(when-not (string/blank? value)
|
||||
(sqlite-run! store
|
||||
(str "insert into sync_conflicts "
|
||||
"(block_uuid, attr, value, remote_t, created_at) "
|
||||
"values (?, ?, ?, ?, ?) "
|
||||
"on conflict(block_uuid, attr, value) do update set "
|
||||
"remote_t = excluded.remote_t, "
|
||||
"created_at = excluded.created_at")
|
||||
[(str block-uuid)
|
||||
attr-str
|
||||
value
|
||||
remote-t
|
||||
now]))))))))
|
||||
|
||||
(defn- sync-conflict-row->map
|
||||
[row]
|
||||
{:id (aget row "id")
|
||||
:block-uuid (parse-uuid-str (aget row "block_uuid"))
|
||||
:attr (str->qualified-kw (aget row "attr"))
|
||||
:value (aget row "value")
|
||||
:remote-t (aget row "remote_t")
|
||||
:created-at (aget row "created_at")})
|
||||
|
||||
(defn get-all-sync-conflicts
|
||||
[repo]
|
||||
(when-let [store (sqlite-store-or-throw repo)]
|
||||
(->> (sqlite-rows store
|
||||
(str "select id, block_uuid, attr, value, remote_t, created_at "
|
||||
"from sync_conflicts order by created_at desc, id desc")
|
||||
[])
|
||||
(map sync-conflict-row->map)
|
||||
(group-by :block-uuid))))
|
||||
|
||||
(defn get-sync-conflicts
|
||||
[repo block-uuid]
|
||||
(when (uuid? block-uuid)
|
||||
(when-let [store (sqlite-store-or-throw repo)]
|
||||
(->> (sqlite-rows store
|
||||
(str "select id, block_uuid, attr, value, remote_t, created_at "
|
||||
"from sync_conflicts where block_uuid = ? "
|
||||
"order by created_at desc, id desc")
|
||||
[(str block-uuid)])
|
||||
(mapv sync-conflict-row->map)))))
|
||||
|
||||
(defn clear-sync-conflicts!
|
||||
[repo block-uuid]
|
||||
(when (uuid? block-uuid)
|
||||
(when-let [store (sqlite-store-or-throw repo)]
|
||||
(sqlite-run! store
|
||||
"delete from sync_conflicts where block_uuid = ?"
|
||||
[(str block-uuid)]))))
|
||||
|
||||
(defn- pending-tx-id?
|
||||
[store tx-id]
|
||||
(let [row (sqlite-row store
|
||||
"select pending from client_ops where kind = 'tx' and tx_id = ?"
|
||||
[(str tx-id)])]
|
||||
(= 1 (some-> row (aget "pending")))))
|
||||
|
||||
(defn mark-pending-txs-false!
|
||||
[repo tx-ids]
|
||||
(when-let [store (sqlite-store-or-throw repo)]
|
||||
(let [tx-ids (->> tx-ids (filter uuid?) vec)
|
||||
pending-to-remove (->> tx-ids
|
||||
(filter (fn [tx-id]
|
||||
(pending-tx-id? store tx-id)))
|
||||
count)]
|
||||
(when (seq tx-ids)
|
||||
(doseq [tx-id tx-ids]
|
||||
(sqlite-run! store
|
||||
"update client_ops set pending = 0 where kind = 'tx' and tx_id = ?"
|
||||
[(str tx-id)])))
|
||||
pending-to-remove)))
|
||||
|
||||
(defn mark-failed-txs!
|
||||
[repo tx-ids]
|
||||
(when-let [store (sqlite-store-or-throw repo)]
|
||||
(let [tx-ids (->> tx-ids (filter uuid?) vec)
|
||||
pending-to-remove (->> tx-ids
|
||||
(filter (fn [tx-id]
|
||||
(pending-tx-id? store tx-id)))
|
||||
count)]
|
||||
(when (seq tx-ids)
|
||||
(doseq [tx-id tx-ids]
|
||||
(sqlite-run! store
|
||||
"update client_ops set pending = 0, failed = 1 where kind = 'tx' and tx_id = ?"
|
||||
[(str tx-id)])))
|
||||
pending-to-remove)))
|
||||
|
||||
(defn history-action-ops-by-tx-id
|
||||
[repo tx-id]
|
||||
(when-let [entry (get-local-tx-entry repo tx-id)]
|
||||
{:db-sync/forward-outliner-ops (some-> (:forward-outliner-ops entry) seq vec)
|
||||
:db-sync/inverse-outliner-ops (some-> (:inverse-outliner-ops entry) seq vec)}))
|
||||
|
||||
(defn- local-asset-op-map
|
||||
[op-type t value]
|
||||
(let [asset-uuid (:block-uuid value)]
|
||||
(case op-type
|
||||
:update-asset {:block/uuid asset-uuid
|
||||
:update-asset [:update-asset t value]}
|
||||
:remove-asset {:block/uuid asset-uuid
|
||||
:remove-asset [:remove-asset t value]}
|
||||
nil)))
|
||||
|
||||
(defn- sqlite-asset-op-by-uuid
|
||||
[store block-uuid]
|
||||
(when-let [row (sqlite-row store
|
||||
(str "select asset_uuid, asset_op, asset_t, asset_value "
|
||||
"from client_ops where kind = 'asset' and asset_uuid = ? limit 1")
|
||||
[(str block-uuid)])]
|
||||
(let [op-type (str->kw (aget row "asset_op"))
|
||||
t (aget row "asset_t")
|
||||
value (or (some-> (aget row "asset_value") sqlite-util/read-transit-str)
|
||||
{:block-uuid block-uuid})]
|
||||
(local-asset-op-map op-type t value))))
|
||||
|
||||
(defn- sqlite-upsert-asset-op!
|
||||
[store op-type t value]
|
||||
(let [block-uuid (:block-uuid value)]
|
||||
(sqlite-with-tx!
|
||||
store
|
||||
(fn [tx]
|
||||
(sqlite-run! tx "delete from client_ops where kind = 'asset' and asset_uuid = ?"
|
||||
[(str block-uuid)])
|
||||
(sqlite-run! tx
|
||||
(str "insert into client_ops ("
|
||||
"kind, created_at, asset_uuid, asset_op, asset_t, asset_value"
|
||||
") values ('asset', ?, ?, ?, ?, ?)")
|
||||
[(.now js/Date)
|
||||
(str block-uuid)
|
||||
(kw->str op-type)
|
||||
t
|
||||
(sqlite-util/write-transit-str value)])))))
|
||||
|
||||
;;; asset ops
|
||||
(defn add-asset-ops
|
||||
[repo asset-ops]
|
||||
(let [store (sqlite-store-or-throw repo)
|
||||
ops (ops-coercer asset-ops)]
|
||||
(assert (some? store) repo)
|
||||
(letfn [(already-removed? [remove-op t]
|
||||
(some-> remove-op second (> t)))
|
||||
(update-after-remove? [update-op t]
|
||||
(some-> update-op second (> t)))]
|
||||
(doseq [op ops]
|
||||
(let [[op-type t value] op
|
||||
{:keys [block-uuid]} value
|
||||
existing-op (sqlite-asset-op-by-uuid store block-uuid)]
|
||||
(case op-type
|
||||
:update-asset
|
||||
(let [remove-asset-op (:remove-asset existing-op)]
|
||||
(when-not (already-removed? remove-asset-op t)
|
||||
(sqlite-upsert-asset-op! store :update-asset t value)))
|
||||
|
||||
:remove-asset
|
||||
(let [update-asset-op (:update-asset existing-op)]
|
||||
(when-not (update-after-remove? update-asset-op t)
|
||||
(sqlite-upsert-asset-op! store :remove-asset t value)))
|
||||
|
||||
nil)))
|
||||
nil)))
|
||||
|
||||
(defn add-all-exists-asset-as-ops
|
||||
[repo]
|
||||
(let [conn (worker-state/get-datascript-conn repo)
|
||||
_ (assert (some? conn))
|
||||
asset-block-uuids (->> (d/datoms @conn :avet :logseq.property.asset/type)
|
||||
(keep (fn [d]
|
||||
(:block/uuid (d/entity @conn (:e d)))))
|
||||
distinct)
|
||||
ops (map (fn [block-uuid] [:update-asset 1 {:block-uuid block-uuid}])
|
||||
asset-block-uuids)]
|
||||
(add-asset-ops repo ops)))
|
||||
|
||||
(defn get-unpushed-asset-ops-count
|
||||
[repo]
|
||||
(when-let [store (sqlite-store-or-throw repo)]
|
||||
(or (some-> (sqlite-row store
|
||||
"select count(*) as c from client_ops where kind = 'asset'"
|
||||
[])
|
||||
(aget "c"))
|
||||
0)))
|
||||
|
||||
(defn get-all-asset-ops
|
||||
[repo]
|
||||
(when-let [store (sqlite-store-or-throw repo)]
|
||||
(->> (sqlite-rows store
|
||||
"select asset_op, asset_t, asset_value from client_ops where kind = 'asset' order by id asc"
|
||||
[])
|
||||
(keep (fn [row]
|
||||
(let [op-type (str->kw (aget row "asset_op"))
|
||||
t (aget row "asset_t")
|
||||
value (some-> (aget row "asset_value") sqlite-util/read-transit-str)]
|
||||
(when (and op-type (map? value) (:block-uuid value))
|
||||
(local-asset-op-map op-type t value)))))
|
||||
vec)))
|
||||
|
||||
(defn remove-asset-op
|
||||
[repo asset-uuid]
|
||||
(when-let [store (sqlite-store-or-throw repo)]
|
||||
(sqlite-run! store
|
||||
"delete from client_ops where kind = 'asset' and asset_uuid = ?"
|
||||
[(str asset-uuid)])))
|
||||
|
||||
(defn cleanup-finished-history-ops!
|
||||
[repo protected-tx-ids]
|
||||
(if-let [store (sqlite-store-or-throw repo)]
|
||||
(let [protected-tx-ids (set protected-tx-ids)
|
||||
tx-id-rows (sqlite-rows store
|
||||
(str "select tx_id from client_ops "
|
||||
"where kind = 'tx' and pending = 0 and tx_id is not null")
|
||||
[])
|
||||
removable-tx-ids (->> tx-id-rows
|
||||
(keep (fn [row]
|
||||
(let [tx-id (parse-uuid-str (aget row "tx_id"))]
|
||||
(when (and (uuid? tx-id)
|
||||
(not (contains? protected-tx-ids tx-id)))
|
||||
tx-id))))
|
||||
vec)]
|
||||
(when (seq removable-tx-ids)
|
||||
(doseq [tx-id removable-tx-ids]
|
||||
(sqlite-run! store
|
||||
"delete from client_ops where kind = 'tx' and tx_id = ?"
|
||||
[(str tx-id)])))
|
||||
(count removable-tx-ids))
|
||||
0))
|
||||
@@ -1,27 +0,0 @@
|
||||
(ns frontend.worker.sync.const
|
||||
"RTC constants"
|
||||
(:require [logseq.common.defkeywords :as common-def :refer [defkeywords]]
|
||||
[logseq.db.frontend.kv-entity :as kv-entity]
|
||||
[logseq.db.frontend.property :as db-property]))
|
||||
|
||||
(defkeywords
|
||||
:db-sync/invalid-e2ee-password
|
||||
{:doc "The supplied E2EE password cannot decrypt the current server private key."}
|
||||
:rtc/ignore-attr-when-syncing
|
||||
{:doc "keyword option for RTC. ignore this *attr* when syncing graph. Default false"}
|
||||
:rtc/ignore-entity-when-init-upload
|
||||
{:doc "keyword option for RTC. ignore this *entity* when initial uploading graph. Default false"})
|
||||
|
||||
(def ignore-attrs-when-syncing
|
||||
(into #{}
|
||||
(keep (fn [[kw config]] (when (get-in config [:rtc :rtc/ignore-attr-when-syncing]) kw)))
|
||||
db-property/built-in-properties))
|
||||
|
||||
(def ignore-entities-when-init-upload
|
||||
(into #{}
|
||||
(keep (fn [[kw config]] (when (get-in config [:rtc :rtc/ignore-entity-when-init-upload]) kw)))
|
||||
kv-entity/kv-entities))
|
||||
|
||||
(def encrypt-attr-set
|
||||
"block attributes that need to be encrypted"
|
||||
#{:block/title :block/name})
|
||||
@@ -1,909 +0,0 @@
|
||||
(ns frontend.worker.sync.crypt
|
||||
"E2EE helpers for db-sync."
|
||||
(:require [clojure.string :as string]
|
||||
[frontend.common.crypt :as crypt]
|
||||
[frontend.common.thread-api :refer [def-thread-api]]
|
||||
[frontend.worker-common.util :as worker-util]
|
||||
[frontend.worker.platform :as platform]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync.auth :as sync-auth]
|
||||
[frontend.worker.sync.const :as sync-const]
|
||||
[frontend.worker.ui-request :as ui-request]
|
||||
[lambdaisland.glogi :as log]
|
||||
[logseq.db :as ldb]
|
||||
[promesa.core :as p]
|
||||
[frontend.worker.sync.util :refer [fail-fast fetch-json coerce-http-request] :as sync-util]))
|
||||
|
||||
(defonce ^:private *graph->aes-key (atom {}))
|
||||
(defonce ^:private *user-rsa-key-pair-inflight (atom {}))
|
||||
(defonce ^:private *ensure-user-rsa-key-pair-inflight (atom {}))
|
||||
(defonce ^:private node-default-auth-file "~/logseq/auth.json")
|
||||
(defonce ^:private e2ee-password-secret-key "logseq-encrypted-password")
|
||||
(def ^:private invalid-transit ::invalid-transit)
|
||||
|
||||
(defn- runtime
|
||||
[platform']
|
||||
(get-in platform' [:env :runtime]))
|
||||
|
||||
(defn- read-transit-safe
|
||||
[value]
|
||||
(try
|
||||
(ldb/read-transit-str value)
|
||||
(catch :default _
|
||||
invalid-transit)))
|
||||
|
||||
(defn- browser-runtime?
|
||||
[platform']
|
||||
(= :browser (runtime platform')))
|
||||
|
||||
(defn- owner-source
|
||||
[platform']
|
||||
(get-in platform' [:env :owner-source]))
|
||||
|
||||
(defn- capacitor-runtime?
|
||||
[platform']
|
||||
(and (= :browser (runtime platform'))
|
||||
(= :capacitor (owner-source platform'))))
|
||||
|
||||
(defn- auth-file-path
|
||||
[]
|
||||
node-default-auth-file)
|
||||
|
||||
(defn- interactive-runtime?
|
||||
[]
|
||||
(let [env (:env (platform/current))
|
||||
runtime' (:runtime env)
|
||||
owner-source' (:owner-source env)]
|
||||
(or (= :browser runtime')
|
||||
(and (= :node runtime')
|
||||
(= :electron owner-source')))))
|
||||
|
||||
(defn- missing-e2ee-password-ex
|
||||
[data]
|
||||
(ex-info "missing-e2ee-password"
|
||||
(merge {:code :db-sync/missing-e2ee-password
|
||||
:field :e2ee-password}
|
||||
data)))
|
||||
|
||||
(defn- fail-missing-e2ee-password!
|
||||
[data]
|
||||
(fail-fast :db-sync/missing-e2ee-password
|
||||
(merge {:code :db-sync/missing-e2ee-password
|
||||
:field :e2ee-password}
|
||||
data)))
|
||||
|
||||
(defn- ensure-refresh-token!
|
||||
[refresh-token]
|
||||
(when-not (seq refresh-token)
|
||||
(fail-missing-e2ee-password! {:reason :missing-refresh-token
|
||||
:hint "Run logseq login first."})))
|
||||
|
||||
(def ^:private non-retriable-user-rsa-key-error-codes
|
||||
#{:ui-interaction-required
|
||||
:ui-request-cancelled
|
||||
:ui-request-rejected
|
||||
:ui-request-timeout})
|
||||
|
||||
(defn- user-rsa-key-cache-retryable-error?
|
||||
[error]
|
||||
(not (contains? non-retriable-user-rsa-key-error-codes
|
||||
(:code (ex-data error)))))
|
||||
|
||||
(defn- parse-auth-file
|
||||
[text]
|
||||
(when (seq text)
|
||||
(try
|
||||
(js->clj (js/JSON.parse text) :keywordize-keys true)
|
||||
(catch :default _
|
||||
invalid-transit))))
|
||||
|
||||
(defn- <read-refresh-token-from-auth-file
|
||||
[platform']
|
||||
(p/let [text (-> (platform/read-text! platform' (auth-file-path))
|
||||
(p/catch (fn [_]
|
||||
nil)))
|
||||
auth-data (parse-auth-file text)]
|
||||
(when (= invalid-transit auth-data)
|
||||
(fail-missing-e2ee-password! {:reason :invalid-auth-file
|
||||
:hint "Run logseq login first."}))
|
||||
(let [refresh-token (:refresh-token auth-data)]
|
||||
(ensure-refresh-token! refresh-token)
|
||||
refresh-token)))
|
||||
|
||||
(defn- <save-e2ee-password
|
||||
[password]
|
||||
(p/let [platform' (platform/current)
|
||||
refresh-token (if (browser-runtime? platform')
|
||||
(:auth/refresh-token @worker-state/*state)
|
||||
(<read-refresh-token-from-auth-file platform'))
|
||||
_ (ensure-refresh-token! refresh-token)
|
||||
result (crypt/<encrypt-text-by-text-password refresh-token password)
|
||||
text (ldb/write-transit-str result)
|
||||
native-saved? (if (capacitor-runtime? platform')
|
||||
(-> (ui-request/<request :native-save-e2ee-password
|
||||
{:key e2ee-password-secret-key
|
||||
:encrypted-text text})
|
||||
(p/then (fn [resp]
|
||||
(true? (:supported? resp))))
|
||||
(p/catch (fn [e]
|
||||
(log/warn :db-sync/save-e2ee-password-native-failed {:error e})
|
||||
false)))
|
||||
false)]
|
||||
(if native-saved?
|
||||
nil
|
||||
(platform/save-secret-text! platform' e2ee-password-secret-key text))))
|
||||
|
||||
(defn- <read-platform-e2ee-password-text
|
||||
[platform']
|
||||
(-> (platform/read-secret-text platform' e2ee-password-secret-key)
|
||||
(p/catch (fn [e]
|
||||
(log/warn :db-sync/read-e2ee-password-secret-failed {:error e})
|
||||
nil))))
|
||||
|
||||
(defn- <read-e2ee-password-text
|
||||
[refresh-token]
|
||||
(ensure-refresh-token! refresh-token)
|
||||
(p/let [platform' (platform/current)
|
||||
native-result (if (capacitor-runtime? platform')
|
||||
(-> (ui-request/<request :native-get-e2ee-password
|
||||
{:key e2ee-password-secret-key})
|
||||
(p/catch (fn [e]
|
||||
(log/warn :db-sync/read-e2ee-password-native-failed {:error e})
|
||||
{:supported? false})))
|
||||
{:supported? false})
|
||||
text (if (:supported? native-result)
|
||||
(:encrypted-text native-result)
|
||||
(<read-platform-e2ee-password-text platform'))]
|
||||
text))
|
||||
|
||||
(defn- <decrypt-e2ee-password-text
|
||||
[refresh-token text]
|
||||
(if-not (seq text)
|
||||
(p/rejected (missing-e2ee-password-ex {:reason :missing-persisted-password
|
||||
:hint "Provide --e2ee-password to persist it."}))
|
||||
(let [data (try
|
||||
(ldb/read-transit-str text)
|
||||
(catch :default _
|
||||
invalid-transit))]
|
||||
(when (= invalid-transit data)
|
||||
(fail-fast :db-sync/invalid-e2ee-password-payload
|
||||
{:field :e2ee-password
|
||||
:reason :invalid-transit-payload}))
|
||||
(crypt/<decrypt-text-by-text-password refresh-token data))))
|
||||
|
||||
(defn- <read-e2ee-password
|
||||
[refresh-token]
|
||||
(p/let [text (<read-e2ee-password-text refresh-token)]
|
||||
(<decrypt-e2ee-password-text refresh-token text)))
|
||||
|
||||
(defn- <clear-e2ee-password!
|
||||
[]
|
||||
(p/let [platform' (platform/current)
|
||||
native-deleted? (if (capacitor-runtime? platform')
|
||||
(-> (ui-request/<request :native-delete-e2ee-password
|
||||
{:key e2ee-password-secret-key})
|
||||
(p/then (fn [resp]
|
||||
(true? (:supported? resp))))
|
||||
(p/catch (fn [e]
|
||||
(log/warn :db-sync/delete-e2ee-password-native-failed {:error e})
|
||||
false)))
|
||||
false)
|
||||
_ (when-not native-deleted?
|
||||
(-> (platform/delete-secret-text! platform' e2ee-password-secret-key)
|
||||
(p/catch (fn [e]
|
||||
(log/warn :db-sync/delete-e2ee-password-secret-failed {:error e})
|
||||
nil))))]
|
||||
nil))
|
||||
|
||||
(defn e2ee-base
|
||||
[]
|
||||
(or (:http-base @worker-state/*db-sync-config)
|
||||
(when-let [ws-url (:ws-url @worker-state/*db-sync-config)]
|
||||
(let [base (cond
|
||||
(string/starts-with? ws-url "wss://")
|
||||
(str "https://" (subs ws-url (count "wss://")))
|
||||
|
||||
(string/starts-with? ws-url "ws://")
|
||||
(str "http://" (subs ws-url (count "ws://")))
|
||||
|
||||
:else ws-url)]
|
||||
(string/replace base #"/sync/%s$" "")))))
|
||||
|
||||
(defn graph-e2ee?
|
||||
[repo]
|
||||
(when-let [conn (worker-state/get-datascript-conn repo)]
|
||||
(ldb/get-graph-rtc-e2ee? @conn)))
|
||||
|
||||
(defn- get-user-uuid []
|
||||
(some-> (sync-util/auth-token) worker-util/parse-jwt :sub))
|
||||
|
||||
(defn- token->user-uuid
|
||||
[token]
|
||||
(some-> token worker-util/parse-jwt :sub))
|
||||
|
||||
(defn- <resolve-user-uuid
|
||||
[]
|
||||
(let [user-id (get-user-uuid)]
|
||||
(if (seq user-id)
|
||||
(p/resolved user-id)
|
||||
(-> (sync-auth/<resolve-ws-token)
|
||||
(p/then token->user-uuid)
|
||||
(p/catch (fn [_error]
|
||||
nil))))))
|
||||
|
||||
(defn- <get-item
|
||||
[k]
|
||||
(assert k)
|
||||
(p/let [r (platform/kv-get (platform/current) k)]
|
||||
(cond
|
||||
(instance? js/ArrayBuffer r) (js/Uint8Array. r)
|
||||
:else r)))
|
||||
|
||||
(defn- <set-item!
|
||||
[k value]
|
||||
(assert k)
|
||||
(platform/kv-set! (platform/current) k value))
|
||||
|
||||
(defn- <clear-item!
|
||||
[k]
|
||||
(assert k)
|
||||
(platform/kv-set! (platform/current) k nil))
|
||||
|
||||
(defn- graph-encrypted-aes-key-idb-key
|
||||
[graph-id]
|
||||
(str "rtc-encrypted-aes-key###" graph-id))
|
||||
|
||||
(defn- user-rsa-key-pair-idb-key
|
||||
[base user-id]
|
||||
(str "rtc-user-rsa-key-pair###" base "###" user-id))
|
||||
|
||||
(defn <fetch-user-rsa-key-pair-raw
|
||||
[base]
|
||||
(fetch-json (str base "/e2ee/user-keys")
|
||||
{:method "GET"}
|
||||
{:response-schema :e2ee/user-keys}))
|
||||
|
||||
(defn- user-rsa-key-pair-valid?
|
||||
[pair]
|
||||
(and (map? pair)
|
||||
(string? (:public-key pair))
|
||||
(string? (:encrypted-private-key pair))))
|
||||
|
||||
(defn- <set-user-rsa-key-pair-to-idb!
|
||||
[base user-id pair]
|
||||
(when (and (string? base)
|
||||
(string? user-id)
|
||||
(user-rsa-key-pair-valid? pair))
|
||||
(<set-item! (user-rsa-key-pair-idb-key base user-id)
|
||||
(ldb/write-transit-str pair)))
|
||||
pair)
|
||||
|
||||
(defn- <get-user-rsa-key-pair-from-idb
|
||||
[base user-id]
|
||||
(when (and (string? base) (string? user-id))
|
||||
(p/let [pair-str (<get-item (user-rsa-key-pair-idb-key base user-id))
|
||||
pair (ldb/read-transit-str pair-str)]
|
||||
(when (user-rsa-key-pair-valid? pair)
|
||||
pair))))
|
||||
|
||||
(defn- <clear-user-rsa-key-pair-cache!
|
||||
[base user-id]
|
||||
(let [k [base user-id]]
|
||||
(swap! *user-rsa-key-pair-inflight dissoc k)
|
||||
(when (and (string? base) (string? user-id))
|
||||
(<clear-item! (user-rsa-key-pair-idb-key base user-id)))))
|
||||
|
||||
(defn- <get-user-rsa-key-pair-raw
|
||||
[base]
|
||||
(p/let [user-id (<resolve-user-uuid)]
|
||||
(when-not (and (string? base) (string? user-id))
|
||||
(fail-fast :db-sync/missing-field {:base base :user-id user-id :field :user-rsa-key-pair}))
|
||||
(let [k [base user-id]]
|
||||
(if-let [inflight (get @*user-rsa-key-pair-inflight k)]
|
||||
inflight
|
||||
(let [task (-> (p/let [cached (<get-user-rsa-key-pair-from-idb base user-id)]
|
||||
(if cached
|
||||
cached
|
||||
(p/let [pair (<fetch-user-rsa-key-pair-raw base)]
|
||||
(if (user-rsa-key-pair-valid? pair)
|
||||
(p/let [_ (<set-user-rsa-key-pair-to-idb! base user-id pair)]
|
||||
pair)
|
||||
nil))))
|
||||
(p/finally (fn []
|
||||
(swap! *user-rsa-key-pair-inflight dissoc k))))]
|
||||
(swap! *user-rsa-key-pair-inflight assoc k task)
|
||||
task)))))
|
||||
|
||||
(defn <upload-user-rsa-key-pair!
|
||||
[base public-key encrypted-private-key]
|
||||
(let [body (coerce-http-request :e2ee/user-keys
|
||||
{:public-key public-key
|
||||
:encrypted-private-key encrypted-private-key})]
|
||||
(when (nil? body)
|
||||
(fail-fast :db-sync/invalid-field {:type :e2ee/user-keys :body body}))
|
||||
(p/let [pair (fetch-json (str base "/e2ee/user-keys")
|
||||
{:method "POST"
|
||||
:headers {"content-type" "application/json"}
|
||||
:body (js/JSON.stringify (clj->js body))}
|
||||
{:response-schema :e2ee/user-keys})
|
||||
user-id (<resolve-user-uuid)
|
||||
_ (<set-user-rsa-key-pair-to-idb! base user-id pair)]
|
||||
pair)))
|
||||
|
||||
(defn- <request-e2ee-password-from-ui
|
||||
[payload]
|
||||
(p/let [{:keys [password]} (ui-request/<request :request-e2ee-password
|
||||
payload
|
||||
{:hint "Provide e2ee-password to continue."})]
|
||||
(when-not (seq password)
|
||||
(fail-fast :db-sync/missing-e2ee-password {:field :e2ee-password
|
||||
:reason :empty-ui-password}))
|
||||
password))
|
||||
|
||||
(defn- <verify-e2ee-password
|
||||
[password encrypted-private-key-or-str]
|
||||
(when-not (seq password)
|
||||
(fail-missing-e2ee-password! {:reason :empty-password}))
|
||||
(p/let [encrypted-private-key (if (string? encrypted-private-key-or-str)
|
||||
(ldb/read-transit-str encrypted-private-key-or-str)
|
||||
encrypted-private-key-or-str)
|
||||
private-key (-> (crypt/<decrypt-private-key password encrypted-private-key)
|
||||
(p/catch (fn [error]
|
||||
(if (true? (:invalid-password? (ex-data error)))
|
||||
(p/rejected
|
||||
(ex-info "invalid-e2ee-password"
|
||||
{:code :db-sync/invalid-e2ee-password}
|
||||
error))
|
||||
(p/rejected error)))))]
|
||||
private-key))
|
||||
|
||||
(defn- <verify-and-save-e2ee-password!
|
||||
[password encrypted-private-key-or-str]
|
||||
(p/let [private-key (<verify-e2ee-password password encrypted-private-key-or-str)
|
||||
_ (<save-e2ee-password password)]
|
||||
private-key))
|
||||
|
||||
(defn- <verify-and-save-e2ee-password-from-server!
|
||||
[password]
|
||||
(let [base (e2ee-base)]
|
||||
(when-not (string? base)
|
||||
(fail-fast :db-sync/missing-field {:base base
|
||||
:field :e2ee-base}))
|
||||
(p/let [{:keys [encrypted-private-key]} (<fetch-user-rsa-key-pair-raw base)]
|
||||
(when-not (string? encrypted-private-key)
|
||||
(fail-fast :db-sync/missing-field {:base base
|
||||
:field :encrypted-private-key}))
|
||||
(<verify-and-save-e2ee-password! password encrypted-private-key))))
|
||||
|
||||
(defn- <generate-and-upload-user-rsa-key-pair!
|
||||
[base {:keys [password]}]
|
||||
(p/let [{:keys [publicKey privateKey]} (crypt/<generate-rsa-key-pair)
|
||||
password (cond
|
||||
(and (string? password) (seq password))
|
||||
password
|
||||
|
||||
(interactive-runtime?)
|
||||
(<request-e2ee-password-from-ui {:reason :generate-user-rsa-key-pair})
|
||||
|
||||
:else
|
||||
(fail-missing-e2ee-password! {:reason :missing-password-for-generate-user-rsa-key-pair
|
||||
:hint "Provide --e2ee-password when running sync ensure-keys --upload-keys."}))
|
||||
encrypted-private-key (crypt/<encrypt-private-key password privateKey)
|
||||
exported-public-key (crypt/<export-public-key publicKey)
|
||||
public-key-str (ldb/write-transit-str exported-public-key)
|
||||
encrypted-private-key-str (ldb/write-transit-str encrypted-private-key)]
|
||||
(p/let [_ (<save-e2ee-password password)
|
||||
_ (<upload-user-rsa-key-pair! base public-key-str encrypted-private-key-str)]
|
||||
{:public-key public-key-str
|
||||
:encrypted-private-key encrypted-private-key-str
|
||||
:password password})))
|
||||
|
||||
(defn- <ensure-user-rsa-key-pair-raw
|
||||
[base {:keys [ensure-server? server-rsa-keys-exists?] :as opts}]
|
||||
(p/let [existing (<get-user-rsa-key-pair-raw base)
|
||||
existing-valid? (user-rsa-key-pair-valid? existing)
|
||||
server-rsa-keys-exists?
|
||||
(cond
|
||||
(boolean? server-rsa-keys-exists?) server-rsa-keys-exists?
|
||||
(and ensure-server? existing-valid?)
|
||||
(p/let [server-pair (<fetch-user-rsa-key-pair-raw base)]
|
||||
(user-rsa-key-pair-valid? server-pair))
|
||||
:else nil)]
|
||||
(cond
|
||||
(and existing-valid? (not= false server-rsa-keys-exists?))
|
||||
existing
|
||||
|
||||
existing-valid?
|
||||
(p/let [_ (<upload-user-rsa-key-pair! base (:public-key existing) (:encrypted-private-key existing))]
|
||||
existing)
|
||||
|
||||
:else
|
||||
(<generate-and-upload-user-rsa-key-pair! base opts))))
|
||||
|
||||
(defn- <ensure-user-rsa-key-pair
|
||||
[base {:keys [password ensure-server? server-rsa-keys-exists?] :as opts}]
|
||||
(if (seq password)
|
||||
(<ensure-user-rsa-key-pair-raw base opts)
|
||||
(p/let [user-id (<resolve-user-uuid)]
|
||||
(when-not (and (string? base) (string? user-id))
|
||||
(fail-fast :db-sync/missing-field {:base base
|
||||
:user-id user-id
|
||||
:field :user-rsa-key-pair}))
|
||||
(let [k [base user-id ensure-server? server-rsa-keys-exists?]]
|
||||
(if-let [inflight (get @*ensure-user-rsa-key-pair-inflight k)]
|
||||
inflight
|
||||
(let [task (-> (<ensure-user-rsa-key-pair-raw base opts)
|
||||
(p/finally (fn []
|
||||
(swap! *ensure-user-rsa-key-pair-inflight dissoc k))))]
|
||||
(swap! *ensure-user-rsa-key-pair-inflight assoc k task)
|
||||
task))))))
|
||||
|
||||
(defn ensure-user-rsa-keys!
|
||||
([]
|
||||
(ensure-user-rsa-keys! nil))
|
||||
([opts]
|
||||
(let [base (e2ee-base)]
|
||||
(if (string? base)
|
||||
(<ensure-user-rsa-key-pair base opts)
|
||||
(do
|
||||
(log/info :db-sync/skip-ensure-user-rsa-keys {:reason :missing-e2ee-base})
|
||||
(p/resolved nil))))))
|
||||
|
||||
(defn- <decrypt-private-key
|
||||
([encrypted-private-key-str]
|
||||
(<decrypt-private-key encrypted-private-key-str nil))
|
||||
([encrypted-private-key-str {:keys [ui-password* save-ui-password?]
|
||||
:or {save-ui-password? true}}]
|
||||
(let [<decrypt-with-ui-request
|
||||
(fn [encrypted-private-key]
|
||||
(p/let [password (or (when (and ui-password* (seq @ui-password*))
|
||||
@ui-password*)
|
||||
(<request-e2ee-password-from-ui {:reason :decrypt-user-rsa-private-key}))
|
||||
_ (when ui-password*
|
||||
(reset! ui-password* password))
|
||||
private-key (<verify-e2ee-password password encrypted-private-key)
|
||||
_ (when save-ui-password?
|
||||
(<save-e2ee-password password))]
|
||||
private-key))
|
||||
|
||||
<decrypt-in-headless
|
||||
(fn [encrypted-private-key]
|
||||
(let [refresh-token (:auth/refresh-token @worker-state/*state)]
|
||||
(p/let [text (<read-e2ee-password-text refresh-token)]
|
||||
(if (seq text)
|
||||
(p/let [password (<decrypt-e2ee-password-text refresh-token text)]
|
||||
(when-not (seq password)
|
||||
(fail-missing-e2ee-password! {:reason :headless-empty-password
|
||||
:hint "Provide --e2ee-password to persist it."}))
|
||||
(<verify-e2ee-password password encrypted-private-key))
|
||||
(p/rejected (missing-e2ee-password-ex {:reason :missing-persisted-password
|
||||
:hint "Provide --e2ee-password to persist it."}))))))]
|
||||
(p/let [encrypted-private-key (ldb/read-transit-str encrypted-private-key-str)]
|
||||
(if (and ui-password* (seq @ui-password*))
|
||||
(<decrypt-with-ui-request encrypted-private-key)
|
||||
(-> (<decrypt-in-headless encrypted-private-key)
|
||||
(p/catch (fn [headless-error]
|
||||
(if-not (interactive-runtime?)
|
||||
(p/rejected headless-error)
|
||||
(<decrypt-with-ui-request encrypted-private-key))))))))))
|
||||
|
||||
(defn- <import-public-key
|
||||
[public-key-str]
|
||||
(p/let [exported (ldb/read-transit-str public-key-str)]
|
||||
(crypt/<import-public-key exported)))
|
||||
|
||||
(defn- <fetch-user-public-key-by-email
|
||||
[base email]
|
||||
(fetch-json (str base "/e2ee/user-public-key?email=" (js/encodeURIComponent email))
|
||||
{:method "GET"}
|
||||
{:response-schema :e2ee/user-public-key}))
|
||||
|
||||
(defn- <fetch-graph-encrypted-aes-key-raw
|
||||
[base graph-id]
|
||||
(fetch-json (str base "/e2ee/graphs/" graph-id "/aes-key")
|
||||
{:method "GET"}
|
||||
{:response-schema :e2ee/graph-aes-key}))
|
||||
|
||||
(defn- <fetch-graph-encrypted-aes-key
|
||||
[base graph-id]
|
||||
(when graph-id
|
||||
(p/let [resp (<fetch-graph-encrypted-aes-key-raw base graph-id)]
|
||||
(when-let [encrypted-aes-key (:encrypted-aes-key resp)]
|
||||
(ldb/read-transit-str encrypted-aes-key)))))
|
||||
|
||||
(defn- <upsert-graph-encrypted-aes-key!
|
||||
[base graph-id encrypted-aes-key]
|
||||
(let [body (coerce-http-request :e2ee/graph-aes-key
|
||||
{:encrypted-aes-key encrypted-aes-key})]
|
||||
(when (nil? body)
|
||||
(fail-fast :db-sync/invalid-field {:type :e2ee/graph-aes-key :body body}))
|
||||
(fetch-json (str base "/e2ee/graphs/" graph-id "/aes-key")
|
||||
{:method "POST"
|
||||
:headers {"content-type" "application/json"}
|
||||
:body (js/JSON.stringify (clj->js body))}
|
||||
{:response-schema :e2ee/graph-aes-key})))
|
||||
|
||||
(defn- <load-user-rsa-key-material
|
||||
[base user-id graph-id]
|
||||
(let [ui-password* (atom nil)]
|
||||
(letfn [(<load-once []
|
||||
(p/let [{:keys [public-key encrypted-private-key]} (<ensure-user-rsa-key-pair-raw base nil)
|
||||
_ (when-not (and (string? public-key) (string? encrypted-private-key))
|
||||
(fail-fast :db-sync/missing-field
|
||||
{:base base
|
||||
:user-id user-id
|
||||
:graph-id graph-id
|
||||
:field :user-rsa-key-pair}))
|
||||
public-key' (<import-public-key public-key)
|
||||
private-key' (<decrypt-private-key encrypted-private-key
|
||||
{:ui-password* ui-password*
|
||||
:save-ui-password? false})]
|
||||
{:public-key public-key'
|
||||
:private-key private-key'}))]
|
||||
(-> (<load-once)
|
||||
(p/catch (fn [error]
|
||||
(if-not (user-rsa-key-cache-retryable-error? error)
|
||||
(p/rejected error)
|
||||
(-> (p/let [_ (<clear-user-rsa-key-pair-cache! base user-id)]
|
||||
(<load-once))
|
||||
(p/catch (fn [retry-error]
|
||||
(log/warn :db-sync/user-rsa-key-cache-invalid
|
||||
{:base base
|
||||
:user-id user-id
|
||||
:graph-id graph-id
|
||||
:first-error error
|
||||
:retry-error retry-error})
|
||||
(throw retry-error)))))))
|
||||
(p/then (fn [key-material]
|
||||
(if-let [password (when (seq @ui-password*)
|
||||
@ui-password*)]
|
||||
(p/let [_ (<save-e2ee-password password)]
|
||||
key-material)
|
||||
key-material)))))))
|
||||
|
||||
(defn <preflight-upload-e2ee!
|
||||
[repo encrypted-graph?]
|
||||
(if-not (true? encrypted-graph?)
|
||||
(p/resolved nil)
|
||||
(let [base (e2ee-base)]
|
||||
(p/let [user-id (<resolve-user-uuid)]
|
||||
(when-not (and (string? base) (string? user-id))
|
||||
(fail-fast :db-sync/missing-field {:repo repo
|
||||
:base base
|
||||
:user-id user-id
|
||||
:field :user-rsa-key-pair}))
|
||||
(p/let [_ (<load-user-rsa-key-material base user-id nil)]
|
||||
nil)))))
|
||||
|
||||
(defn <ensure-graph-aes-key
|
||||
[repo graph-id]
|
||||
(if-not (graph-e2ee? repo)
|
||||
(p/resolved nil)
|
||||
(if-let [cached (get @*graph->aes-key graph-id)]
|
||||
(p/resolved cached)
|
||||
(let [base (e2ee-base)]
|
||||
(p/let [user-id (<resolve-user-uuid)]
|
||||
(when-not (and (string? base) (string? user-id))
|
||||
(fail-fast :db-sync/missing-field {:base base :user-id user-id :graph-id graph-id}))
|
||||
(p/let [{:keys [public-key private-key]} (<load-user-rsa-key-material base user-id graph-id)
|
||||
local-encrypted (when graph-id
|
||||
(<get-item (graph-encrypted-aes-key-idb-key graph-id)))
|
||||
remote-encrypted (when (and (nil? local-encrypted) graph-id)
|
||||
(<fetch-graph-encrypted-aes-key base graph-id))
|
||||
encrypted-aes-key (or local-encrypted remote-encrypted)
|
||||
aes-key (if encrypted-aes-key
|
||||
(-> (crypt/<decrypt-aes-key private-key encrypted-aes-key)
|
||||
(p/catch (fn [error]
|
||||
(if-not (and graph-id local-encrypted)
|
||||
(throw error)
|
||||
(let [aes-key-k (graph-encrypted-aes-key-idb-key graph-id)]
|
||||
(-> (p/let [_ (<clear-item! aes-key-k)
|
||||
refetched-encrypted (<fetch-graph-encrypted-aes-key base graph-id)]
|
||||
(if-not refetched-encrypted
|
||||
(throw error)
|
||||
(p/let [aes-key (crypt/<decrypt-aes-key private-key refetched-encrypted)
|
||||
_ (<set-item! aes-key-k refetched-encrypted)]
|
||||
aes-key)))
|
||||
(p/catch (fn [retry-error]
|
||||
(log/warn :db-sync/graph-aes-key-cache-invalid
|
||||
{:base base
|
||||
:user-id user-id
|
||||
:graph-id graph-id
|
||||
:first-error error
|
||||
:retry-error retry-error})
|
||||
(throw retry-error)))))))))
|
||||
(p/let [aes-key (crypt/<generate-aes-key)
|
||||
encrypted (crypt/<encrypt-aes-key public-key aes-key)
|
||||
encrypted-str (ldb/write-transit-str encrypted)
|
||||
_ (<upsert-graph-encrypted-aes-key! base graph-id encrypted-str)
|
||||
_ (<set-item! (graph-encrypted-aes-key-idb-key graph-id) encrypted)]
|
||||
aes-key))
|
||||
_ (when (and graph-id encrypted-aes-key (nil? local-encrypted))
|
||||
(<set-item! (graph-encrypted-aes-key-idb-key graph-id) encrypted-aes-key))]
|
||||
(swap! *graph->aes-key assoc graph-id aes-key)
|
||||
aes-key))))))
|
||||
|
||||
(defn <fetch-graph-aes-key-for-download
|
||||
[graph-id]
|
||||
(let [base (e2ee-base)
|
||||
aes-key-k (graph-encrypted-aes-key-idb-key graph-id)]
|
||||
(when-not (and (string? base) (string? graph-id))
|
||||
(fail-fast :db-sync/missing-field {:base base :graph-id graph-id}))
|
||||
(letfn [(<fetch-once [{:keys [public-key encrypted-private-key]}]
|
||||
(p/let [_ (<clear-item! aes-key-k)]
|
||||
(when-not (and (string? public-key) (string? encrypted-private-key))
|
||||
(fail-fast :db-sync/missing-field {:graph-id graph-id :field :user-rsa-key-pair}))
|
||||
(p/let [private-key (<decrypt-private-key encrypted-private-key)
|
||||
encrypted-aes-key (p/let [resp (<fetch-graph-encrypted-aes-key-raw base graph-id)]
|
||||
(when-let [encrypted-aes-key (:encrypted-aes-key resp)]
|
||||
(ldb/read-transit-str encrypted-aes-key)))]
|
||||
(if-not encrypted-aes-key
|
||||
(fail-fast :db-sync/missing-field {:graph-id graph-id :field :encrypted-aes-key})
|
||||
(<set-item! aes-key-k encrypted-aes-key))
|
||||
(p/let [aes-key (crypt/<decrypt-aes-key private-key encrypted-aes-key)]
|
||||
(swap! *graph->aes-key assoc graph-id aes-key)
|
||||
aes-key))))]
|
||||
(p/let [pair (<get-user-rsa-key-pair-raw base)]
|
||||
(-> (<fetch-once pair)
|
||||
(p/catch
|
||||
(fn [error]
|
||||
(let [user-id (get-user-uuid)]
|
||||
(if (and (= "decrypt-aes-key" (ex-message error))
|
||||
(string? user-id))
|
||||
(-> (p/let [_ (<clear-user-rsa-key-pair-cache! base user-id)
|
||||
refreshed-pair (<get-user-rsa-key-pair-raw base)]
|
||||
(<fetch-once refreshed-pair))
|
||||
(p/catch (fn [retry-error]
|
||||
(log/warn :db-sync/user-rsa-key-cache-invalid-on-download
|
||||
{:base base
|
||||
:user-id user-id
|
||||
:graph-id graph-id
|
||||
:first-error error
|
||||
:retry-error retry-error})
|
||||
(throw retry-error))))
|
||||
(throw error))))))))))
|
||||
|
||||
(defn <grant-graph-access!
|
||||
[repo graph-id target-email]
|
||||
(if-not (graph-e2ee? repo)
|
||||
(p/resolved nil)
|
||||
(let [base (e2ee-base)]
|
||||
(when-not (string? base)
|
||||
(fail-fast :db-sync/missing-field {:base base :graph-id graph-id}))
|
||||
(p/let [aes-key (<ensure-graph-aes-key repo graph-id)
|
||||
_ (when (nil? aes-key)
|
||||
(fail-fast :db-sync/missing-field {:repo repo :field :aes-key}))
|
||||
resp (<fetch-user-public-key-by-email base target-email)
|
||||
public-key-str (:public-key resp)]
|
||||
(if-not (string? public-key-str)
|
||||
(fail-fast :db-sync/missing-field {:repo repo :field :public-key :email target-email})
|
||||
(p/let [public-key (<import-public-key public-key-str)
|
||||
encrypted (crypt/<encrypt-aes-key public-key aes-key)
|
||||
encrypted-str (ldb/write-transit-str encrypted)
|
||||
body (coerce-http-request :e2ee/grant-access
|
||||
{:target-user-email+encrypted-aes-key-coll
|
||||
[{:email target-email
|
||||
:encrypted-aes-key encrypted-str}]})
|
||||
_ (when (nil? body)
|
||||
(fail-fast :db-sync/invalid-field {:type :e2ee/grant-access :body body}))
|
||||
_ (fetch-json (str base "/e2ee/graphs/" graph-id "/grant-access")
|
||||
{:method "POST"
|
||||
:headers {"content-type" "application/json"}
|
||||
:body (js/JSON.stringify (clj->js body))}
|
||||
{:response-schema :e2ee/grant-access})]
|
||||
nil))))))
|
||||
|
||||
(defn <encrypt-text-value
|
||||
[aes-key value]
|
||||
(assert (string? value) (str "encrypting value should be a string, value: " value))
|
||||
(p/let [encrypted (crypt/<encrypt-text aes-key (ldb/write-transit-str value))]
|
||||
(ldb/write-transit-str encrypted)))
|
||||
|
||||
(defn <decrypt-text-value
|
||||
[aes-key value]
|
||||
(assert (string? value) (str "encrypted value should be a string, value: " value))
|
||||
(let [decoded (read-transit-safe value)]
|
||||
(if (= decoded invalid-transit)
|
||||
(p/resolved value)
|
||||
(p/let [value (or (crypt/<decrypt-text-if-encrypted aes-key decoded)
|
||||
decoded)
|
||||
value' (if (string? value)
|
||||
(read-transit-safe value)
|
||||
value)]
|
||||
(if (= value' invalid-transit)
|
||||
value
|
||||
value')))))
|
||||
|
||||
(defn- encrypt-tx-item
|
||||
[aes-key item]
|
||||
(cond
|
||||
(and (vector? item) (<= 4 (count item)))
|
||||
(let [attr (nth item 2)
|
||||
v (nth item 3)]
|
||||
(if (contains? sync-const/encrypt-attr-set attr)
|
||||
(p/let [v' (<encrypt-text-value aes-key v)]
|
||||
(assoc item 3 v'))
|
||||
(p/resolved item)))
|
||||
|
||||
:else
|
||||
(p/resolved item)))
|
||||
|
||||
(defn- decrypt-tx-item
|
||||
[aes-key item]
|
||||
(cond
|
||||
(and (vector? item) (<= 4 (count item)))
|
||||
(let [attr (nth item 2)
|
||||
v (nth item 3)]
|
||||
(if (contains? sync-const/encrypt-attr-set attr)
|
||||
(p/let [v' (<decrypt-text-value aes-key v)]
|
||||
(assoc item 3 v'))
|
||||
(p/resolved item)))
|
||||
|
||||
:else
|
||||
(p/resolved item)))
|
||||
|
||||
(defn <encrypt-tx-data
|
||||
[aes-key tx-data]
|
||||
(p/let [items (p/all (mapv (fn [item] (encrypt-tx-item aes-key item)) tx-data))]
|
||||
items))
|
||||
|
||||
(defn <decrypt-tx-data
|
||||
[aes-key tx-data]
|
||||
(p/let [items (p/all (mapv (fn [item] (decrypt-tx-item aes-key item)) tx-data))]
|
||||
items))
|
||||
|
||||
(defn- <decrypt-datoms
|
||||
[aes-key data]
|
||||
(p/all
|
||||
(map
|
||||
(fn [[e a v t]]
|
||||
(if (contains? sync-const/encrypt-attr-set a)
|
||||
(p/let [v' (<decrypt-text-value aes-key v)]
|
||||
[e a v' t])
|
||||
[e a v t]))
|
||||
data)))
|
||||
|
||||
(defn- <decrypt-snapshot-row
|
||||
[aes-key row]
|
||||
(let [[addr raw-content raw-addresses] row
|
||||
data (ldb/read-transit-str raw-content)
|
||||
addresses (when raw-addresses
|
||||
(js/JSON.parse raw-addresses))]
|
||||
(if (map? data)
|
||||
(p/let [keys (:keys data)
|
||||
keys' (if (seq keys)
|
||||
(<decrypt-datoms aes-key (:keys data))
|
||||
keys)
|
||||
result (assoc data :keys keys')]
|
||||
[addr (ldb/write-transit-str (cond-> result
|
||||
(some? addresses)
|
||||
(assoc :addresses addresses)))
|
||||
raw-addresses])
|
||||
(p/let [result (p/all (map #(<decrypt-datoms aes-key %) data))]
|
||||
[addr (ldb/write-transit-str result) raw-addresses]))))
|
||||
|
||||
(defn <decrypt-snapshot-rows-batch
|
||||
[aes-key rows-batch]
|
||||
(p/all (map #(<decrypt-snapshot-row aes-key %) rows-batch)))
|
||||
|
||||
(defn <decrypt-snapshot-datoms-batch
|
||||
[aes-key datoms]
|
||||
(p/all
|
||||
(map (fn [{:keys [a v] :as datom}]
|
||||
(if (contains? sync-const/encrypt-attr-set a)
|
||||
(p/let [v' (<decrypt-text-value aes-key v)]
|
||||
(assoc datom :v v'))
|
||||
(p/resolved datom)))
|
||||
datoms)))
|
||||
|
||||
(defn <encrypt-datoms
|
||||
([aes-key datoms]
|
||||
(<encrypt-datoms aes-key datoms nil))
|
||||
([aes-key datoms progress-f]
|
||||
(let [batch-size 5000
|
||||
total-count (count datoms)
|
||||
batches (partition-all batch-size datoms)]
|
||||
(p/loop [remaining batches
|
||||
result []
|
||||
encrypted-count 0]
|
||||
(if (empty? remaining)
|
||||
result
|
||||
(p/let [batch (first remaining)
|
||||
encrypted (p/all (map (fn [datom]
|
||||
(if (contains? sync-const/encrypt-attr-set (:a datom))
|
||||
(p/let [v' (<encrypt-text-value aes-key (:v datom))]
|
||||
(assoc datom :v v'))
|
||||
(p/resolved datom)))
|
||||
batch))]
|
||||
(let [encrypted-count' (+ encrypted-count (count batch))]
|
||||
(when progress-f
|
||||
(progress-f encrypted-count' total-count))
|
||||
(p/recur (rest remaining) (into result encrypted) encrypted-count'))))))))
|
||||
|
||||
(defn- <re-encrypt-private-key
|
||||
[encrypted-private-key-str old-password new-password]
|
||||
(p/let [encrypted-private-key (ldb/read-transit-str encrypted-private-key-str)
|
||||
private-key (crypt/<decrypt-private-key old-password encrypted-private-key)
|
||||
new-encrypted-private-key (crypt/<encrypt-private-key new-password private-key)]
|
||||
(ldb/write-transit-str new-encrypted-private-key)))
|
||||
|
||||
(defn <change-e2ee-password!
|
||||
[_refresh-token user-uuid old-password new-password]
|
||||
(let [base (e2ee-base)]
|
||||
(when-not (string? base)
|
||||
(fail-fast :db-sync/missing-field {:base base :user-uuid user-uuid}))
|
||||
(p/let [{:keys [public-key encrypted-private-key]} (<get-user-rsa-key-pair-raw base)]
|
||||
(when-not (and (string? public-key) (string? encrypted-private-key))
|
||||
(fail-fast :db-sync/missing-field {:base base :user-uuid user-uuid :field :user-rsa-key-pair}))
|
||||
(p/let [encrypted-private-key' (<re-encrypt-private-key encrypted-private-key old-password new-password)
|
||||
_ (<upload-user-rsa-key-pair! base public-key encrypted-private-key')
|
||||
_ (<save-e2ee-password new-password)]
|
||||
nil))))
|
||||
|
||||
(defn cancel-ui-requests!
|
||||
[context]
|
||||
(ui-request/cancel-all! context))
|
||||
|
||||
(def-thread-api :thread-api/get-user-rsa-key-pair
|
||||
[_token _user-uuid]
|
||||
(let [base (e2ee-base)]
|
||||
(when-not (string? base)
|
||||
(fail-fast :db-sync/missing-field {:base base}))
|
||||
(p/let [{:keys [public-key encrypted-private-key]} (<get-user-rsa-key-pair-raw base)]
|
||||
(when (and public-key encrypted-private-key)
|
||||
{:public-key public-key
|
||||
:encrypted-private-key encrypted-private-key}))))
|
||||
|
||||
(def-thread-api :thread-api/init-user-rsa-key-pair
|
||||
[_token _refresh-token _user-uuid]
|
||||
(let [base (e2ee-base)]
|
||||
(when-not (string? base)
|
||||
(fail-fast :db-sync/missing-field {:base base}))
|
||||
(p/let [existing (<get-user-rsa-key-pair-raw base)]
|
||||
(when-not (and (string? (:public-key existing))
|
||||
(string? (:encrypted-private-key existing)))
|
||||
(p/let [{:keys [publicKey privateKey]} (crypt/<generate-rsa-key-pair)
|
||||
password (<request-e2ee-password-from-ui {:reason :init-user-rsa-key-pair})
|
||||
encrypted-private-key (crypt/<encrypt-private-key password privateKey)
|
||||
exported-public-key (crypt/<export-public-key publicKey)
|
||||
public-key-str (ldb/write-transit-str exported-public-key)
|
||||
encrypted-private-key-str (ldb/write-transit-str encrypted-private-key)
|
||||
_ (<upload-user-rsa-key-pair! base public-key-str encrypted-private-key-str)
|
||||
_ (<save-e2ee-password password)]
|
||||
nil)))))
|
||||
|
||||
(def-thread-api :thread-api/reset-user-rsa-key-pair
|
||||
[_token _refresh-token _user-uuid new-password]
|
||||
(p/let [{:keys [publicKey privateKey]} (crypt/<generate-rsa-key-pair)
|
||||
encrypted-private-key (crypt/<encrypt-private-key new-password privateKey)
|
||||
exported-public-key (crypt/<export-public-key publicKey)
|
||||
public-key-str (ldb/write-transit-str exported-public-key)
|
||||
encrypted-private-key-str (ldb/write-transit-str encrypted-private-key)
|
||||
base (e2ee-base)]
|
||||
(when-not (string? base)
|
||||
(fail-fast :db-sync/missing-field {:base base}))
|
||||
(p/let [_ (<upload-user-rsa-key-pair! base public-key-str encrypted-private-key-str)
|
||||
_ (<save-e2ee-password new-password)]
|
||||
nil)))
|
||||
|
||||
(def-thread-api :thread-api/change-e2ee-password
|
||||
[_token refresh-token user-uuid old-password new-password]
|
||||
(<change-e2ee-password! refresh-token user-uuid old-password new-password))
|
||||
|
||||
(def-thread-api :thread-api/verify-and-save-e2ee-password
|
||||
[_refresh-token password]
|
||||
(p/let [_ (<verify-and-save-e2ee-password-from-server! password)]
|
||||
nil))
|
||||
|
||||
(def-thread-api :thread-api/get-e2ee-password
|
||||
[refresh-token]
|
||||
(p/let [password (<read-e2ee-password refresh-token)]
|
||||
{:password password}))
|
||||
|
||||
(def-thread-api :thread-api/save-e2ee-password
|
||||
[password]
|
||||
(<save-e2ee-password password))
|
||||
|
||||
(def-thread-api :thread-api/clear-e2ee-password
|
||||
[]
|
||||
(<clear-e2ee-password!))
|
||||
@@ -1,576 +0,0 @@
|
||||
(ns frontend.worker.sync.download
|
||||
"Download helpers for graph snapshots."
|
||||
(:require
|
||||
[datascript.core :as d]
|
||||
[frontend.common.thread-api :as thread-api]
|
||||
[frontend.worker-common.util :as worker-util]
|
||||
[frontend.worker.platform :as platform]
|
||||
[frontend.worker.search :as search]
|
||||
[frontend.worker.shared-service :as shared-service]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync.auth :as sync-auth]
|
||||
[frontend.worker.sync.client-op :as client-op]
|
||||
[frontend.worker.sync.crypt :as sync-crypt]
|
||||
[frontend.worker.sync.log-and-state :as rtc-log-and-state]
|
||||
[frontend.worker.sync.temp-sqlite :as sync-temp-sqlite]
|
||||
[frontend.worker.sync.util :refer [fail-fast] :as sync-util]
|
||||
[lambdaisland.glogi :as log]
|
||||
[logseq.db-sync.snapshot :as snapshot]
|
||||
[logseq.db.common.sqlite :as common-sqlite]
|
||||
[logseq.db.frontend.schema :as db-schema]
|
||||
[promesa.core :as p]
|
||||
[logseq.db :as ldb]))
|
||||
|
||||
(defn- ->uint8 [data]
|
||||
(cond
|
||||
(instance? js/Uint8Array data) data
|
||||
(instance? js/ArrayBuffer data) (js/Uint8Array. data)
|
||||
(string? data) (.encode (js/TextEncoder.) data)
|
||||
:else (js/Uint8Array. data)))
|
||||
|
||||
(defn- gzip-bytes?
|
||||
[^js payload]
|
||||
(and (some? payload)
|
||||
(>= (.-byteLength payload) 2)
|
||||
(= 31 (aget payload 0))
|
||||
(= 139 (aget payload 1))))
|
||||
|
||||
(defn- bytes->stream
|
||||
[^js payload]
|
||||
(js/ReadableStream.
|
||||
#js {:start (fn [controller]
|
||||
(.enqueue controller payload)
|
||||
(.close controller))}))
|
||||
|
||||
(defn- <decompress-gzip-bytes
|
||||
[^js payload]
|
||||
(if (exists? js/DecompressionStream)
|
||||
(p/let [stream (bytes->stream payload)
|
||||
decompressed (.pipeThrough stream (js/DecompressionStream. "gzip"))
|
||||
resp (js/Response. decompressed)
|
||||
buf (.arrayBuffer resp)]
|
||||
(->uint8 buf))
|
||||
(p/rejected (ex-info "gzip decompression not supported"
|
||||
{:type :db-sync/decompression-not-supported}))))
|
||||
|
||||
(defn- <snapshot-response-bytes
|
||||
[^js resp]
|
||||
(p/let [buf (.arrayBuffer resp)
|
||||
chunk (->uint8 buf)]
|
||||
(if (gzip-bytes? chunk)
|
||||
(<decompress-gzip-bytes chunk)
|
||||
chunk)))
|
||||
|
||||
(defn- <stream-starts-with-gzip?
|
||||
[^js stream]
|
||||
(let [reader (.getReader stream)]
|
||||
(-> (.read reader)
|
||||
(p/then (fn [result]
|
||||
(if (.-done result)
|
||||
false
|
||||
(gzip-bytes? (->uint8 (.-value result))))))
|
||||
(p/catch (fn [_] false))
|
||||
(p/finally (fn []
|
||||
(try
|
||||
(.releaseLock reader)
|
||||
(catch :default _)))))))
|
||||
|
||||
(defn- <response-body-stream
|
||||
[^js resp]
|
||||
(let [body (.-body resp)
|
||||
encoding (some-> resp .-headers (.get "content-encoding"))]
|
||||
(cond
|
||||
(nil? body)
|
||||
(p/resolved nil)
|
||||
|
||||
;; Never trust `content-encoding` alone.
|
||||
;; Some runtimes (e.g. Node/undici fetch) may auto-decompress body while
|
||||
;; still exposing `content-encoding: gzip` in headers.
|
||||
;; We only stream-decompress when the first bytes are actual gzip magic.
|
||||
(and (= "gzip" encoding)
|
||||
(exists? js/DecompressionStream)
|
||||
(fn? (.-tee body)))
|
||||
(let [branches (.tee body)
|
||||
probe (aget branches 0)
|
||||
payload (aget branches 1)]
|
||||
(-> (<stream-starts-with-gzip? probe)
|
||||
(p/then (fn [gzip?]
|
||||
(if gzip?
|
||||
(.pipeThrough payload (js/DecompressionStream. "gzip"))
|
||||
payload)))
|
||||
;; If probing fails, keep original payload stream.
|
||||
(p/catch (fn [_] payload))))
|
||||
|
||||
;; If we cannot safely probe (no tee support), do not guess.
|
||||
;; Fall back to the arrayBuffer path where we inspect magic bytes first.
|
||||
(= "gzip" encoding)
|
||||
(p/resolved nil)
|
||||
|
||||
:else
|
||||
(p/resolved body))))
|
||||
|
||||
(defn- <flush-row-batches!
|
||||
[rows batch-size on-batch]
|
||||
(p/loop [remaining rows]
|
||||
(if (>= (count remaining) batch-size)
|
||||
(let [batch (subvec remaining 0 batch-size)
|
||||
rest-rows (subvec remaining batch-size)]
|
||||
(p/let [_ (on-batch batch)]
|
||||
(p/recur rest-rows)))
|
||||
remaining)))
|
||||
|
||||
(defn- <stream-snapshot-row-batches!
|
||||
[^js resp batch-size on-batch]
|
||||
(p/let [stream (<response-body-stream resp)]
|
||||
(if stream
|
||||
(let [reader (.getReader stream)]
|
||||
(p/loop [buffer nil
|
||||
pending []]
|
||||
(p/let [result (.read reader)]
|
||||
(if (.-done result)
|
||||
(let [pending (if (and buffer (pos? (.-byteLength buffer)))
|
||||
(into pending (snapshot/finalize-framed-buffer buffer))
|
||||
pending)]
|
||||
(if (seq pending)
|
||||
(p/let [_ (on-batch pending)]
|
||||
{:chunk-count 1})
|
||||
{:chunk-count 0}))
|
||||
(let [{rows :rows next-buffer :buffer} (snapshot/parse-framed-chunk buffer (->uint8 (.-value result)))
|
||||
pending (into pending rows)]
|
||||
(p/let [pending (<flush-row-batches! pending batch-size on-batch)]
|
||||
(p/recur next-buffer pending)))))))
|
||||
(p/let [snapshot-bytes (<snapshot-response-bytes resp)
|
||||
rows (vec (snapshot/finalize-framed-buffer snapshot-bytes))]
|
||||
(if (seq rows)
|
||||
(p/let [_ (on-batch rows)]
|
||||
{:chunk-count 1})
|
||||
{:chunk-count 0})))))
|
||||
|
||||
(defn- with-auth-headers
|
||||
[opts]
|
||||
(sync-auth/with-auth-headers
|
||||
#(sync-auth/auth-headers (worker-state/get-id-token))
|
||||
opts))
|
||||
|
||||
(defn- fetch-json
|
||||
[url opts schema]
|
||||
(sync-util/fetch-json url opts {:response-schema schema}))
|
||||
|
||||
(defonce ^:private *import-state (atom nil))
|
||||
(def ^:private snapshot-import-datoms-batch-size 10000)
|
||||
|
||||
(defn complete-datoms-import!
|
||||
[repo graph-id remote-tx]
|
||||
(-> (p/do!
|
||||
(when-let [search-db (worker-state/get-sqlite-conn repo :search)]
|
||||
(search/truncate-table! search-db))
|
||||
(rtc-log-and-state/rtc-log :rtc.log/download
|
||||
{:sub-type :download-progress
|
||||
:graph-uuid graph-id
|
||||
:message "Saving data to DB"})
|
||||
(->
|
||||
(if-let [rehydrate-f (@thread-api/*thread-apis :thread-api/db-sync-rehydrate-large-titles)]
|
||||
(rehydrate-f repo graph-id)
|
||||
(fail-fast :db-sync/missing-field {:field :thread-api/db-sync-rehydrate-large-titles}))
|
||||
(p/catch (fn [error]
|
||||
(log/error ::rehydrate-large-title-failed error))))
|
||||
(rtc-log-and-state/rtc-log :rtc.log/download
|
||||
{:sub-type :download-completed
|
||||
:graph-uuid graph-id
|
||||
:message "Graph is ready!"})
|
||||
(when-let [^js db (worker-state/get-sqlite-conn repo :db)]
|
||||
(.exec db "PRAGMA wal_checkpoint(TRUNCATE)"))
|
||||
(client-op/update-local-tx repo remote-tx)
|
||||
(shared-service/broadcast-to-clients! :add-repo {:repo repo}))
|
||||
(p/catch (fn [error]
|
||||
(js/console.error error)))))
|
||||
|
||||
(defn- require-thread-api-f!
|
||||
[k]
|
||||
(if-let [f (@thread-api/*thread-apis k)]
|
||||
f
|
||||
(fail-fast :db-sync/missing-field {:field k})))
|
||||
|
||||
(defn- stale-import-ex-info
|
||||
[repo graph-id import-id]
|
||||
(ex-info "stale db sync import"
|
||||
{:type :db-sync/stale-import
|
||||
:repo repo
|
||||
:graph-id graph-id
|
||||
:import-id import-id}))
|
||||
|
||||
(defn- import-temp-pool-name
|
||||
[repo]
|
||||
(worker-util/get-pool-name (str "download-import-" repo)))
|
||||
|
||||
(defn- close-import-state!
|
||||
[{:keys [rows-db rows-pool]}]
|
||||
(when rows-db
|
||||
(try
|
||||
(.close rows-db)
|
||||
(catch :default _)))
|
||||
(when rows-pool
|
||||
(-> (platform/remove-storage-pool! (platform/current) rows-pool)
|
||||
(p/catch (fn [_] nil)))))
|
||||
|
||||
(defn close-import-state-for-repo!
|
||||
[repo]
|
||||
(when-let [state @*import-state]
|
||||
(when (= repo (:repo state))
|
||||
(close-import-state! state)
|
||||
(reset! *import-state nil))))
|
||||
|
||||
(defn- clear-import-state!
|
||||
[import-id]
|
||||
(when-let [state @*import-state]
|
||||
(when (= import-id (:import-id state))
|
||||
(close-import-state! state)
|
||||
(reset! *import-state nil))))
|
||||
|
||||
(defn- require-import-state!
|
||||
[repo graph-id import-id]
|
||||
(let [state @*import-state]
|
||||
(when-not (and state
|
||||
(= import-id (:import-id state))
|
||||
(or (nil? repo) (= repo (:repo state)))
|
||||
(= graph-id (:graph-id state)))
|
||||
(throw (stale-import-ex-info repo graph-id import-id)))
|
||||
state))
|
||||
|
||||
(defn- upsert-addr-content!
|
||||
[^js db data]
|
||||
(.transaction
|
||||
db
|
||||
(fn [tx]
|
||||
(doseq [item data]
|
||||
(.exec tx #js {:sql "INSERT INTO kvs (addr, content, addresses) values ($addr, $content, $addresses) on conflict(addr) do update set content = $content, addresses = $addresses"
|
||||
:bind item})))))
|
||||
|
||||
(defn import-rows-batch!
|
||||
[{:keys [rows-db]} rows]
|
||||
(when-not rows-db
|
||||
(throw (ex-info "missing import rows db"
|
||||
{:type :db-sync/missing-field
|
||||
:field :rows-db})))
|
||||
(let [data (map (fn [[addr content addresses]]
|
||||
#js {:$addr addr
|
||||
:$content content
|
||||
:$addresses addresses})
|
||||
rows)]
|
||||
(upsert-addr-content! rows-db data))
|
||||
(count rows))
|
||||
|
||||
(defn- <create-import-temp-db!
|
||||
[repo]
|
||||
(if-let [sqlite @worker-state/*sqlite]
|
||||
(let [current-platform (platform/current)
|
||||
pool-name (import-temp-pool-name repo)]
|
||||
(p/let [pool (platform/install-storage-pool current-platform sqlite pool-name)
|
||||
path (platform/resolve-db-path current-platform pool-name pool "/download-import.sqlite")
|
||||
db (platform/sqlite-open current-platform
|
||||
{:sqlite sqlite
|
||||
:pool pool
|
||||
:path path
|
||||
:mode "c"})]
|
||||
(common-sqlite/create-kvs-table! db)
|
||||
{:rows-db db
|
||||
:rows-path path
|
||||
:rows-pool pool}))
|
||||
(fail-fast :db-sync/missing-field {:repo repo :field :sqlite})))
|
||||
|
||||
(defn- <ensure-import-rows-db!
|
||||
[{:keys [import-id repo rows-db] :as state}]
|
||||
(if rows-db
|
||||
(p/resolved state)
|
||||
(p/let [{:keys [rows-db rows-path rows-pool]} (<create-import-temp-db! repo)]
|
||||
(swap! *import-state
|
||||
(fn [current]
|
||||
(if (= import-id (:import-id current))
|
||||
(assoc current
|
||||
:rows-db rows-db
|
||||
:rows-path rows-path
|
||||
:rows-pool rows-pool)
|
||||
current)))
|
||||
(assoc state
|
||||
:rows-db rows-db
|
||||
:rows-path rows-path
|
||||
:rows-pool rows-pool))))
|
||||
|
||||
(defn- datom->tx
|
||||
[{:keys [e a v]}]
|
||||
[:db/add e a v])
|
||||
|
||||
(def ^:private snapshot-local-only-attrs
|
||||
#{:block/tx-id})
|
||||
|
||||
(defn- import-datoms-batch!
|
||||
[conn aes-key graph-e2ee? datoms]
|
||||
(p/let [datoms-batch (if graph-e2ee?
|
||||
(sync-crypt/<decrypt-snapshot-datoms-batch aes-key datoms)
|
||||
datoms)
|
||||
datoms-batch (remove #(contains? snapshot-local-only-attrs (:a %))
|
||||
datoms-batch)
|
||||
block-eids (into #{}
|
||||
(comp (filter #(= :block/uuid (:a %)))
|
||||
(map :e))
|
||||
datoms-batch)
|
||||
schema-tx-data (into [] (comp (filter #(= "db" (namespace (:a %))))
|
||||
(map datom->tx))
|
||||
datoms-batch)
|
||||
regular-tx-data (into [] (comp (remove #(= "db" (namespace (:a %))))
|
||||
(map datom->tx))
|
||||
datoms-batch)
|
||||
tx-id (inc (:max-tx @conn))
|
||||
tx-data (into (into schema-tx-data regular-tx-data)
|
||||
(map (fn [block-eid]
|
||||
[:db/add block-eid :block/tx-id tx-id]))
|
||||
block-eids)]
|
||||
(when (seq tx-data)
|
||||
(d/transact! conn tx-data {:sync-download-graph? true}))))
|
||||
|
||||
(defn- schema-datom?
|
||||
[ident-eids schema-version-eid datom]
|
||||
(or (= schema-version-eid (:e datom))
|
||||
(and (contains? ident-eids (:e datom))
|
||||
(or (= :db/ident (:a datom))
|
||||
(= "db" (namespace (:a datom)))))))
|
||||
|
||||
(defn snapshot-datoms-in-import-order
|
||||
[conn]
|
||||
(let [db @conn
|
||||
schema-version-eid (some-> (d/entity db :logseq.kv/schema-version) :db/id)
|
||||
ident-eids (into #{}
|
||||
(map :e)
|
||||
(d/datoms db :avet :db/ident))
|
||||
schema-datom?* #(schema-datom? ident-eids schema-version-eid %)
|
||||
ordered-datoms (fn [pred]
|
||||
(sequence
|
||||
(comp (filter pred)
|
||||
(map #(select-keys % [:e :a :v])))
|
||||
(d/datoms db :eavt)))]
|
||||
(concat (ordered-datoms schema-datom?*)
|
||||
(ordered-datoms #(not (schema-datom?* %))))))
|
||||
|
||||
(defn- take-import-datoms-batch
|
||||
[datoms batch-size]
|
||||
(loop [batch (transient [])
|
||||
remaining (seq datoms)
|
||||
n 0]
|
||||
(if (or (nil? remaining)
|
||||
(>= n batch-size))
|
||||
[(persistent! batch) remaining]
|
||||
(recur (conj! batch (first remaining))
|
||||
(next remaining)
|
||||
(inc n)))))
|
||||
|
||||
(defn- <yield-next-tick
|
||||
[]
|
||||
(js/Promise. (fn [resolve] (js/setTimeout resolve 0))))
|
||||
|
||||
(defn- log-import-progress!
|
||||
[graph-id import-id datoms-count]
|
||||
(when (pos? datoms-count)
|
||||
(let [{:keys [imported-datoms total-datoms]}
|
||||
(swap! *import-state
|
||||
(fn [state]
|
||||
(if (= import-id (:import-id state))
|
||||
(update state :imported-datoms (fnil + 0) datoms-count)
|
||||
state)))]
|
||||
(rtc-log-and-state/rtc-log :rtc.log/download
|
||||
{:sub-type :download-progress
|
||||
:graph-uuid graph-id
|
||||
:message (if (some? total-datoms)
|
||||
(str "Importing data " imported-datoms "/" total-datoms)
|
||||
(str "Importing data " imported-datoms))}))))
|
||||
|
||||
(defn- <replay-imported-rows!
|
||||
[{:keys [conn rows-db aes-key graph-e2ee? graph-id import-id]}]
|
||||
(if (nil? rows-db)
|
||||
(p/resolved nil)
|
||||
(let [source-storage (sync-temp-sqlite/new-temp-sqlite-storage rows-db)
|
||||
source-conn (common-sqlite/get-storage-conn source-storage db-schema/schema)]
|
||||
(p/loop [remaining (seq (snapshot-datoms-in-import-order source-conn))]
|
||||
(if (seq remaining)
|
||||
(let [[batch remaining'] (take-import-datoms-batch remaining snapshot-import-datoms-batch-size)]
|
||||
(p/let [_ (import-datoms-batch! conn aes-key graph-e2ee? batch)
|
||||
_ (log-import-progress! graph-id import-id (count batch))
|
||||
_ (<yield-next-tick)]
|
||||
(p/recur remaining')))
|
||||
(p/resolved nil))))))
|
||||
|
||||
(defn prepare-import!
|
||||
[repo reset? graph-id graph-e2ee? & [total-datoms]]
|
||||
(let [graph-e2ee? (if (nil? graph-e2ee?) true (true? graph-e2ee?))]
|
||||
(-> (p/let [close-db-f (require-thread-api-f! :thread-api/db-sync-close-db)
|
||||
unlink-db-f (require-thread-api-f! :thread-api/unsafe-unlink-db)
|
||||
recreate-lock-f (require-thread-api-f! :thread-api/db-sync-recreate-lock)
|
||||
invalidate-search-db-f (require-thread-api-f! :thread-api/db-sync-invalidate-search-db)
|
||||
create-or-open-db-f (require-thread-api-f! :thread-api/create-or-open-db)
|
||||
_ (when-let [state @*import-state]
|
||||
(close-import-state! state)
|
||||
(close-db-f (:repo state)))
|
||||
_ (reset! *import-state nil)
|
||||
_ (when reset? (close-db-f repo))
|
||||
_ (when reset? (unlink-db-f repo))
|
||||
_ (when reset? (recreate-lock-f repo))
|
||||
_ (when reset? (invalidate-search-db-f repo))
|
||||
import-id (str (random-uuid))
|
||||
aes-key (when graph-e2ee?
|
||||
(sync-crypt/<fetch-graph-aes-key-for-download graph-id))
|
||||
_ (when (and graph-e2ee? (nil? aes-key))
|
||||
(fail-fast :db-sync/missing-field {:repo repo :field :aes-key}))
|
||||
_ (create-or-open-db-f repo {:close-other-db? true
|
||||
:sync-download-graph? true})
|
||||
conn (worker-state/get-datascript-conn repo)
|
||||
_ (when-not conn
|
||||
(fail-fast :db-sync/missing-field {:repo repo :field :datascript-conn}))]
|
||||
(reset! *import-state {:aes-key aes-key
|
||||
:conn conn
|
||||
:graph-e2ee? graph-e2ee?
|
||||
:graph-id graph-id
|
||||
:import-id import-id
|
||||
:imported-datoms 0
|
||||
:rows-db nil
|
||||
:rows-imported? false
|
||||
:rows-path nil
|
||||
:rows-pool nil
|
||||
:repo repo
|
||||
:total-datoms total-datoms})
|
||||
{:import-id import-id})
|
||||
(p/catch (fn [error]
|
||||
(reset! *import-state nil)
|
||||
(throw error))))))
|
||||
|
||||
(defn import-rows-chunk!
|
||||
[rows graph-id import-id]
|
||||
(-> (p/let [state (require-import-state! nil graph-id import-id)
|
||||
state (<ensure-import-rows-db! state)
|
||||
_ (import-rows-batch! state rows)
|
||||
_ (swap! *import-state
|
||||
(fn [current]
|
||||
(if (= import-id (:import-id current))
|
||||
(assoc current :rows-imported? true)
|
||||
current)))]
|
||||
true)
|
||||
(p/catch (fn [error]
|
||||
(when-not (= :db-sync/stale-import (:type (ex-data error)))
|
||||
(clear-import-state! import-id))
|
||||
(throw error)))))
|
||||
|
||||
(defn finalize-import!
|
||||
[repo graph-id remote-tx import-id]
|
||||
(-> (p/let [state (require-import-state! repo graph-id import-id)
|
||||
_ (when (:rows-imported? state)
|
||||
(<replay-imported-rows! state))
|
||||
result (complete-datoms-import! repo graph-id remote-tx)
|
||||
_ (clear-import-state! import-id)]
|
||||
result)
|
||||
(p/catch (fn [error]
|
||||
(when-not (= :db-sync/stale-import (:type (ex-data error)))
|
||||
(clear-import-state! import-id))
|
||||
(throw error)))))
|
||||
|
||||
(defn- set-graph-sync-metadata!
|
||||
[conn graph-id graph-e2ee?]
|
||||
(assert (uuid? graph-id))
|
||||
(ldb/transact! conn [(ldb/kv :logseq.kv/graph-uuid graph-id)
|
||||
(ldb/kv :logseq.kv/graph-remote? true)
|
||||
(ldb/kv :logseq.kv/graph-rtc-e2ee? (true? graph-e2ee?))]
|
||||
{:persist-op? false}))
|
||||
|
||||
(defn download-graph-by-id!
|
||||
[repo graph-id graph-e2ee?]
|
||||
(let [base (sync-auth/http-base-url @worker-state/*db-sync-config)]
|
||||
(if (and (seq repo) (seq graph-id) (seq base))
|
||||
(let [stage* (atom :init)
|
||||
import-id* (atom nil)
|
||||
log-f (fn [payload]
|
||||
(rtc-log-and-state/rtc-log :rtc.log/download payload))]
|
||||
(-> (p/let [_ (log-f {:sub-type :download-progress
|
||||
:graph-uuid graph-id
|
||||
:message "Preparing graph snapshot download"})
|
||||
_ (reset! stage* :fetch-pull)
|
||||
pull-resp (fetch-json (str base "/sync/" graph-id "/pull")
|
||||
{:method "GET"}
|
||||
:sync/pull)
|
||||
remote-tx (:t pull-resp)
|
||||
_ (when-not (integer? remote-tx)
|
||||
(throw (ex-info "non-integer remote-tx when downloading graph"
|
||||
{:repo repo
|
||||
:remote-tx remote-tx})))
|
||||
_ (reset! stage* :fetch-snapshot-download)
|
||||
snapshot-resp (fetch-json (str base "/sync/" graph-id "/snapshot/download")
|
||||
{:method "GET"}
|
||||
:sync/snapshot-download)
|
||||
_ (when graph-e2ee?
|
||||
(reset! stage* :prepare-e2ee)
|
||||
(sync-crypt/<fetch-graph-aes-key-for-download graph-id))
|
||||
_ (reset! stage* :fetch-snapshot-stream)
|
||||
resp (js/fetch (:url snapshot-resp)
|
||||
(clj->js (with-auth-headers {:method "GET"})))
|
||||
_ (log-f {:sub-type :download-progress
|
||||
:graph-uuid graph-id
|
||||
:message "Start downloading graph snapshot"})]
|
||||
(when-not (.-ok resp)
|
||||
(throw (ex-info "snapshot download failed"
|
||||
{:repo repo
|
||||
:status (.-status resp)})))
|
||||
(let [ensure-import! (fn []
|
||||
(if-let [import-id @import-id*]
|
||||
(p/resolved import-id)
|
||||
(p/let [_ (reset! stage* :prepare-import)
|
||||
{:keys [import-id]} (prepare-import! repo true graph-id graph-e2ee?)]
|
||||
(reset! import-id* import-id)
|
||||
import-id)))]
|
||||
(p/let [_ (do
|
||||
(reset! stage* :stream-snapshot)
|
||||
(<stream-snapshot-row-batches!
|
||||
resp
|
||||
25000
|
||||
(fn [rows]
|
||||
(p/let [import-id (ensure-import!)]
|
||||
(import-rows-chunk! rows graph-id import-id)))))
|
||||
_ (log-f {:sub-type :download-completed
|
||||
:graph-uuid graph-id
|
||||
:message "Graph snapshot downloaded"})
|
||||
_ (when-let [import-id @import-id*]
|
||||
(reset! stage* :finalize-import)
|
||||
(finalize-import! repo graph-id remote-tx import-id))]
|
||||
(when-let [conn (worker-state/get-datascript-conn repo)]
|
||||
(set-graph-sync-metadata! conn (uuid graph-id) graph-e2ee?))
|
||||
{:repo repo
|
||||
:graph-id graph-id
|
||||
:remote-tx remote-tx
|
||||
:graph-e2ee? graph-e2ee?})))
|
||||
(p/catch (fn [error]
|
||||
(when-let [import-id @import-id*]
|
||||
(clear-import-state! import-id))
|
||||
(log-f {:sub-type :download-completed
|
||||
:graph-uuid graph-id
|
||||
:message "Graph snapshot download failed"})
|
||||
(log/error :db-sync/download-graph-by-id-failed
|
||||
{:repo repo
|
||||
:graph-id graph-id
|
||||
:graph-e2ee? graph-e2ee?
|
||||
:stage @stage*
|
||||
:error error
|
||||
:error-stack (when (instance? js/Error error)
|
||||
(.-stack error))
|
||||
:error-cause (when (instance? js/Error error)
|
||||
(some-> (.-cause error) (.-message)))})
|
||||
(throw (ex-info "db-sync download failed"
|
||||
{:repo repo
|
||||
:graph-id graph-id
|
||||
:graph-e2ee? graph-e2ee?
|
||||
:stage @stage*
|
||||
:code (:code (ex-data error))
|
||||
:error-message (or (ex-message error)
|
||||
(when (instance? js/Error error)
|
||||
(.-message error)))
|
||||
:error-cause (when (instance? js/Error error)
|
||||
(some-> (.-cause error) (.-message)))}
|
||||
error))))))
|
||||
(p/rejected (ex-info "db-sync missing graph download info"
|
||||
{:repo repo
|
||||
:graph-id graph-id
|
||||
:base base})))))
|
||||
@@ -1,391 +0,0 @@
|
||||
(ns frontend.worker.sync.handle-message
|
||||
"WebSocket message handlers for db sync."
|
||||
(:require [frontend.worker.shared-service :as shared-service]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync.apply-txs :as sync-apply]
|
||||
[frontend.worker.sync.assets :as sync-assets]
|
||||
[frontend.worker.sync.auth :as sync-auth]
|
||||
[frontend.worker.sync.client-op :as client-op]
|
||||
[frontend.worker.sync.crypt :as sync-crypt]
|
||||
[frontend.worker.sync.log-and-state :as sync-log-state]
|
||||
[frontend.worker.sync.presence :as sync-presence]
|
||||
[frontend.worker.sync.transport :as sync-transport]
|
||||
[frontend.worker.sync.util :as sync-util]
|
||||
[lambdaisland.glogi :as log]
|
||||
[promesa.core :as p]
|
||||
[frontend.worker-common.util :as worker-util]))
|
||||
|
||||
(defn- fail-fast
|
||||
[tag data]
|
||||
(log/error tag data)
|
||||
(throw (ex-info (name tag) data)))
|
||||
|
||||
(defn- sync-counts
|
||||
[repo]
|
||||
(sync-presence/sync-counts
|
||||
{:get-datascript-conn worker-state/get-datascript-conn
|
||||
:get-client-ops-conn worker-state/get-client-ops-conn
|
||||
:get-pending-local-tx-count client-op/get-pending-local-tx-count
|
||||
:get-unpushed-asset-ops-count client-op/get-unpushed-asset-ops-count
|
||||
:get-missing-asset-upload-files sync-assets/get-missing-asset-upload-files
|
||||
:get-local-tx client-op/get-local-tx
|
||||
:get-local-checksum client-op/get-local-checksum
|
||||
:get-graph-uuid client-op/get-graph-uuid
|
||||
:latest-remote-tx @sync-apply/*repo->latest-remote-tx
|
||||
:latest-remote-checksum @sync-apply/*repo->latest-remote-checksum}
|
||||
repo))
|
||||
|
||||
(defn- broadcast-rtc-state!
|
||||
[client]
|
||||
(when client
|
||||
(shared-service/broadcast-to-clients!
|
||||
:rtc-sync-state
|
||||
(sync-presence/rtc-state-payload sync-counts client))))
|
||||
|
||||
(defn- update-online-users!
|
||||
[client users]
|
||||
(sync-presence/update-online-users! broadcast-rtc-state! client users))
|
||||
|
||||
(defn- update-user-presence!
|
||||
[client user-id* editing-block-uuid]
|
||||
(sync-presence/update-user-presence! broadcast-rtc-state! client user-id* editing-block-uuid))
|
||||
|
||||
(defn- get-user-uuid
|
||||
[]
|
||||
(sync-auth/get-user-uuid (worker-state/get-id-token)))
|
||||
|
||||
(defn- send!
|
||||
[ws message]
|
||||
(sync-transport/send! sync-transport/coerce-ws-client-message ws message))
|
||||
|
||||
(defn- ws-open?
|
||||
[ws]
|
||||
(sync-transport/ws-open? ws))
|
||||
|
||||
(defn- enqueue-asset-task!
|
||||
[client task]
|
||||
(when-let [queue (:asset-queue client)]
|
||||
(swap! queue
|
||||
(fn [prev]
|
||||
(p/then prev (fn [_] (task)))))))
|
||||
|
||||
(defn- enqueue-send-task!
|
||||
[client task]
|
||||
(if-let [queue (:send-queue client)]
|
||||
(swap! queue
|
||||
(fn [prev]
|
||||
(-> (or prev (p/resolved nil))
|
||||
(p/catch (fn [_] nil))
|
||||
(p/then (fn [_] (task)))
|
||||
(p/catch (fn [error]
|
||||
(log/error :db-sync/send-queue-task-failed
|
||||
{:repo (:repo client)
|
||||
:error error}))))))
|
||||
(task)))
|
||||
|
||||
(defn- current-client
|
||||
[repo]
|
||||
(sync-presence/current-client worker-state/*db-sync-client repo))
|
||||
|
||||
(defn- require-number
|
||||
[value context]
|
||||
(when-not (number? value)
|
||||
(fail-fast :db-sync/invalid-field (assoc context :value value))))
|
||||
|
||||
(defn- require-non-negative
|
||||
[value context]
|
||||
(require-number value context)
|
||||
(when (neg? value)
|
||||
(fail-fast :db-sync/invalid-field (assoc context :value value))))
|
||||
|
||||
(defn- require-seq
|
||||
[value context]
|
||||
(when-not (sequential? value)
|
||||
(fail-fast :db-sync/invalid-field (assoc context :value value))))
|
||||
|
||||
(defn- require-uuid
|
||||
[value context]
|
||||
(when-not (uuid? value)
|
||||
(fail-fast :db-sync/invalid-field (assoc context :value value))))
|
||||
|
||||
(defn- parse-transit
|
||||
[value context]
|
||||
(sync-transport/parse-transit fail-fast value context))
|
||||
|
||||
(defn- request-pull!
|
||||
[client since]
|
||||
(when (and (:ws client) (ws-open? (:ws client)))
|
||||
(enqueue-send-task!
|
||||
client
|
||||
(fn []
|
||||
(when (and (:ws client) (ws-open? (:ws client)))
|
||||
(if-let [*pending (:pending-pull-since client)]
|
||||
(let [pending @*pending]
|
||||
(when (or (nil? pending) (< since pending))
|
||||
(reset! *pending since)
|
||||
(send! (:ws client) {:type "pull" :since since})))
|
||||
(send! (:ws client) {:type "pull" :since since})))))))
|
||||
|
||||
(defn- clear-pending-pull!
|
||||
[client]
|
||||
(when-let [*pending (:pending-pull-since client)]
|
||||
(reset! *pending nil)))
|
||||
|
||||
(defn- pending-local-tx?
|
||||
[repo]
|
||||
(pos? (or (client-op/get-pending-local-tx-count repo) 0)))
|
||||
|
||||
(defn- synced-checksum-ready?
|
||||
[repo client local-t remote-t]
|
||||
(and (= local-t remote-t)
|
||||
(not (pending-local-tx? repo))
|
||||
(empty? @(:inflight client))))
|
||||
|
||||
(defn- checksum-compare-ready?
|
||||
[repo client local-t remote-t]
|
||||
(and (synced-checksum-ready? repo client local-t remote-t)
|
||||
(string? (client-op/get-local-checksum repo))))
|
||||
|
||||
(defn- verify-sync-checksum!
|
||||
[repo client local-tx remote-tx remote-checksum context]
|
||||
(when worker-util/dev-or-test?
|
||||
(when (and (string? remote-checksum)
|
||||
(checksum-compare-ready? repo client local-tx remote-tx))
|
||||
(let [local-checksum (client-op/get-local-checksum repo)]
|
||||
(when-not (= local-checksum remote-checksum)
|
||||
(let [mismatch-data (merge context
|
||||
{:type :db-sync/checksum-mismatch
|
||||
:repo repo
|
||||
:message-type (:type context)
|
||||
:local-tx local-tx
|
||||
:remote-tx remote-tx
|
||||
:local-checksum local-checksum
|
||||
:remote-checksum remote-checksum})]
|
||||
(sync-log-state/rtc-log :rtc.log/checksum-mismatch mismatch-data)
|
||||
(log/warn :db-sync/checksum-mismatch mismatch-data)))))))
|
||||
|
||||
(defn- handle-tx-reject!
|
||||
[repo client message local-tx]
|
||||
(sync-apply/clear-upload-response-timeout! client)
|
||||
(let [reason (:reason message)
|
||||
remote-tx (:t message)
|
||||
success-tx-ids (:success-tx-ids message)
|
||||
failed-tx-id (:failed-tx-id message)
|
||||
missing-block-uuids (:missing-block-uuids message)]
|
||||
(when (nil? reason)
|
||||
(fail-fast :db-sync/missing-field
|
||||
{:repo repo :type "tx/reject" :field :reason}))
|
||||
(when (contains? message :t)
|
||||
(require-non-negative remote-tx {:repo repo :type "tx/reject"}))
|
||||
(when (contains? message :success-tx-ids)
|
||||
(require-seq success-tx-ids {:repo repo :type "tx/reject" :field :success-tx-ids})
|
||||
(doseq [tx-id success-tx-ids]
|
||||
(require-uuid tx-id {:repo repo :type "tx/reject" :field :success-tx-ids})))
|
||||
(when (contains? message :failed-tx-id)
|
||||
(require-uuid failed-tx-id {:repo repo :type "tx/reject" :field :failed-tx-id}))
|
||||
(when (contains? message :missing-block-uuids)
|
||||
(require-seq missing-block-uuids {:repo repo :type "tx/reject" :field :missing-block-uuids})
|
||||
(doseq [block-uuid missing-block-uuids]
|
||||
(require-uuid block-uuid {:repo repo :type "tx/reject" :field :missing-block-uuids})))
|
||||
(case reason
|
||||
"stale"
|
||||
(request-pull! client local-tx)
|
||||
|
||||
(let [inflight @(:inflight client)
|
||||
inflight-set (set inflight)
|
||||
successful-tx-ids (->> (or success-tx-ids [])
|
||||
(filter inflight-set)
|
||||
vec)
|
||||
failed-tx-id (when (and failed-tx-id (contains? inflight-set failed-tx-id))
|
||||
failed-tx-id)
|
||||
data (when-let [raw-data (:data message)]
|
||||
(parse-transit raw-data
|
||||
{:repo repo
|
||||
:type "tx/reject"
|
||||
:reason reason
|
||||
:field :data}))
|
||||
rejected-data (cond-> {:type :db-sync/tx-rejected
|
||||
:repo repo
|
||||
:message-type "tx/reject"
|
||||
:reason reason}
|
||||
(contains? message :t) (assoc :t remote-tx)
|
||||
(seq successful-tx-ids) (assoc :success-tx-ids successful-tx-ids)
|
||||
(some? failed-tx-id) (assoc :failed-tx-id failed-tx-id)
|
||||
(seq missing-block-uuids) (assoc :missing-block-uuids (vec missing-block-uuids))
|
||||
(some? data) (assoc :data data))]
|
||||
(if (or (contains? message :success-tx-ids)
|
||||
(contains? message :failed-tx-id))
|
||||
(do
|
||||
(sync-apply/mark-pending-txs-false! repo successful-tx-ids)
|
||||
(when failed-tx-id
|
||||
(sync-apply/rollback-and-mark-failed-txs! repo [failed-tx-id])))
|
||||
;; Backward compatibility for older servers without per-tx reject metadata.
|
||||
(sync-apply/rollback-and-mark-failed-txs! repo inflight))
|
||||
(reset! (:inflight client) [])
|
||||
(broadcast-rtc-state! client)
|
||||
(sync-log-state/rtc-log :rtc.log/tx-rejected rejected-data)
|
||||
(fail-fast :db-sync/tx-rejected
|
||||
rejected-data)))))
|
||||
|
||||
(defn- handle-hello!
|
||||
[repo client local-tx remote-tx remote-checksum]
|
||||
(require-non-negative remote-tx {:repo repo :type "hello"})
|
||||
(verify-sync-checksum! repo client local-tx remote-tx remote-checksum {:type "hello"})
|
||||
(broadcast-rtc-state! client)
|
||||
(when (> remote-tx local-tx)
|
||||
(request-pull! client local-tx))
|
||||
(sync-assets/enqueue-asset-sync!
|
||||
repo client
|
||||
{:enqueue-asset-task-f enqueue-asset-task!
|
||||
:current-client-f current-client
|
||||
:broadcast-rtc-state!-f broadcast-rtc-state!
|
||||
:fail-fast-f fail-fast})
|
||||
(log/info :db-sync/handle-hello
|
||||
{:empty-inflight? (empty? @(:inflight client))
|
||||
:online? (worker-state/online?)
|
||||
:ws-open? (when-let [ws (:ws client)]
|
||||
(ws-open? ws))
|
||||
:pending-txs-count (count (sync-apply/pending-txs repo {:limit 50}))})
|
||||
(sync-apply/enqueue-flush-pending! repo client))
|
||||
|
||||
(defn- handle-online-users!
|
||||
[repo client message]
|
||||
(let [users (:online-users message)]
|
||||
(when (and (some? users) (not (sequential? users)))
|
||||
(fail-fast :db-sync/invalid-field
|
||||
{:repo repo :type "online-users" :field :online-users}))
|
||||
(update-online-users! client (or users []))))
|
||||
|
||||
(defn- handle-presence!
|
||||
[client message]
|
||||
(let [{:keys [user-id editing-block-uuid]} message]
|
||||
(when-not (= (get-user-uuid) user-id)
|
||||
(update-user-presence! client user-id editing-block-uuid))))
|
||||
|
||||
(defn- handle-tx-batch-ok!
|
||||
[repo client remote-tx remote-checksum]
|
||||
(require-non-negative remote-tx {:repo repo :type "tx/batch/ok"})
|
||||
(sync-apply/ack-upload-response! repo client)
|
||||
(let [current-local-tx (client-op/get-local-tx repo)
|
||||
next-local-tx (max current-local-tx remote-tx)]
|
||||
(client-op/update-local-tx repo next-local-tx)
|
||||
(sync-util/clear-last-sync-error! client)
|
||||
(sync-apply/mark-pending-txs-false! repo @(:inflight client))
|
||||
(reset! (:inflight client) [])
|
||||
(broadcast-rtc-state! client)
|
||||
(verify-sync-checksum! repo client next-local-tx remote-tx remote-checksum {:type "tx/batch/ok"})
|
||||
(sync-apply/enqueue-flush-pending! repo client)))
|
||||
|
||||
(defn- update-latest-remote-state!
|
||||
[repo message]
|
||||
(let [message-type (:type message)
|
||||
remote-tx (:t message)
|
||||
remote-checksum (:checksum message)
|
||||
has-checksum? (contains? message :checksum)
|
||||
latest-remote-tx (get @sync-apply/*repo->latest-remote-tx repo)
|
||||
authoritative? (contains? #{"hello" "changed"} message-type)
|
||||
stale-remote-tx? (and (number? remote-tx)
|
||||
(number? latest-remote-tx)
|
||||
(< remote-tx latest-remote-tx)
|
||||
(not authoritative?))]
|
||||
(when (number? remote-tx)
|
||||
(if authoritative?
|
||||
(swap! sync-apply/*repo->latest-remote-tx assoc repo remote-tx)
|
||||
(swap! sync-apply/*repo->latest-remote-tx
|
||||
update repo
|
||||
(fn [prev]
|
||||
(if (number? prev)
|
||||
(max prev remote-tx)
|
||||
remote-tx)))))
|
||||
(when (and has-checksum? (not stale-remote-tx?))
|
||||
(swap! sync-apply/*repo->latest-remote-checksum assoc repo remote-checksum))
|
||||
{:stale-remote-tx? stale-remote-tx?
|
||||
:latest-remote-tx-before latest-remote-tx}))
|
||||
|
||||
(declare handle-pull-ok! handle-changed!)
|
||||
|
||||
(defn- validate-local-tx!
|
||||
[repo message local-tx]
|
||||
(let [message-type (:type message)]
|
||||
(when (contains? #{"hello" "tx/batch/ok" "pull/ok" "changed" "tx/reject"} message-type)
|
||||
(let [validate? (and (integer? local-tx) (>= local-tx 0))]
|
||||
(when-not validate?
|
||||
(throw (ex-info "Invalid local tx"
|
||||
{:repo repo
|
||||
:message-type message-type
|
||||
:local-tx local-tx})))))))
|
||||
(defn handle-message!
|
||||
[repo client raw]
|
||||
(let [message (-> raw
|
||||
sync-transport/parse-message
|
||||
sync-transport/coerce-ws-server-message)]
|
||||
(when-not (map? message)
|
||||
(fail-fast :db-sync/response-parse-failed {:repo repo :raw raw}))
|
||||
(let [local-tx (client-op/get-local-tx repo)
|
||||
remote-tx (:t message)
|
||||
remote-checksum (:checksum message)]
|
||||
(validate-local-tx! repo message local-tx)
|
||||
(update-latest-remote-state! repo message)
|
||||
(case (:type message)
|
||||
"hello" (handle-hello! repo client local-tx remote-tx remote-checksum)
|
||||
"online-users" (handle-online-users! repo client message)
|
||||
"presence" (handle-presence! client message)
|
||||
"tx/batch/ok" (handle-tx-batch-ok! repo client remote-tx remote-checksum)
|
||||
"pull/ok" (handle-pull-ok! repo client local-tx remote-tx remote-checksum message)
|
||||
"changed" (handle-changed! repo client local-tx remote-tx)
|
||||
"tx/reject" (handle-tx-reject! repo client message local-tx)
|
||||
"pong" nil
|
||||
(fail-fast :db-sync/invalid-field
|
||||
{:repo repo :type (:type message)})))))
|
||||
|
||||
(defn- handle-pull-ok!
|
||||
[repo client local-tx remote-tx remote-checksum message]
|
||||
(clear-pending-pull! client)
|
||||
;; (log/info ::handle-pull-ok! {:repo repo
|
||||
;; :remote-tx remote-tx
|
||||
;; :local-tx local-tx})
|
||||
(when (> remote-tx local-tx)
|
||||
(let [txs (:txs message)]
|
||||
(require-non-negative remote-tx {:repo repo :type "pull/ok"})
|
||||
(require-seq txs {:repo repo :type "pull/ok" :field :txs})
|
||||
(let [remote-txs (mapv (fn [data]
|
||||
{:t (:t data)
|
||||
:outliner-op (:outliner-op data)
|
||||
:tx-data (parse-transit (:tx data) {:repo repo :type "pull/ok"})})
|
||||
txs)]
|
||||
;; (log/info ::handle-pull-remote-txs-count {:count (count remote-txs)})
|
||||
(when (seq remote-txs)
|
||||
(->
|
||||
(p/let [graph-e2ee? (sync-crypt/graph-e2ee? repo)
|
||||
;; _ (log/info ::handle-pull-request-aes-key {})
|
||||
aes-key (sync-crypt/<ensure-graph-aes-key repo (:graph-id client))
|
||||
;; _ (when (some? aes-key)
|
||||
;; (log/info ::handle-pull-request-aes-key-success {}))
|
||||
_ (when (and graph-e2ee? (nil? aes-key))
|
||||
(fail-fast :db-sync/missing-field {:repo repo :field :aes-key}))
|
||||
remote-txs* (if aes-key
|
||||
(p/all (mapv (fn [{:keys [tx-data] :as remote-tx}]
|
||||
(p/let [tx-data* (sync-crypt/<decrypt-tx-data aes-key tx-data)]
|
||||
(assoc remote-tx :tx-data tx-data*)))
|
||||
remote-txs))
|
||||
(p/resolved remote-txs))
|
||||
_ (try
|
||||
(sync-apply/apply-remote-txs! repo client remote-txs*)
|
||||
(catch :default e
|
||||
(log/error ::apply-remote-tx e)
|
||||
(throw e)))]
|
||||
(client-op/update-local-tx repo remote-tx)
|
||||
(broadcast-rtc-state! client)
|
||||
(verify-sync-checksum! repo client remote-tx remote-tx remote-checksum {:type "pull/ok"})
|
||||
(sync-apply/enqueue-flush-pending! repo client))
|
||||
(p/then (fn [_]
|
||||
(sync-util/clear-last-sync-error! client)))
|
||||
(p/catch (fn [error]
|
||||
(sync-util/set-last-sync-error! client error)))))))))
|
||||
|
||||
(defn- handle-changed!
|
||||
[repo client local-tx remote-tx]
|
||||
(require-non-negative remote-tx {:repo repo :type "changed"})
|
||||
(broadcast-rtc-state! client)
|
||||
(when (< local-tx remote-tx)
|
||||
(request-pull! client local-tx)))
|
||||
@@ -1,267 +0,0 @@
|
||||
(ns frontend.worker.sync.large-title
|
||||
"Large title offload and rehydration helpers for db sync."
|
||||
(:require [datascript.core :as d]
|
||||
[logseq.db :as ldb]
|
||||
[promesa.core :as p]
|
||||
[frontend.worker.sync.util :refer [get-graph-id]]))
|
||||
|
||||
(def large-title-byte-limit 4096)
|
||||
(def large-title-asset-type "txt")
|
||||
(def large-title-object-attr :logseq.property.sync/large-title-object)
|
||||
(def text-encoder (js/TextEncoder.))
|
||||
(def text-decoder (js/TextDecoder.))
|
||||
|
||||
(defn utf8-byte-length
|
||||
[value]
|
||||
(when (string? value)
|
||||
(.-length (.encode text-encoder value))))
|
||||
|
||||
(defn large-title?
|
||||
[value]
|
||||
(when-let [byte-length (utf8-byte-length value)]
|
||||
(> byte-length large-title-byte-limit)))
|
||||
|
||||
(defn large-title-datom?
|
||||
[datom]
|
||||
(and (= :block/title (:a datom))
|
||||
(string? (:v datom))
|
||||
(large-title? (:v datom))))
|
||||
|
||||
(defn assoc-datom-value
|
||||
[datom new-value]
|
||||
(let [[op e a _v & others] datom]
|
||||
(into [op e a new-value] others)))
|
||||
|
||||
(defn large-title-object
|
||||
[asset-uuid asset-type]
|
||||
{:asset-uuid asset-uuid
|
||||
:asset-type asset-type})
|
||||
|
||||
(defn large-title-object?
|
||||
[value]
|
||||
(and (map? value)
|
||||
(string? (:asset-uuid value))
|
||||
(string? (:asset-type value))))
|
||||
|
||||
(defn- large-title-object-datoms
|
||||
[db]
|
||||
(if (get-in (:schema db) [large-title-object-attr :db/index])
|
||||
(d/datoms db :avet large-title-object-attr)
|
||||
(filter #(= large-title-object-attr (:a %))
|
||||
(d/datoms db :eavt))))
|
||||
|
||||
(defn- find-large-title-object-eid
|
||||
[db obj]
|
||||
(some (fn [datom]
|
||||
(when (= obj (:v datom))
|
||||
(:e datom)))
|
||||
(large-title-object-datoms db)))
|
||||
|
||||
(defn- resolve-large-title-item-eid
|
||||
[db {:keys [e obj]}]
|
||||
(or (when (number? e)
|
||||
e)
|
||||
(some-> (d/entity db e) :db/id)
|
||||
(find-large-title-object-eid db obj)))
|
||||
|
||||
(defn asset-url
|
||||
[base graph-id asset-uuid asset-type]
|
||||
(str base "/assets/" graph-id "/" asset-uuid "." asset-type))
|
||||
|
||||
(defn upload-large-title!
|
||||
[{:keys [repo
|
||||
graph-id
|
||||
title
|
||||
aes-key
|
||||
http-base
|
||||
auth-headers
|
||||
fail-fast-f
|
||||
encrypt-text-value-f]}]
|
||||
(when-not (seq http-base)
|
||||
(fail-fast-f :db-sync/missing-field {:repo repo :field :http-base}))
|
||||
(when-not (seq graph-id)
|
||||
(fail-fast-f :db-sync/missing-field {:repo repo :field :graph-id}))
|
||||
(let [asset-uuid (str (random-uuid))
|
||||
asset-type large-title-asset-type
|
||||
url (asset-url http-base graph-id asset-uuid asset-type)]
|
||||
(p/let [payload (if aes-key
|
||||
(p/let [payload-str (encrypt-text-value-f aes-key title)]
|
||||
(.encode text-encoder payload-str))
|
||||
(p/resolved title))
|
||||
headers (merge {"content-type" "text/plain; charset=utf-8"
|
||||
"x-amz-meta-type" asset-type}
|
||||
auth-headers)
|
||||
resp (js/fetch url #js {:method "PUT"
|
||||
:headers (clj->js headers)
|
||||
:body payload})]
|
||||
(if (.-ok resp)
|
||||
(large-title-object asset-uuid asset-type)
|
||||
(fail-fast-f :db-sync/large-title-upload-failed
|
||||
{:repo repo :status (.-status resp)})))))
|
||||
|
||||
(defn download-large-title!
|
||||
[{:keys [repo
|
||||
graph-id
|
||||
obj
|
||||
aes-key
|
||||
http-base
|
||||
auth-headers
|
||||
fail-fast-f
|
||||
decrypt-text-value-f]}]
|
||||
(when-not (seq http-base)
|
||||
(fail-fast-f :db-sync/missing-field {:repo repo :field :http-base}))
|
||||
(when-not (seq graph-id)
|
||||
(fail-fast-f :db-sync/missing-field {:repo repo :field :graph-id}))
|
||||
(let [url (asset-url http-base graph-id (:asset-uuid obj) (:asset-type obj))]
|
||||
(p/let [resp (js/fetch url #js {:method "GET"
|
||||
:headers (clj->js auth-headers)})]
|
||||
(when-not (.-ok resp)
|
||||
(fail-fast-f :db-sync/large-title-download-failed
|
||||
{:repo repo :status (.-status resp)}))
|
||||
(p/let [buf (.arrayBuffer resp)
|
||||
payload (js/Uint8Array. buf)
|
||||
payload-str (.decode text-decoder payload)
|
||||
data (if aes-key
|
||||
(-> (decrypt-text-value-f aes-key payload-str)
|
||||
(p/catch (fn [_] payload-str)))
|
||||
(p/resolved payload-str))]
|
||||
data))))
|
||||
|
||||
(defn offload-large-titles
|
||||
[tx-data {:keys [repo graph-id upload-fn aes-key]}]
|
||||
(p/loop [remaining tx-data
|
||||
acc []]
|
||||
(if (empty? remaining)
|
||||
acc
|
||||
(let [item (first remaining)]
|
||||
(if (and (vector? item)
|
||||
(= :db/add (nth item 0 nil))
|
||||
(= :block/title (nth item 2 nil))
|
||||
(string? (nth item 3 nil))
|
||||
(large-title? (nth item 3 nil)))
|
||||
(let [title (nth item 3)]
|
||||
(p/let [obj (upload-fn repo graph-id title aes-key)
|
||||
placeholder (assoc-datom-value item "")]
|
||||
(p/recur (rest remaining)
|
||||
(conj acc placeholder
|
||||
[:db/add (nth item 1) large-title-object-attr obj]))))
|
||||
(p/recur (rest remaining) (conj acc item)))))))
|
||||
|
||||
(defn rehydrate-large-titles!
|
||||
[repo {:keys [graph-id
|
||||
download-fn
|
||||
aes-key
|
||||
tx-data
|
||||
conn
|
||||
get-conn-f
|
||||
graph-e2ee?-f
|
||||
ensure-graph-aes-key-f
|
||||
fail-fast-f]}]
|
||||
(when-let [conn* (or conn (get-conn-f repo))]
|
||||
(let [graph-id* (or graph-id (get-graph-id repo))
|
||||
items (if (some? tx-data)
|
||||
(->> tx-data
|
||||
(keep (fn [item]
|
||||
(when (and (vector? item)
|
||||
(= :db/add (nth item 0))
|
||||
(= large-title-object-attr (nth item 2))
|
||||
(large-title-object? (nth item 3)))
|
||||
{:e (nth item 1)
|
||||
:obj (nth item 3)})))
|
||||
(distinct))
|
||||
(->> (large-title-object-datoms @conn*)
|
||||
(keep (fn [datom]
|
||||
(let [obj (:v datom)]
|
||||
(when (large-title-object? obj)
|
||||
{:e (:e datom)
|
||||
:obj obj}))))
|
||||
(distinct)))]
|
||||
(when (seq items)
|
||||
(p/let [aes-key* (or aes-key
|
||||
(when (graph-e2ee?-f repo)
|
||||
(ensure-graph-aes-key-f repo graph-id*)))
|
||||
_ (when (and (graph-e2ee?-f repo) (nil? aes-key*))
|
||||
(fail-fast-f :db-sync/missing-field {:repo repo :field :aes-key}))]
|
||||
(p/all
|
||||
(mapv (fn [{:keys [e obj]}]
|
||||
(let [eid (resolve-large-title-item-eid @conn* {:e e :obj obj})]
|
||||
(when-not eid
|
||||
(fail-fast-f :db-sync/large-title-entity-missing
|
||||
{:repo repo :e e :obj obj}))
|
||||
(p/let [title (download-fn repo graph-id* obj aes-key*)]
|
||||
(ldb/transact! conn*
|
||||
[[:db/add eid :block/title title]]
|
||||
{:rtc-tx? true
|
||||
:persist-op? false
|
||||
:op :large-title-rehydrate}))))
|
||||
items)))))))
|
||||
|
||||
(defn offload-large-titles-in-datoms-batch
|
||||
([repo graph-id datoms aes-key upload-fn]
|
||||
(offload-large-titles-in-datoms-batch
|
||||
repo graph-id datoms aes-key upload-fn
|
||||
(into #{}
|
||||
(keep (fn [datom]
|
||||
(when (large-title-datom? datom)
|
||||
(:e datom))))
|
||||
datoms)))
|
||||
([repo graph-id datoms aes-key upload-fn offloaded-title-eids]
|
||||
(p/loop [remaining datoms
|
||||
acc []]
|
||||
(if (empty? remaining)
|
||||
acc
|
||||
(let [datom (first remaining)]
|
||||
(cond
|
||||
(large-title-datom? datom)
|
||||
(p/let [obj (upload-fn repo graph-id (:v datom) aes-key)]
|
||||
(p/recur (rest remaining)
|
||||
(conj acc
|
||||
(assoc datom :v "")
|
||||
(assoc datom :a large-title-object-attr :v obj))))
|
||||
|
||||
(and (= large-title-object-attr (:a datom))
|
||||
(contains? offloaded-title-eids (:e datom)))
|
||||
(p/recur (rest remaining) acc)
|
||||
|
||||
:else
|
||||
(p/recur (rest remaining) (conj acc datom))))))))
|
||||
|
||||
(defn take-upload-datoms-batch
|
||||
[datoms batch-size]
|
||||
(loop [batch (transient [])
|
||||
remaining (seq datoms)
|
||||
n 0]
|
||||
(if (or (nil? remaining) (>= n batch-size))
|
||||
[(persistent! batch) remaining]
|
||||
(recur (conj! batch (first remaining))
|
||||
(next remaining)
|
||||
(inc n)))))
|
||||
|
||||
(defn datom->tx
|
||||
[datom]
|
||||
[:db/add (:e datom) (:a datom) (:v datom)])
|
||||
|
||||
(defn process-upload-datoms-in-batches!
|
||||
[datoms {:keys [batch-size process-batch-f progress-f]
|
||||
:or {batch-size 100000}}]
|
||||
(let [total-count (count datoms)]
|
||||
(p/loop [remaining (seq datoms)
|
||||
processed 0]
|
||||
(if (seq remaining)
|
||||
(let [[batch remaining'] (take-upload-datoms-batch remaining batch-size)
|
||||
processed' (+ processed (count batch))]
|
||||
(p/let [_ (process-batch-f batch)]
|
||||
(when progress-f
|
||||
(progress-f processed' total-count))
|
||||
(p/let [_ (js/Promise. (fn [resolve] (js/setTimeout resolve 0)))]
|
||||
(p/recur remaining' processed'))))
|
||||
nil))))
|
||||
|
||||
(defn rehydrate-large-titles-from-db!
|
||||
[repo graph-id {:keys [get-conn-f rehydrate-large-titles!-f]}]
|
||||
(when-let [conn (get-conn-f repo)]
|
||||
(let [tx-data (mapv (fn [datom]
|
||||
[:db/add (:e datom) large-title-object-attr (:v datom)])
|
||||
(large-title-object-datoms @conn))]
|
||||
(when (seq tx-data)
|
||||
(rehydrate-large-titles!-f repo {:tx-data tx-data :graph-id graph-id})))))
|
||||
@@ -1,41 +0,0 @@
|
||||
(ns frontend.worker.sync.log-and-state
|
||||
"Fns to generate rtc related logs"
|
||||
(:require [frontend.worker.shared-service :as shared-service]
|
||||
[logseq.common.defkeywords :refer [defkeywords]]
|
||||
[malli.core :as ma]))
|
||||
|
||||
(def ^:private *rtc-log (atom nil))
|
||||
|
||||
(def ^:private rtc-log-type-schema
|
||||
(vec
|
||||
(concat
|
||||
[:enum]
|
||||
(take-nth
|
||||
2
|
||||
(defkeywords
|
||||
:rtc.log/upload {:doc "rtc log type for upload-graph."}
|
||||
:rtc.log/download {:doc "rtc log type for upload-graph."}
|
||||
:rtc.log/checksum-mismatch {:doc "local/remote checksum mismatch detected"}
|
||||
:rtc.log/tx-rejected {:doc "tx/reject received from server"}
|
||||
:rtc.asset.log/upload-assets {:doc "upload local assets to remote"}
|
||||
:rtc.asset.log/download-assets {:doc "download assets from remote"}
|
||||
:rtc.asset.log/remove-assets {:doc "remove remote assets"}
|
||||
:rtc.asset.log/asset-too-large {:doc "asset is too large to upload"}
|
||||
:rtc.asset.log/initial-download-missing-assets {:doc "download assets if not exists in rtc-asset-sync initial phase"})))))
|
||||
|
||||
(def ^:private rtc-log-type-validator (ma/validator rtc-log-type-schema))
|
||||
|
||||
(defn rtc-log
|
||||
[type m]
|
||||
{:pre [(map? m) (rtc-log-type-validator type)]}
|
||||
(reset! *rtc-log (assoc m :type type :created-at (js/Date.)))
|
||||
nil)
|
||||
|
||||
;;; subscribe-logs, push to frontend
|
||||
;;; TODO: refactor background log subscription
|
||||
(defn- subscribe-logs
|
||||
[]
|
||||
(remove-watch *rtc-log :subscribe-logs)
|
||||
(add-watch *rtc-log :subscribe-logs
|
||||
(fn [_ _ _ n] (when n (shared-service/broadcast-to-clients! :rtc-log n)))))
|
||||
(subscribe-logs)
|
||||
@@ -1,120 +0,0 @@
|
||||
(ns frontend.worker.sync.presence
|
||||
"Presence and rtc state helpers for db sync."
|
||||
(:require [logseq.common.util :as common-util]
|
||||
[frontend.worker.state :as worker-state]))
|
||||
|
||||
(defn current-client
|
||||
[db-sync-client repo]
|
||||
(let [client @db-sync-client]
|
||||
(when (= repo (:repo client))
|
||||
client)))
|
||||
|
||||
(defn client-ops-conn
|
||||
[get-client-ops-conn repo]
|
||||
(get-client-ops-conn repo))
|
||||
|
||||
(defn sync-counts
|
||||
[{:keys [get-datascript-conn
|
||||
get-pending-local-tx-count
|
||||
get-unpushed-asset-ops-count
|
||||
get-missing-asset-upload-files
|
||||
get-local-tx
|
||||
get-local-checksum
|
||||
get-graph-uuid
|
||||
latest-remote-tx
|
||||
latest-remote-checksum]}
|
||||
repo]
|
||||
(when (get-datascript-conn repo)
|
||||
(let [pending-local (if get-pending-local-tx-count
|
||||
(get-pending-local-tx-count repo)
|
||||
0)
|
||||
pending-asset (get-unpushed-asset-ops-count repo)
|
||||
missing-asset-upload-files (if get-missing-asset-upload-files
|
||||
(get-missing-asset-upload-files repo)
|
||||
[])
|
||||
local-tx (get-local-tx repo)
|
||||
remote-tx (get latest-remote-tx repo)
|
||||
local-checksum (when get-local-checksum
|
||||
(get-local-checksum repo))
|
||||
remote-checksum (get latest-remote-checksum repo)
|
||||
pending-server (when (and (number? local-tx) (number? remote-tx))
|
||||
(max 0 (- remote-tx local-tx)))
|
||||
graph-uuid (get-graph-uuid repo)
|
||||
client (current-client worker-state/*db-sync-client repo)
|
||||
ws-url (:ws-url @worker-state/*db-sync-config)
|
||||
ws-state (or (some-> client :ws-state deref)
|
||||
(if (seq ws-url) :stopped :inactive))
|
||||
last-error (some-> client :last-sync-error deref)]
|
||||
{:repo repo
|
||||
:graph-id graph-uuid
|
||||
:pending-local pending-local
|
||||
:pending-asset pending-asset
|
||||
:missing-asset-upload-files missing-asset-upload-files
|
||||
:pending-server pending-server
|
||||
:local-tx local-tx
|
||||
:remote-tx remote-tx
|
||||
:local-checksum local-checksum
|
||||
:remote-checksum remote-checksum
|
||||
:ws-state ws-state
|
||||
:last-error last-error})))
|
||||
|
||||
(defn normalize-online-users
|
||||
[users]
|
||||
(->> users
|
||||
(keep (fn [{:keys [user-id email username name]}]
|
||||
(when (string? user-id)
|
||||
(let [display-name (or username name user-id)]
|
||||
(cond-> {:user/uuid user-id
|
||||
:user/name display-name}
|
||||
(string? email) (assoc :user/email email))))))
|
||||
(common-util/distinct-by :user/uuid)
|
||||
(vec)))
|
||||
|
||||
(defn rtc-state-payload
|
||||
[sync-counts-f client]
|
||||
(let [repo (:repo client)
|
||||
ws-state @(:ws-state client)
|
||||
online-users @(:online-users client)
|
||||
{:keys [pending-local pending-asset missing-asset-upload-files pending-server
|
||||
local-tx remote-tx local-checksum remote-checksum graph-id]}
|
||||
(sync-counts-f repo)]
|
||||
{:rtc-state {:ws-state ws-state}
|
||||
:rtc-lock (= :open ws-state)
|
||||
:online-users (or online-users [])
|
||||
:unpushed-block-update-count (or pending-local 0)
|
||||
:pending-asset-ops-count (or pending-asset 0)
|
||||
:missing-asset-upload-files (or missing-asset-upload-files [])
|
||||
:missing-asset-upload-files-count (count missing-asset-upload-files)
|
||||
:pending-server-ops-count (or pending-server 0)
|
||||
:local-tx local-tx
|
||||
:remote-tx remote-tx
|
||||
:local-checksum local-checksum
|
||||
:remote-checksum remote-checksum
|
||||
:graph-uuid graph-id}))
|
||||
|
||||
(defn set-ws-state!
|
||||
[broadcast-f client ws-state]
|
||||
(when-let [*ws-state (:ws-state client)]
|
||||
(reset! *ws-state ws-state)
|
||||
(broadcast-f client)))
|
||||
|
||||
(defn update-online-users!
|
||||
[broadcast-f client users]
|
||||
(when-let [*online-users (:online-users client)]
|
||||
(let [users' (normalize-online-users users)]
|
||||
(when (not= users' @*online-users)
|
||||
(reset! *online-users users')
|
||||
(broadcast-f client)))))
|
||||
|
||||
(defn update-user-presence!
|
||||
[broadcast-f client user-id* editing-block-uuid]
|
||||
(when (and user-id* editing-block-uuid)
|
||||
(when-let [*online-users (:online-users client)]
|
||||
(swap! *online-users
|
||||
(fn [users]
|
||||
(mapv (fn [user]
|
||||
(if (= user-id* (:user/uuid user))
|
||||
(assoc user :user/editing-block-uuid editing-block-uuid)
|
||||
user))
|
||||
users)))
|
||||
(broadcast-f client))))
|
||||
@@ -1,92 +0,0 @@
|
||||
(ns frontend.worker.sync.temp-sqlite
|
||||
"Temporary sqlite helpers for db sync uploads."
|
||||
(:require [cljs-bean.core :as bean]
|
||||
[datascript.core :as d]
|
||||
[datascript.storage :refer [IStorage]]
|
||||
[logseq.db.common.sqlite :as common-sqlite]
|
||||
[logseq.db.sqlite.util :as sqlite-util]
|
||||
[promesa.core :as p]
|
||||
[frontend.worker-common.util :as worker-util]
|
||||
[frontend.worker.platform :as platform]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync.util :refer [fail-fast]]))
|
||||
|
||||
(def upload-temp-pool-name (worker-util/get-pool-name "upload-temp"))
|
||||
|
||||
(defn- upsert-addr-content!
|
||||
[^js db data]
|
||||
(.transaction
|
||||
db
|
||||
(fn [tx]
|
||||
(doseq [item data]
|
||||
(.exec tx #js {:sql (str "INSERT INTO kvs (addr, content, addresses) "
|
||||
"values ($addr, $content, $addresses) "
|
||||
"on conflict(addr) do update set content = $content, addresses = $addresses")
|
||||
:bind item})))))
|
||||
|
||||
(defn- restore-data-from-addr
|
||||
[^js db addr]
|
||||
(when-let [result (-> (.exec db #js {:sql "select content, addresses from kvs where addr = ?"
|
||||
:bind #js [addr]
|
||||
:rowMode "array"})
|
||||
first)]
|
||||
(let [[content addresses] (bean/->clj result)
|
||||
addresses (when addresses (js/JSON.parse addresses))
|
||||
data (sqlite-util/read-transit-str content)]
|
||||
(if (and addresses (map? data))
|
||||
(assoc data :addresses addresses)
|
||||
data))))
|
||||
|
||||
(defn new-temp-sqlite-storage
|
||||
[^js db]
|
||||
(reify IStorage
|
||||
(-store [_ addr+data-seq _delete-addrs]
|
||||
(let [data (map
|
||||
(fn [[addr data]]
|
||||
(let [data' (if (map? data) (dissoc data :addresses) data)
|
||||
addresses (when (map? data)
|
||||
(when-let [addresses (:addresses data)]
|
||||
(js/JSON.stringify (bean/->js addresses))))]
|
||||
#js {:$addr addr
|
||||
:$content (sqlite-util/write-transit-str data')
|
||||
:$addresses addresses}))
|
||||
addr+data-seq)]
|
||||
(upsert-addr-content! db data)))
|
||||
(-restore [_ addr]
|
||||
(restore-data-from-addr db addr))))
|
||||
|
||||
(defn <create-temp-sqlite-db!
|
||||
[]
|
||||
(if-let [sqlite @worker-state/*sqlite]
|
||||
(let [current-platform (platform/current)]
|
||||
(p/let [pool (platform/install-storage-pool current-platform sqlite upload-temp-pool-name)
|
||||
path (platform/resolve-db-path current-platform upload-temp-pool-name pool "/upload.sqlite")
|
||||
db (platform/sqlite-open current-platform
|
||||
{:sqlite sqlite
|
||||
:pool pool
|
||||
:path path
|
||||
:mode "c"})]
|
||||
(common-sqlite/create-kvs-table! db)
|
||||
{:db db
|
||||
:pool pool
|
||||
:path path}))
|
||||
(fail-fast :db-sync/missing-field {:field :sqlite})))
|
||||
|
||||
(defn <create-temp-sqlite-conn
|
||||
[schema datoms]
|
||||
(p/let [{:keys [db path pool]} (<create-temp-sqlite-db!)
|
||||
storage (new-temp-sqlite-storage db)
|
||||
conn (d/conn-from-datoms datoms schema {:storage storage})]
|
||||
{:db db
|
||||
:conn conn
|
||||
:pool pool
|
||||
:path path}))
|
||||
|
||||
(defn cleanup-temp-sqlite!
|
||||
[{:keys [db conn pool]}]
|
||||
(when conn
|
||||
(reset! conn nil))
|
||||
(when db
|
||||
(.close db))
|
||||
(when pool
|
||||
(platform/remove-storage-pool! (platform/current) pool)))
|
||||
@@ -1,112 +0,0 @@
|
||||
(ns frontend.worker.sync.transport
|
||||
"Transport and message-shaping helpers for db sync."
|
||||
(:require
|
||||
[clojure.string :as string]
|
||||
[lambdaisland.glogi :as log]
|
||||
[logseq.db-sync.malli-schema :as db-sync-schema]
|
||||
[logseq.db.sqlite.util :as sqlite-util]))
|
||||
|
||||
(def invalid-coerce ::invalid-coerce)
|
||||
|
||||
(defn format-ws-url
|
||||
[base graph-id]
|
||||
(cond
|
||||
(string/includes? base "%s")
|
||||
(string/replace base "%s" graph-id)
|
||||
|
||||
(string/ends-with? base "/")
|
||||
(str base graph-id)
|
||||
|
||||
:else
|
||||
(str base "/" graph-id)))
|
||||
|
||||
(defn append-token
|
||||
[url token]
|
||||
(if (string? token)
|
||||
(let [separator (if (string/includes? url "?") "&" "?")]
|
||||
(str url separator "token=" (js/encodeURIComponent token)))
|
||||
url))
|
||||
|
||||
(defn ready-state
|
||||
[ws]
|
||||
(.-readyState ws))
|
||||
|
||||
(defn ws-open?
|
||||
[ws]
|
||||
(= 1 (ready-state ws)))
|
||||
|
||||
(defn coerce
|
||||
[coercer value context]
|
||||
(try
|
||||
(coercer value)
|
||||
(catch :default e
|
||||
(log/error :db-sync/malli-coerce-failed (merge context {:error e :value value}))
|
||||
invalid-coerce)))
|
||||
|
||||
(defn coerce-ws-client-message
|
||||
[message]
|
||||
(when message
|
||||
(let [coerced (coerce db-sync-schema/ws-client-message-coercer message {:schema :ws/client})]
|
||||
(when-not (= coerced invalid-coerce)
|
||||
coerced))))
|
||||
|
||||
(defn coerce-ws-server-message
|
||||
[message]
|
||||
(when message
|
||||
(letfn [(uuid-like->string [value]
|
||||
(cond
|
||||
(uuid? value) (str value)
|
||||
(and (map? value) (string? (:uuid value))) (:uuid value)
|
||||
:else value))
|
||||
(normalize-legacy-tx-reject [m]
|
||||
(if (= "tx/reject" (:type m))
|
||||
(cond-> m
|
||||
(contains? m :failed-tx-id) (update :failed-tx-id uuid-like->string)
|
||||
(contains? m :success-tx-ids) (update :success-tx-ids
|
||||
(fn [ids]
|
||||
(mapv uuid-like->string (or ids []))))
|
||||
(contains? m :missing-block-uuids) (update :missing-block-uuids
|
||||
(fn [ids]
|
||||
(mapv uuid-like->string (or ids [])))))
|
||||
m))]
|
||||
(let [message* (normalize-legacy-tx-reject message)
|
||||
coerced (coerce db-sync-schema/ws-server-message-coercer message* {:schema :ws/server})]
|
||||
(when-not (= coerced invalid-coerce)
|
||||
coerced)))))
|
||||
|
||||
(defn parse-transit
|
||||
[fail-fast-f value context]
|
||||
(try
|
||||
(sqlite-util/read-transit-str value)
|
||||
(catch :default e
|
||||
(fail-fast-f :db-sync/response-parse-failed (assoc context :error e)))))
|
||||
|
||||
(defn reconnect-delay-ms
|
||||
[attempt {:keys [base-delay-ms max-delay-ms jitter-ms]}]
|
||||
(let [exp (js/Math.pow 2 attempt)
|
||||
delay (min max-delay-ms (* base-delay-ms exp))
|
||||
jitter (rand-int jitter-ms)]
|
||||
(+ delay jitter)))
|
||||
|
||||
(defn parse-message
|
||||
[raw]
|
||||
(try
|
||||
(js->clj (js/JSON.parse raw) :keywordize-keys true)
|
||||
(catch :default _
|
||||
nil)))
|
||||
|
||||
(defn send!
|
||||
[coerce-ws-client-message-f ws message]
|
||||
(when (ws-open? ws)
|
||||
(if-let [coerced (coerce-ws-client-message-f message)]
|
||||
(let [message* (if (= "tx/batch" (:type coerced))
|
||||
(update coerced :txs
|
||||
(fn [txs]
|
||||
(mapv (fn [tx-entry]
|
||||
(if-let [tx-id (:tx-id tx-entry)]
|
||||
(assoc tx-entry :tx-id (str tx-id))
|
||||
tx-entry))
|
||||
txs)))
|
||||
coerced)]
|
||||
(.send ws (js/JSON.stringify (clj->js message*))))
|
||||
(log/error :db-sync/ws-request-invalid {:message message}))))
|
||||
@@ -1,428 +0,0 @@
|
||||
(ns frontend.worker.sync.upload
|
||||
"Snapshot upload helpers for db sync."
|
||||
(:require
|
||||
[cljs-bean.core :as bean]
|
||||
[datascript.core :as d]
|
||||
[frontend.worker-common.util :as worker-util]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync.apply-txs :as sync-apply]
|
||||
[frontend.worker.sync.auth :as sync-auth]
|
||||
[frontend.worker.sync.client-op :as client-op]
|
||||
[frontend.worker.sync.crypt :as sync-crypt]
|
||||
[frontend.worker.sync.large-title :as sync-large-title]
|
||||
[frontend.worker.sync.temp-sqlite :as sync-temp-sqlite]
|
||||
[frontend.worker.sync.util :refer [coerce-http-request fail-fast fetch-json] :as sync-util]
|
||||
[logseq.common.config :as common-config]
|
||||
[logseq.db :as ldb]
|
||||
[logseq.db-sync.checksum :as sync-checksum]
|
||||
[logseq.db.sqlite.util :as sqlite-util]
|
||||
[promesa.core :as p]))
|
||||
|
||||
(def upload-kvs-batch-size 500)
|
||||
(def upload-prepare-datoms-batch-size 100000)
|
||||
(def snapshot-upload-max-bytes 1000000)
|
||||
(def snapshot-frame-header-bytes 4)
|
||||
(def ignored-oversized-upload-attrs #{:logseq.property.tldraw/page})
|
||||
(def snapshot-local-only-attrs #{:block/tx-id})
|
||||
(def snapshot-content-type "application/transit+json")
|
||||
(def snapshot-content-encoding "gzip")
|
||||
(def snapshot-text-encoder (js/TextEncoder.))
|
||||
|
||||
(defn- http-base-url
|
||||
[]
|
||||
(sync-auth/http-base-url @worker-state/*db-sync-config))
|
||||
|
||||
(defn- ensure-client-graph-uuid!
|
||||
[repo graph-id]
|
||||
(when (seq graph-id)
|
||||
(client-op/update-graph-uuid repo graph-id)))
|
||||
|
||||
(defn fetch-kvs-rows
|
||||
[db last-addr limit]
|
||||
(.exec db #js {:sql "select addr, content, addresses from kvs where addr > ? order by addr asc limit ?"
|
||||
:bind #js [last-addr limit]
|
||||
:rowMode "array"}))
|
||||
|
||||
(defn count-kvs-rows
|
||||
[db]
|
||||
(when-let [result (-> (.exec db #js {:sql "select count(*) from kvs"
|
||||
:rowMode "array"})
|
||||
first)]
|
||||
(first (bean/->clj result))))
|
||||
|
||||
(defn normalize-snapshot-rows
|
||||
[rows]
|
||||
(mapv (fn [row] (vec row)) (array-seq rows)))
|
||||
|
||||
(defn encode-snapshot-rows
|
||||
[rows]
|
||||
(.encode snapshot-text-encoder (sqlite-util/write-transit-str rows)))
|
||||
|
||||
(defn- datom-value-byte-length
|
||||
[value]
|
||||
(.-byteLength ^js (.encode snapshot-text-encoder (sqlite-util/write-transit-str value))))
|
||||
|
||||
(defn- drop-oversized-upload-datoms
|
||||
[datoms]
|
||||
(let [threshold (- snapshot-upload-max-bytes snapshot-frame-header-bytes)]
|
||||
(reduce (fn [{:keys [kept dropped]} datom]
|
||||
(let [attr (:a datom)
|
||||
size (when (contains? ignored-oversized-upload-attrs attr)
|
||||
(datom-value-byte-length (:v datom)))]
|
||||
(if (and size (> size threshold))
|
||||
{:kept kept
|
||||
:dropped (conj dropped {:a attr
|
||||
:e (:e datom)
|
||||
:bytes size})}
|
||||
{:kept (conj kept datom)
|
||||
:dropped dropped})))
|
||||
{:kept []
|
||||
:dropped []}
|
||||
datoms)))
|
||||
|
||||
(defn- snapshot-rows-byte-length
|
||||
[rows]
|
||||
(+ snapshot-frame-header-bytes
|
||||
(.-byteLength ^js (encode-snapshot-rows rows))))
|
||||
|
||||
(defn- max-prefix-rows-within-bytes
|
||||
[rows max-bytes]
|
||||
(let [rows-count (count rows)]
|
||||
(loop [low 1
|
||||
high rows-count
|
||||
best 0]
|
||||
(if (> low high)
|
||||
best
|
||||
(let [mid (quot (+ low high) 2)
|
||||
rows' (subvec rows 0 mid)
|
||||
size (snapshot-rows-byte-length rows')]
|
||||
(if (<= size max-bytes)
|
||||
(recur (inc mid) high mid)
|
||||
(recur low (dec mid) best)))))))
|
||||
|
||||
(defn- split-snapshot-rows-by-max-bytes
|
||||
[rows max-bytes]
|
||||
(loop [remaining rows
|
||||
batches []]
|
||||
(if (empty? remaining)
|
||||
batches
|
||||
(let [prefix-count (max-prefix-rows-within-bytes remaining max-bytes)]
|
||||
(if (pos? prefix-count)
|
||||
(let [batch (subvec remaining 0 prefix-count)
|
||||
remaining' (subvec remaining prefix-count)]
|
||||
(recur remaining' (conj batches batch)))
|
||||
(let [row (first remaining)
|
||||
row-size (snapshot-rows-byte-length [row])]
|
||||
(fail-fast :db-sync/snapshot-row-too-large
|
||||
{:max-bytes max-bytes
|
||||
:row-size row-size
|
||||
:addr (first row)})))))))
|
||||
|
||||
(defn frame-bytes
|
||||
[^js data]
|
||||
(let [len (.-byteLength data)
|
||||
out (js/Uint8Array. (+ 4 len))
|
||||
view (js/DataView. (.-buffer out))]
|
||||
(.setUint32 view 0 len false)
|
||||
(.set out data 4)
|
||||
out))
|
||||
|
||||
(defn maybe-compress-stream
|
||||
[stream]
|
||||
(if (exists? js/CompressionStream)
|
||||
(.pipeThrough stream (js/CompressionStream. "gzip"))
|
||||
stream))
|
||||
|
||||
(defn <buffer-stream
|
||||
[stream]
|
||||
(p/let [resp (js/Response. stream)
|
||||
buf (.arrayBuffer resp)]
|
||||
buf))
|
||||
|
||||
(defn <snapshot-upload-body
|
||||
[rows]
|
||||
(let [frame (frame-bytes (encode-snapshot-rows rows))
|
||||
stream (js/ReadableStream.
|
||||
#js {:start (fn [controller]
|
||||
(.enqueue controller frame)
|
||||
(.close controller))})
|
||||
use-compression? (exists? js/CompressionStream)
|
||||
body (if use-compression? (maybe-compress-stream stream) stream)]
|
||||
(if use-compression?
|
||||
(p/let [buf (<buffer-stream body)]
|
||||
{:body buf :encoding snapshot-content-encoding})
|
||||
(p/resolved {:body frame :encoding nil}))))
|
||||
|
||||
(defn- snapshot-upload-url
|
||||
[base graph-id reset? finished? checksum]
|
||||
(str base "/sync/" graph-id "/snapshot/upload?reset="
|
||||
(if reset? "true" "false")
|
||||
"&finished="
|
||||
(if finished? "true" "false")
|
||||
(when finished?
|
||||
(str "&checksum=" (js/encodeURIComponent checksum)))))
|
||||
|
||||
(defn- <upload-snapshot-rows-batches!
|
||||
[rows-batches {:keys [base graph-id first-batch? finished? checksum auth-fetch-f]}]
|
||||
(p/loop [remaining rows-batches
|
||||
first-request? first-batch?]
|
||||
(if-let [rows-batch (first remaining)]
|
||||
(let [last-request? (nil? (next remaining))
|
||||
finished-request? (and finished? last-request?)
|
||||
upload-url (snapshot-upload-url base graph-id first-request? finished-request? checksum)]
|
||||
(p/let [{:keys [body encoding]} (<snapshot-upload-body rows-batch)
|
||||
headers (cond-> {"content-type" snapshot-content-type}
|
||||
(string? encoding) (assoc "content-encoding" encoding))
|
||||
_ (auth-fetch-f upload-url headers body)]
|
||||
(p/recur (next remaining) false)))
|
||||
nil)))
|
||||
(defn <prepare-upload-temp-sqlite!
|
||||
[repo graph-id source-conn aes-key update-progress]
|
||||
(p/let [temp (sync-temp-sqlite/<create-temp-sqlite-conn (d/schema @source-conn) [])
|
||||
datoms (remove #(contains? snapshot-local-only-attrs (:a %))
|
||||
(d/datoms @source-conn :eavt))
|
||||
large-title-eids (into #{}
|
||||
(keep (fn [datom]
|
||||
(when (sync-large-title/large-title-datom? datom)
|
||||
(:e datom))))
|
||||
datoms)
|
||||
_ (sync-large-title/process-upload-datoms-in-batches!
|
||||
datoms
|
||||
{:batch-size upload-prepare-datoms-batch-size
|
||||
:process-batch-f
|
||||
(fn [batch]
|
||||
(p/let [datoms* (sync-large-title/offload-large-titles-in-datoms-batch
|
||||
repo graph-id batch aes-key
|
||||
sync-apply/upload-large-title! large-title-eids)
|
||||
{:keys [kept dropped]} (drop-oversized-upload-datoms datoms*)
|
||||
_ (when (seq dropped)
|
||||
(prn :db-sync/drop-oversized-upload-datoms
|
||||
{:repo repo
|
||||
:count (count dropped)
|
||||
:attrs (vec (distinct (map :a dropped)))
|
||||
:max-bytes (apply max (map :bytes dropped))}))
|
||||
encrypted-datoms (if aes-key
|
||||
(sync-crypt/<encrypt-datoms aes-key kept)
|
||||
kept)
|
||||
tx-data (mapv sync-large-title/datom->tx encrypted-datoms)]
|
||||
(d/transact! (:conn temp) tx-data {:initial-db? true})
|
||||
nil))
|
||||
:progress-f
|
||||
(fn [processed total]
|
||||
(update-progress {:sub-type :upload-progress
|
||||
:message (if aes-key
|
||||
(str "Encrypting " processed "/" total)
|
||||
(str "Preparing " processed "/" total))}))})]
|
||||
temp))
|
||||
|
||||
(defn- normalize-graph-e2ee?
|
||||
[graph-e2ee?]
|
||||
(if (nil? graph-e2ee?)
|
||||
true
|
||||
(true? graph-e2ee?)))
|
||||
|
||||
(defn- graph-id->uuid
|
||||
[repo graph-id]
|
||||
(when-not (seq graph-id)
|
||||
(fail-fast :db-sync/missing-field {:repo repo :field :graph-id}))
|
||||
(try
|
||||
(uuid graph-id)
|
||||
(catch :default e
|
||||
(fail-fast :db-sync/invalid-field {:repo repo
|
||||
:field :graph-id
|
||||
:value graph-id
|
||||
:error e}))))
|
||||
|
||||
(defn- set-graph-sync-metadata!
|
||||
[repo graph-id graph-e2ee?]
|
||||
(when-let [conn (worker-state/get-datascript-conn repo)]
|
||||
(ldb/transact! conn [(ldb/kv :logseq.kv/graph-uuid (graph-id->uuid repo graph-id))
|
||||
(ldb/kv :logseq.kv/graph-remote? true)
|
||||
(ldb/kv :logseq.kv/graph-rtc-e2ee? (true? graph-e2ee?))]
|
||||
{:outliner-op :set-kvs})))
|
||||
|
||||
(defn- persist-upload-graph-identity!
|
||||
[repo graph-id graph-e2ee?]
|
||||
(let [graph-id (some-> graph-id str)
|
||||
graph-e2ee? (normalize-graph-e2ee? graph-e2ee?)]
|
||||
(when-not (seq graph-id)
|
||||
(fail-fast :db-sync/missing-field {:repo repo :field :graph-id}))
|
||||
(set-graph-sync-metadata! repo graph-id graph-e2ee?)
|
||||
(ensure-client-graph-uuid! repo graph-id)
|
||||
{:graph-id graph-id
|
||||
:graph-e2ee? graph-e2ee?}))
|
||||
|
||||
(defn- <create-remote-graph-aux!
|
||||
[repo {:keys [graph-e2ee? graph-ready-for-use?]}]
|
||||
(let [base (http-base-url)
|
||||
graph-name (some-> repo common-config/strip-leading-db-version-prefix)
|
||||
schema-version (some-> (worker-state/get-datascript-conn repo)
|
||||
deref
|
||||
ldb/get-graph-schema-version
|
||||
:major
|
||||
str)
|
||||
graph-e2ee? (normalize-graph-e2ee? graph-e2ee?)]
|
||||
(cond
|
||||
(not (seq base))
|
||||
(fail-fast :db-sync/missing-field {:repo repo :field :http-base})
|
||||
|
||||
(not (seq graph-name))
|
||||
(fail-fast :db-sync/missing-field {:repo repo :field :graph-name})
|
||||
|
||||
:else
|
||||
(do
|
||||
(sync-util/require-auth-token! {:repo repo :field :auth-token})
|
||||
(p/let [_ (when graph-e2ee?
|
||||
(sync-crypt/ensure-user-rsa-keys! {:ensure-server? true}))
|
||||
body (coerce-http-request :graphs/create
|
||||
{:graph-name graph-name
|
||||
:schema-version schema-version
|
||||
:graph-e2ee? graph-e2ee?
|
||||
:graph-ready-for-use? (not= false graph-ready-for-use?)})
|
||||
_ (when (nil? body)
|
||||
(fail-fast :db-sync/invalid-field {:repo repo
|
||||
:field :create-graph-body}))
|
||||
result (fetch-json (str base "/graphs")
|
||||
{:method "POST"
|
||||
:headers {"content-type" "application/json"}
|
||||
:body (js/JSON.stringify (clj->js body))}
|
||||
{:response-schema :graphs/create})
|
||||
graph-id (:graph-id result)
|
||||
graph-e2ee? (normalize-graph-e2ee? (if (contains? result :graph-e2ee?)
|
||||
(:graph-e2ee? result)
|
||||
graph-e2ee?))]
|
||||
(when-not (seq graph-id)
|
||||
(fail-fast :db-sync/missing-field {:repo repo
|
||||
:field :graph-id
|
||||
:op :create-graph}))
|
||||
(persist-upload-graph-identity! repo graph-id graph-e2ee?)
|
||||
{:graph-id graph-id
|
||||
:graph-e2ee? graph-e2ee?})))))
|
||||
|
||||
(defn list-remote-graphs!
|
||||
[]
|
||||
(let [base (sync-auth/http-base-url @worker-state/*db-sync-config)]
|
||||
(if-not (seq base)
|
||||
(p/resolved [])
|
||||
(do
|
||||
(sync-util/require-auth-token! {:op :list-remote-graphs})
|
||||
(p/let [resp (fetch-json (str base "/graphs")
|
||||
{:method "GET"}
|
||||
{:response-schema :graphs/list})]
|
||||
(vec (or (:graphs resp) [])))))))
|
||||
|
||||
(defn- fail-upload-graph-already-exists!
|
||||
[repo {:keys [graph-id graph-name]}]
|
||||
(throw (ex-info "remote graph already exists; delete it before uploading again"
|
||||
{:code :db-sync/graph-already-exists
|
||||
:repo repo
|
||||
:graph-id graph-id
|
||||
:graph-name graph-name})))
|
||||
|
||||
(defn- remote-graph-matches-upload-target?
|
||||
[target-graph-name {:keys [graph-name]}]
|
||||
(= target-graph-name graph-name))
|
||||
|
||||
(defn create-remote-graph!
|
||||
[repo {:keys [graph-e2ee? graph-ready-for-use?]}]
|
||||
(let [target-graph-name (some-> repo common-config/strip-leading-db-version-prefix)]
|
||||
(cond
|
||||
(not (seq target-graph-name))
|
||||
(fail-fast :db-sync/missing-field {:repo repo :field :graph-name})
|
||||
|
||||
:else
|
||||
(p/let [remote-graphs (list-remote-graphs!)
|
||||
matching-graphs (filterv (partial remote-graph-matches-upload-target?
|
||||
target-graph-name)
|
||||
remote-graphs)]
|
||||
(cond
|
||||
(> (count matching-graphs) 1)
|
||||
(fail-fast :db-sync/ambiguous-graph-match {:repo repo
|
||||
:graph-name target-graph-name
|
||||
:match-count (count matching-graphs)})
|
||||
|
||||
(= 1 (count matching-graphs))
|
||||
(fail-upload-graph-already-exists! repo {:graph-name target-graph-name})
|
||||
|
||||
:else
|
||||
(p/let [_ (sync-crypt/<preflight-upload-e2ee! repo graph-e2ee?)]
|
||||
(<create-remote-graph-aux! repo {:graph-e2ee? graph-e2ee?
|
||||
:graph-ready-for-use? graph-ready-for-use?})))))))
|
||||
|
||||
(defn upload-graph!
|
||||
[repo]
|
||||
(let [base (http-base-url)
|
||||
update-progress (fn [payload]
|
||||
(worker-util/post-message :rtc-log
|
||||
(merge {:type :rtc.log/upload}
|
||||
payload)))]
|
||||
(if-not (seq base)
|
||||
(p/rejected (ex-info "db-sync missing base"
|
||||
{:repo repo :base base}))
|
||||
(if-let [source-conn (worker-state/get-datascript-conn repo)]
|
||||
(p/let [graph-e2ee? (normalize-graph-e2ee? (sync-crypt/graph-e2ee? repo))
|
||||
{:keys [graph-id]} (create-remote-graph! repo {:graph-e2ee? graph-e2ee?
|
||||
:graph-ready-for-use? false})]
|
||||
(p/let [aes-key (when graph-e2ee?
|
||||
(sync-crypt/<ensure-graph-aes-key repo graph-id))
|
||||
_ (when (and graph-e2ee? (nil? aes-key))
|
||||
(fail-fast :db-sync/missing-field {:repo repo :field :aes-key}))]
|
||||
(let [snapshot-checksum (sync-checksum/recompute-checksum @source-conn)]
|
||||
(client-op/update-local-checksum repo snapshot-checksum)
|
||||
(p/let [_ (update-progress {:sub-type :upload-progress
|
||||
:message (if graph-e2ee? "Encrypting..." "Preparing...")})
|
||||
{:keys [db] :as temp} (<prepare-upload-temp-sqlite!
|
||||
repo graph-id source-conn aes-key update-progress)
|
||||
total-rows (count-kvs-rows db)]
|
||||
(-> (p/loop [last-addr -1
|
||||
first-batch? true
|
||||
loaded 0]
|
||||
(let [rows (fetch-kvs-rows db last-addr upload-kvs-batch-size)]
|
||||
(if (empty? rows)
|
||||
(do
|
||||
(sync-apply/clear-pending-txs! repo)
|
||||
(client-op/reset-local-tx repo)
|
||||
(client-op/add-all-exists-asset-as-ops repo)
|
||||
(update-progress {:sub-type :upload-completed
|
||||
:message "Graph upload finished!"})
|
||||
{:graph-id graph-id})
|
||||
(let [max-addr (apply max (map first rows))
|
||||
rows* (normalize-snapshot-rows rows)
|
||||
loaded' (+ loaded (count rows*))
|
||||
finished? (= loaded' total-rows)
|
||||
row-batches (split-snapshot-rows-by-max-bytes rows* snapshot-upload-max-bytes)
|
||||
batch-payloads
|
||||
(mapv (fn [rows-batch]
|
||||
{:rows (count rows-batch)
|
||||
:payload-bytes (snapshot-rows-byte-length rows-batch)})
|
||||
row-batches)]
|
||||
(prn :db-sync/upload-kvs-batch
|
||||
{:total-kvs-rows total-rows
|
||||
:fetched-kvs-rows (count rows*)
|
||||
:upload-kvs-batch-size upload-kvs-batch-size
|
||||
:split-batch-count (count row-batches)
|
||||
:split-batches batch-payloads
|
||||
:max-request-bytes snapshot-upload-max-bytes})
|
||||
(p/let [_ (<upload-snapshot-rows-batches!
|
||||
row-batches
|
||||
{:base base
|
||||
:graph-id graph-id
|
||||
:first-batch? first-batch?
|
||||
:finished? finished?
|
||||
:checksum snapshot-checksum
|
||||
:auth-fetch-f
|
||||
(fn [upload-url headers body]
|
||||
(fetch-json
|
||||
upload-url
|
||||
{:method "POST"
|
||||
:headers headers
|
||||
:body body}
|
||||
{:response-schema :sync/snapshot-upload}))})]
|
||||
(update-progress {:sub-type :upload-progress
|
||||
:message (str "Uploading " loaded' "/" total-rows)})
|
||||
(p/recur max-addr false loaded'))))))
|
||||
(p/finally
|
||||
(fn []
|
||||
(sync-temp-sqlite/cleanup-temp-sqlite! temp))))))))
|
||||
(p/rejected (ex-info "db-sync missing datascript conn"
|
||||
{:repo repo}))))))
|
||||
@@ -1,138 +0,0 @@
|
||||
(ns frontend.worker.sync.util
|
||||
"Helpers for sync"
|
||||
(:require [lambdaisland.glogi :as log]
|
||||
[frontend.worker.platform :as platform]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[logseq.db :as ldb]
|
||||
[frontend.worker.sync.client-op :as client-op]
|
||||
[logseq.common.util :as common-util]
|
||||
[logseq.common.version :as build-version]
|
||||
[logseq.db-sync.malli-schema :as db-sync-schema]
|
||||
[promesa.core :as p]))
|
||||
|
||||
(defn fail-fast [tag data]
|
||||
(log/error tag data)
|
||||
(throw (ex-info (name tag) data)))
|
||||
|
||||
(defn cli-node-owner?
|
||||
[]
|
||||
(try
|
||||
(let [env (:env (platform/current))]
|
||||
(and (= :node (:runtime env))
|
||||
(= :cli (:owner-source env))))
|
||||
(catch :default _ false)))
|
||||
|
||||
(defn auth-token
|
||||
[]
|
||||
(let [state @worker-state/*state]
|
||||
(or (:auth/id-token state)
|
||||
(:auth/access-token state))))
|
||||
|
||||
(defn get-graph-id
|
||||
[repo]
|
||||
(or (when-let [conn (worker-state/get-datascript-conn repo)]
|
||||
(let [db @conn
|
||||
graph-uuid (ldb/get-graph-rtc-uuid db)]
|
||||
(when graph-uuid
|
||||
(str graph-uuid))))
|
||||
(some-> (client-op/get-graph-uuid repo) str)))
|
||||
|
||||
(defn require-auth-token!
|
||||
[context]
|
||||
(when-not (seq (auth-token))
|
||||
(fail-fast :db-sync/missing-field (assoc context :field :auth-token))))
|
||||
|
||||
(defn- ex-message->code
|
||||
[message]
|
||||
(when (and (string? message)
|
||||
(re-matches #"[a-zA-Z0-9._/\-]+" message))
|
||||
(keyword message)))
|
||||
|
||||
(defn- error->diagnostic
|
||||
[error]
|
||||
(let [data (or (ex-data error) {})
|
||||
code (or (:code data)
|
||||
(ex-message->code (ex-message error))
|
||||
:exception)]
|
||||
{:code code
|
||||
:message (or (ex-message error) (str error))
|
||||
:at (common-util/time-ms)
|
||||
:data (when (seq data) data)}))
|
||||
|
||||
(defn set-last-sync-error!
|
||||
[client error]
|
||||
(when-let [*last-error (:last-sync-error client)]
|
||||
(reset! *last-error (error->diagnostic error))))
|
||||
|
||||
(defn clear-last-sync-error!
|
||||
[client]
|
||||
(when-let [*last-error (:last-sync-error client)]
|
||||
(reset! *last-error nil)))
|
||||
|
||||
(def ^:private invalid-coerce ::invalid-coerce)
|
||||
|
||||
(defn coerce
|
||||
[coercer value context]
|
||||
(try
|
||||
(coercer value)
|
||||
(catch :default e
|
||||
(log/error :db-sync/malli-coerce-failed (merge context {:error e :value value}))
|
||||
invalid-coerce)))
|
||||
|
||||
(defn- with-client-revision
|
||||
[schema-key body]
|
||||
(cond-> body
|
||||
(and (= :sync/tx-batch schema-key)
|
||||
(map? body)
|
||||
(not (contains? body :client-revision)))
|
||||
(assoc :client-revision (build-version/revision))))
|
||||
|
||||
(defn coerce-http-request [schema-key body]
|
||||
(if-let [coercer (get db-sync-schema/http-request-coercers schema-key)]
|
||||
(let [coerced (coerce coercer (with-client-revision schema-key body) {:schema schema-key :dir :request})]
|
||||
(when-not (= coerced invalid-coerce)
|
||||
coerced))
|
||||
body))
|
||||
|
||||
(defn coerce-http-response [schema-key body]
|
||||
(if-let [coercer (get db-sync-schema/http-response-coercers schema-key)]
|
||||
(let [coerced (coerce coercer body {:schema schema-key :dir :response})]
|
||||
(when-not (= coerced invalid-coerce)
|
||||
coerced))
|
||||
body))
|
||||
|
||||
(defn- auth-headers []
|
||||
(let [token (auth-token)]
|
||||
(when (nil? token)
|
||||
(throw (ex-info "Empty token" {})))
|
||||
{"authorization" (str "Bearer " token)}))
|
||||
|
||||
(defn- with-auth-headers [opts]
|
||||
(if-let [auth (auth-headers)]
|
||||
(assoc opts :headers (merge (or (:headers opts) {}) auth))
|
||||
opts))
|
||||
|
||||
(defn fetch-json
|
||||
[url opts {:keys [response-schema error-schema] :or {error-schema :error}}]
|
||||
(p/let [resp (js/fetch url (clj->js (with-auth-headers opts)))
|
||||
text (.text resp)
|
||||
data (when (seq text) (js/JSON.parse text))]
|
||||
(if (.-ok resp)
|
||||
(let [body (js->clj data :keywordize-keys true)
|
||||
body (if response-schema
|
||||
(coerce-http-response response-schema body)
|
||||
body)]
|
||||
(if (or (nil? response-schema) body)
|
||||
body
|
||||
(throw (ex-info "db-sync invalid response"
|
||||
{:status (.-status resp)
|
||||
:url url
|
||||
:body body}))))
|
||||
(let [body (when data (js->clj data :keywordize-keys true))
|
||||
body (if error-schema
|
||||
(coerce-http-response error-schema body)
|
||||
body)]
|
||||
(throw (ex-info "db-sync request failed"
|
||||
{:status (.-status resp)
|
||||
:url url
|
||||
:body body}))))))
|
||||
@@ -1,175 +0,0 @@
|
||||
(ns frontend.worker.ui-request
|
||||
"UI request/response manager for worker-side interactive flows."
|
||||
(:require [frontend.common.thread-api :refer [def-thread-api]]
|
||||
[frontend.worker.platform :as platform]
|
||||
[lambdaisland.glogi :as log]
|
||||
[promesa.core :as p]))
|
||||
|
||||
(defonce ^:private *ui-requests-in-flight (atom {}))
|
||||
|
||||
(def ^:private default-timeout-ms 60000)
|
||||
|
||||
(defn- interactive-runtime?
|
||||
[]
|
||||
(let [env (:env (platform/current))
|
||||
runtime (:runtime env)
|
||||
owner-source (:owner-source env)]
|
||||
(or (= :browser runtime)
|
||||
(and (= :node runtime)
|
||||
(= :electron owner-source)))))
|
||||
|
||||
(defn- ui-interaction-required-error
|
||||
[action hint]
|
||||
(ex-info "ui-interaction-required"
|
||||
(cond-> {:code :ui-interaction-required
|
||||
:action action}
|
||||
(seq hint) (assoc :hint hint))))
|
||||
|
||||
(defn- normalize-timeout-ms
|
||||
[timeout-ms]
|
||||
(if (and (number? timeout-ms) (pos? timeout-ms))
|
||||
timeout-ms
|
||||
default-timeout-ms))
|
||||
|
||||
(defn- take-request!
|
||||
[request-id]
|
||||
(let [request (get @*ui-requests-in-flight request-id)]
|
||||
(when request
|
||||
(swap! *ui-requests-in-flight dissoc request-id)
|
||||
request)))
|
||||
|
||||
(defn- timeout-error
|
||||
[request-id action timeout-ms]
|
||||
(ex-info "ui-request-timeout"
|
||||
{:code :ui-request-timeout
|
||||
:request-id request-id
|
||||
:action action
|
||||
:timeout-ms timeout-ms}))
|
||||
|
||||
(defn- cancel-error
|
||||
[request-id action context]
|
||||
(ex-info "ui-request-cancelled"
|
||||
{:code :ui-request-cancelled
|
||||
:request-id request-id
|
||||
:action action
|
||||
:context context}))
|
||||
|
||||
(defn- reject-error
|
||||
[request-id action error]
|
||||
(let [data (ex-data error)
|
||||
message (or (ex-message error) "ui-request-rejected")]
|
||||
(ex-info message
|
||||
(merge {:code :ui-request-rejected
|
||||
:request-id request-id
|
||||
:action action}
|
||||
(when (seq data)
|
||||
{:data data})))))
|
||||
|
||||
(defn- ->rejectable-error
|
||||
[request-id action error]
|
||||
(cond
|
||||
(instance? cljs.core.ExceptionInfo error)
|
||||
(reject-error request-id action error)
|
||||
|
||||
(instance? js/Error error)
|
||||
(reject-error request-id action error)
|
||||
|
||||
(map? error)
|
||||
(let [message (or (:message error) "ui-request-rejected")
|
||||
data (dissoc error :message)]
|
||||
(ex-info message
|
||||
(merge {:code :ui-request-rejected
|
||||
:request-id request-id
|
||||
:action action}
|
||||
(when (seq data)
|
||||
{:data data}))))
|
||||
|
||||
(string? error)
|
||||
(ex-info error
|
||||
{:code :ui-request-rejected
|
||||
:request-id request-id
|
||||
:action action})
|
||||
|
||||
:else
|
||||
(ex-info "ui-request-rejected"
|
||||
{:code :ui-request-rejected
|
||||
:request-id request-id
|
||||
:action action
|
||||
:error error})))
|
||||
|
||||
(defn <request
|
||||
([action payload]
|
||||
(<request action payload nil))
|
||||
([action payload {:keys [timeout-ms hint]}]
|
||||
(if-not (interactive-runtime?)
|
||||
(p/rejected (ui-interaction-required-error action hint))
|
||||
(let [request-id (str (random-uuid))
|
||||
timeout-ms (normalize-timeout-ms timeout-ms)
|
||||
deferred (p/deferred)
|
||||
timeout-id (js/setTimeout
|
||||
(fn []
|
||||
(when-let [{:keys [deferred action timeout-ms]} (take-request! request-id)]
|
||||
(p/reject! deferred (timeout-error request-id action timeout-ms))))
|
||||
timeout-ms)]
|
||||
(swap! *ui-requests-in-flight assoc request-id
|
||||
{:request-id request-id
|
||||
:action action
|
||||
:deferred deferred
|
||||
:timeout-id timeout-id
|
||||
:timeout-ms timeout-ms})
|
||||
(try
|
||||
(platform/post-message! (platform/current)
|
||||
:db-worker/ui-request
|
||||
{:request-id request-id
|
||||
:action action
|
||||
:payload payload
|
||||
:timeout-ms timeout-ms})
|
||||
(catch :default error
|
||||
(when-let [{:keys [deferred timeout-id action]} (take-request! request-id)]
|
||||
(js/clearTimeout timeout-id)
|
||||
(p/reject! deferred (reject-error request-id action error)))))
|
||||
deferred))))
|
||||
|
||||
(defn resolve-request!
|
||||
[request-id result]
|
||||
(if-let [{:keys [deferred timeout-id]} (take-request! request-id)]
|
||||
(do
|
||||
(js/clearTimeout timeout-id)
|
||||
(p/resolve! deferred result)
|
||||
{:ok true})
|
||||
(do
|
||||
(log/warn :db-worker/ui-request-missing-resolve {:request-id request-id})
|
||||
{:ok false :reason :request-not-found :request-id request-id})))
|
||||
|
||||
(defn reject-request!
|
||||
[request-id error]
|
||||
(if-let [{:keys [deferred timeout-id action]} (take-request! request-id)]
|
||||
(do
|
||||
(js/clearTimeout timeout-id)
|
||||
(p/reject! deferred (->rejectable-error request-id action error))
|
||||
{:ok true})
|
||||
(do
|
||||
(log/warn :db-worker/ui-request-missing-reject {:request-id request-id
|
||||
:error error})
|
||||
{:ok false :reason :request-not-found :request-id request-id})))
|
||||
|
||||
(defn cancel-all!
|
||||
[context]
|
||||
(let [requests (vals @*ui-requests-in-flight)]
|
||||
(reset! *ui-requests-in-flight {})
|
||||
(doseq [{:keys [request-id deferred timeout-id action]} requests]
|
||||
(js/clearTimeout timeout-id)
|
||||
(p/reject! deferred (cancel-error request-id action context)))
|
||||
{:ok true :cancelled (count requests)}))
|
||||
|
||||
(def-thread-api :thread-api/resolve-ui-request
|
||||
[request-id result]
|
||||
(resolve-request! request-id result))
|
||||
|
||||
(def-thread-api :thread-api/reject-ui-request
|
||||
[request-id error]
|
||||
(reject-request! request-id error))
|
||||
|
||||
(def-thread-api :thread-api/cancel-ui-requests
|
||||
[context]
|
||||
(cancel-all! context))
|
||||
@@ -2,7 +2,7 @@
|
||||
"Undo redo new implementation"
|
||||
(:require [datascript.core :as d]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync.client-op :as client-op]
|
||||
[frontend.worker.local-tx-log :as local-tx-log]
|
||||
[lambdaisland.glogi :as log]
|
||||
[logseq.common.defkeywords :refer [defkeywords]]
|
||||
[malli.core :as m]
|
||||
@@ -308,7 +308,7 @@
|
||||
(defn- pending-history-action-ops
|
||||
[repo tx-id]
|
||||
(when (uuid? tx-id)
|
||||
(client-op/history-action-ops-by-tx-id repo tx-id)))
|
||||
(local-tx-log/history-action-ops-by-tx-id repo tx-id)))
|
||||
|
||||
(defn gen-undo-ops!
|
||||
[repo {:keys [tx-data tx-meta db-after db-before]} tx-id
|
||||
|
||||
547
src/main/frontend/worker/undo_redo/replay.cljs
Normal file
547
src/main/frontend/worker/undo_redo/replay.cljs
Normal file
@@ -0,0 +1,547 @@
|
||||
(ns frontend.worker.undo-redo.replay
|
||||
"Local undo/redo history recording and replay.
|
||||
|
||||
Ported from the now-deleted frontend.worker.sync.apply-txs (Fase 8 in
|
||||
docs/agent-guide), which mixed this together with the remote-sync tx
|
||||
upload/apply protocol. This is the part that survives: every local
|
||||
outliner edit gets its forward/inverse outliner ops derived and stashed in
|
||||
frontend.worker.local-tx-log (keyed by tx-id) via persist-local-tx!, then
|
||||
immediately read back and pushed onto the in-memory undo/redo stack by
|
||||
worker-undo-redo/gen-undo-ops! -- the local-tx-log row is a short-lived
|
||||
relay, not a source of truth (the undo-stack entry embeds the ops
|
||||
directly), so it's safe for db_core.cljs's periodic
|
||||
cleanup-finished-history-ops! to reclaim it later. apply-history-action! is
|
||||
the reverse direction: it's what actually runs when the user hits
|
||||
undo/redo, replaying an entry's forward or inverse outliner ops (or, for
|
||||
edits that bypassed the structured outliner-op layer entirely, a raw
|
||||
normalized/reversed tx-data fallback) against the *current* db state --
|
||||
this is registered as worker-undo-redo/*apply-history-action!, the
|
||||
extension point undo_redo.cljs calls on every undo/redo."
|
||||
(:require [datascript.core :as d]
|
||||
[frontend.worker.local-tx-log :as local-tx-log]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.undo-redo :as worker-undo-redo]
|
||||
[lambdaisland.glogi :as log]
|
||||
[logseq.db :as ldb]
|
||||
[logseq.db.common.entity-plus :as entity-plus]
|
||||
[logseq.db.common.normalize :as db-normalize]
|
||||
[logseq.outliner.core :as outliner-core]
|
||||
[logseq.outliner.op :as outliner-op]
|
||||
[logseq.outliner.op.construct :as op-construct]
|
||||
[logseq.outliner.page :as outliner-page]
|
||||
[logseq.outliner.property :as outliner-property]
|
||||
[logseq.outliner.recycle :as outliner-recycle]))
|
||||
|
||||
;; ---- recording: derive + stash + push onto the undo stack ----
|
||||
|
||||
(defn- tx-meta-outliner-op
|
||||
[tx-meta]
|
||||
(or (:outliner-op tx-meta)
|
||||
(when (:db-migrate? tx-meta)
|
||||
:db-migrate)))
|
||||
|
||||
(defn- inferred-outliner-ops?
|
||||
[tx-meta]
|
||||
(and (nil? (:outliner-ops tx-meta))
|
||||
(not (:undo? tx-meta))
|
||||
(not (:redo? tx-meta))
|
||||
(not= :batch-import-edn (:outliner-op tx-meta))))
|
||||
|
||||
(defn- normalize-tx-data [db-after db-before tx-data]
|
||||
(db-normalize/normalize-tx-data db-after db-before tx-data))
|
||||
|
||||
(defn- reverse-tx-data [db-before db-after tx-data]
|
||||
(->> tx-data
|
||||
reverse
|
||||
(keep (fn [[e a v t added]]
|
||||
(let [reversed-datom (d/datom e a v t (not added))]
|
||||
;; trick: reverse the order of `db-before` and `db-after`
|
||||
(db-normalize/normalize-datom db-before db-after reversed-datom))))
|
||||
(db-normalize/replace-attr-retract-with-retract-entity-v2 db-after)
|
||||
db-normalize/reorder-retract-entity))
|
||||
|
||||
(declare apply-history-action!)
|
||||
|
||||
(defn persist-local-tx!
|
||||
"Runs on every committed local tx (see db_listener.cljs's :persist-local-tx
|
||||
handler): derives this tx's forward/inverse outliner ops, stashes them in
|
||||
local-tx-log keyed by tx-id, then hands off to gen-undo-ops! to push the
|
||||
undo-stack entry (and, the first time, register apply-history-action! as
|
||||
the default replay implementation -- same lazy-registration pattern the
|
||||
deleted sync code used). A no-op when the graph has no client-ops conn
|
||||
open (e.g. :publishing)."
|
||||
[repo {:keys [db-before db-after tx-data tx-meta] :as tx-report}]
|
||||
(when (and (worker-state/get-client-ops-conn repo)
|
||||
(:persist-op? tx-meta true))
|
||||
(let [tx-id (or (:db-sync/tx-id tx-meta) (random-uuid))
|
||||
outliner-op (tx-meta-outliner-op tx-meta)
|
||||
{:keys [forward-outliner-ops inverse-outliner-ops]}
|
||||
(op-construct/derive-history-outliner-ops db-before db-after tx-data tx-meta)
|
||||
normalized-tx-data (normalize-tx-data db-after db-before tx-data)
|
||||
reversed-datoms (reverse-tx-data db-before db-after tx-data)]
|
||||
(local-tx-log/upsert-local-tx-entry!
|
||||
repo
|
||||
{:tx-id tx-id
|
||||
:created-at (.now js/Date)
|
||||
;; No upload queue to mark this false later -- write it already
|
||||
;; settled so db_core.cljs's periodic cleanup-finished-history-ops!
|
||||
;; (which only reclaims pending = 0 rows, protecting whatever the
|
||||
;; live undo/redo stack still references) is the sole garbage
|
||||
;; collector for this table.
|
||||
:pending? false
|
||||
:failed? false
|
||||
:outliner-op outliner-op
|
||||
:undo-redo (cond
|
||||
(:undo? tx-meta) :undo
|
||||
(:redo? tx-meta) :redo
|
||||
:else :none)
|
||||
:forward-outliner-ops forward-outliner-ops
|
||||
:inverse-outliner-ops inverse-outliner-ops
|
||||
:inferred-outliner-ops? (inferred-outliner-ops? tx-meta)
|
||||
:normalized-tx-data normalized-tx-data
|
||||
:reversed-tx-data reversed-datoms})
|
||||
(worker-undo-redo/gen-undo-ops! repo tx-report tx-id
|
||||
{:apply-history-action! apply-history-action!})
|
||||
tx-id)))
|
||||
|
||||
;; ---- replay: apply a recorded entry's ops against the current db ----
|
||||
|
||||
(defn- inline-history-action
|
||||
[tx-meta]
|
||||
(let [forward-outliner-ops (or (:db-sync/forward-outliner-ops tx-meta)
|
||||
(:forward-outliner-ops tx-meta))
|
||||
inverse-outliner-ops (or (:db-sync/inverse-outliner-ops tx-meta)
|
||||
(:inverse-outliner-ops tx-meta))]
|
||||
(when (and (seq forward-outliner-ops) (seq inverse-outliner-ops))
|
||||
{:outliner-op (:outliner-op tx-meta)
|
||||
:forward-outliner-ops forward-outliner-ops
|
||||
:inverse-outliner-ops inverse-outliner-ops})))
|
||||
|
||||
(defn- normalize-tx-data-for-rebase
|
||||
[tx-data]
|
||||
(some->> tx-data
|
||||
(mapv (fn [item]
|
||||
(if (and (vector? item) (= 5 (count item)))
|
||||
(let [[op e a v _t] item]
|
||||
[op e a v])
|
||||
item)))
|
||||
db-normalize/reorder-retract-entity))
|
||||
|
||||
(defn- expected-history-action-error-reason?
|
||||
[reason]
|
||||
(contains? #{:invalid-history-action-ops
|
||||
:invalid-history-action-tx}
|
||||
reason))
|
||||
|
||||
(defn- history-action-error-reason
|
||||
[error]
|
||||
(let [message (ex-message error)]
|
||||
(if (or (= "invalid rebase op" message)
|
||||
(= "Non-transact outliner ops contain numeric entity ids" message))
|
||||
:invalid-history-action-ops
|
||||
:error)))
|
||||
|
||||
(defn- invalid-rebase-op!
|
||||
[op data]
|
||||
(throw (ex-info "invalid rebase op" (assoc data :op op))))
|
||||
|
||||
(defn- replay-entity-id-value
|
||||
[db v]
|
||||
(cond
|
||||
(number? v)
|
||||
v
|
||||
|
||||
(uuid? v)
|
||||
(some-> (d/entity db [:block/uuid v]) :db/id)
|
||||
|
||||
(or (vector? v) (qualified-keyword? v))
|
||||
(some-> (d/entity db v) :db/id)
|
||||
|
||||
:else
|
||||
v))
|
||||
|
||||
(defn- replay-entity-id-coll
|
||||
[db ids]
|
||||
(mapv #(or (replay-entity-id-value db %) %) ids))
|
||||
|
||||
(defn- rebase-find-existing-left-sibling
|
||||
[current-db target]
|
||||
(loop [sibling (ldb/get-left-sibling target)]
|
||||
(if (nil? sibling)
|
||||
nil
|
||||
(if-let [current-sibling (and sibling (d/entity current-db [:block/uuid (:block/uuid sibling)]))]
|
||||
current-sibling
|
||||
(recur (ldb/get-left-sibling sibling))))))
|
||||
|
||||
(defn- rebase-target-ref
|
||||
[target-id]
|
||||
(cond
|
||||
(and (vector? target-id)
|
||||
(= :block/uuid (first target-id))
|
||||
(uuid? (second target-id)))
|
||||
target-id
|
||||
|
||||
(uuid? target-id)
|
||||
[:block/uuid target-id]
|
||||
|
||||
(and (map? target-id) (uuid? (:block/uuid target-id)))
|
||||
[:block/uuid (:block/uuid target-id)]
|
||||
|
||||
:else
|
||||
target-id))
|
||||
|
||||
(defn- rebase-resolve-target-and-sibling
|
||||
[current-db rebase-db-before target-id sibling?]
|
||||
(let [target-ref (rebase-target-ref target-id)
|
||||
target (d/entity current-db target-ref)
|
||||
target-before (when rebase-db-before
|
||||
(d/entity rebase-db-before target-ref))
|
||||
parent-before (when rebase-db-before
|
||||
(:block/parent (d/entity rebase-db-before target-ref)))]
|
||||
(cond
|
||||
target
|
||||
[target sibling?]
|
||||
|
||||
(and target-before parent-before sibling?)
|
||||
(if-let [left-sibling (rebase-find-existing-left-sibling current-db target-before)]
|
||||
[left-sibling true]
|
||||
(when-let [parent (d/entity current-db [:block/uuid (:block/uuid parent-before)])]
|
||||
[parent false]))
|
||||
|
||||
:else
|
||||
nil)))
|
||||
|
||||
(defn- template-parent-ref
|
||||
[parent]
|
||||
(cond
|
||||
(and (vector? parent) (= :block/uuid (first parent)))
|
||||
parent
|
||||
|
||||
(uuid? parent)
|
||||
[:block/uuid parent]
|
||||
|
||||
(and (map? parent) (uuid? (:block/uuid parent)))
|
||||
[:block/uuid (:block/uuid parent)]
|
||||
|
||||
:else
|
||||
parent))
|
||||
|
||||
(defn- sanitize-template-block
|
||||
[current-db rebase-db-before block]
|
||||
(let [m (into {} block)
|
||||
block-id (:db/id m)
|
||||
block-uuid (or (:block/uuid m)
|
||||
(when (number? block-id)
|
||||
(or (some-> rebase-db-before (d/entity block-id) :block/uuid)
|
||||
(some-> (d/entity current-db block-id) :block/uuid)))
|
||||
(when (and (vector? block-id)
|
||||
(= :block/uuid (first block-id))
|
||||
(uuid? (second block-id)))
|
||||
(second block-id)))]
|
||||
(cond-> (-> m
|
||||
(dissoc :db/id :block/order :block/page :block/tx-id)
|
||||
(update :block/parent template-parent-ref))
|
||||
(uuid? block-uuid)
|
||||
(assoc :block/uuid block-uuid))))
|
||||
|
||||
(defn- block-ref?
|
||||
[v]
|
||||
(or (number? v)
|
||||
(and (vector? v)
|
||||
(= :block/uuid (first v))
|
||||
(uuid? (second v)))))
|
||||
|
||||
(defn- block-entity
|
||||
[db ref]
|
||||
(when (block-ref? ref)
|
||||
(d/entity db ref)))
|
||||
|
||||
(defn- block-entity-ref
|
||||
[entity]
|
||||
(if-let [block-uuid (:block/uuid entity)]
|
||||
[:block/uuid block-uuid]
|
||||
(:db/id entity)))
|
||||
|
||||
(defn- block-descendants
|
||||
[entity]
|
||||
(letfn [(collect [block]
|
||||
(mapcat (fn [child]
|
||||
(cons child (collect child)))
|
||||
(sort-by :block/order
|
||||
(entity-plus/lookup-kv-then-entity block :block/_raw-parent))))]
|
||||
(collect entity)))
|
||||
|
||||
(defn- retract-entity-op?
|
||||
[item]
|
||||
(and (vector? item)
|
||||
(contains? #{:db/retractEntity :db.fn/retractEntity} (first item))))
|
||||
|
||||
(defn- expand-block-retracts-to-descendants
|
||||
[db tx-data]
|
||||
(let [explicit-retracts (->> tx-data
|
||||
(keep (fn [item]
|
||||
(when (and (retract-entity-op? item)
|
||||
(block-ref? (second item)))
|
||||
(some-> (block-entity db (second item)) :db/id))))
|
||||
set)]
|
||||
(mapcat
|
||||
(fn [item]
|
||||
(if (and (retract-entity-op? item)
|
||||
(block-ref? (second item))
|
||||
(block-entity db (second item)))
|
||||
(let [root (block-entity db (second item))]
|
||||
(concat (->> (block-descendants root)
|
||||
(remove #(contains? explicit-retracts (:db/id %)))
|
||||
(map (fn [entity]
|
||||
[:db/retractEntity (block-entity-ref entity)])))
|
||||
[item]))
|
||||
[item]))
|
||||
tx-data)))
|
||||
|
||||
(defn- ^:large-vars/cleanup-todo replay-canonical-outliner-op!
|
||||
[conn [op args] rebase-db-before]
|
||||
(case op
|
||||
:save-block
|
||||
(let [[block opts] args
|
||||
db @conn
|
||||
block-uuid (:block/uuid block)
|
||||
block-ent (when block-uuid
|
||||
(d/entity db [:block/uuid block-uuid]))
|
||||
block-base (dissoc block :db/id :block/order)
|
||||
block' (merge block-base
|
||||
(op-construct/rewrite-block-title-with-retracted-refs db block-base))]
|
||||
(when (nil? block-ent)
|
||||
(invalid-rebase-op! op {:args args
|
||||
:reason :missing-block}))
|
||||
(outliner-core/save-block! conn block' opts))
|
||||
|
||||
:insert-blocks
|
||||
(let [[blocks target-id opts] args
|
||||
db @conn
|
||||
[target sibling?] (rebase-resolve-target-and-sibling db rebase-db-before target-id (:sibling? opts))]
|
||||
(when-not (and target (seq blocks))
|
||||
(invalid-rebase-op! op {:args args}))
|
||||
(outliner-core/insert-blocks! conn
|
||||
(mapv #(op-construct/rewrite-block-title-with-retracted-refs db %) blocks)
|
||||
target
|
||||
(assoc opts :sibling? sibling?)))
|
||||
|
||||
:apply-template
|
||||
(let [[template-id target-id opts] args
|
||||
template-id' (replay-entity-id-value @conn template-id)
|
||||
target-id' (replay-entity-id-value @conn target-id)
|
||||
[target sibling?] (rebase-resolve-target-and-sibling @conn rebase-db-before target-id' (:sibling? opts))]
|
||||
(when-not (and template-id' (d/entity @conn template-id') target)
|
||||
(invalid-rebase-op! op {:args args
|
||||
:reason :missing-template-or-target-block}))
|
||||
(let [template-uuid (:block/uuid (d/entity @conn template-id'))
|
||||
target-uuid (:block/uuid target)]
|
||||
(when-not (and (uuid? template-uuid) (uuid? target-uuid))
|
||||
(invalid-rebase-op! op {:args args
|
||||
:reason :missing-template-or-target-uuid}))
|
||||
(let [replace-empty-target? (:replace-empty-target? opts)
|
||||
template-blocks' (some->> (:template-blocks opts)
|
||||
(map-indexed
|
||||
(fn [idx block]
|
||||
(let [block' (sanitize-template-block @conn rebase-db-before block)
|
||||
block'' (if (and replace-empty-target?
|
||||
(zero? idx)
|
||||
(nil? (:block/uuid block')))
|
||||
;; Keep replace-empty-target replay consistent with
|
||||
;; initial apply-template payload where the first
|
||||
;; block uuid is the target uuid.
|
||||
(assoc block' :block/uuid target-uuid)
|
||||
block')]
|
||||
(when (:block/uuid block'')
|
||||
block''))))
|
||||
(remove nil?)
|
||||
seq
|
||||
vec)
|
||||
opts' (cond-> (-> opts
|
||||
(assoc :sibling? sibling?)
|
||||
(dissoc :template-blocks))
|
||||
template-blocks'
|
||||
(assoc :template-blocks template-blocks'))]
|
||||
(outliner-op/apply-ops!
|
||||
conn
|
||||
[[:apply-template [template-uuid
|
||||
target-uuid
|
||||
opts']]]
|
||||
{:gen-undo-ops? false}))))
|
||||
|
||||
:move-blocks
|
||||
(let [[ids target-id opts] args
|
||||
ids' (replay-entity-id-coll @conn ids)
|
||||
target-id' (replay-entity-id-value @conn target-id)
|
||||
blocks (keep #(d/entity @conn %) ids')
|
||||
[target sibling?] (rebase-resolve-target-and-sibling @conn rebase-db-before target-id' (:sibling? opts))]
|
||||
(when (or (empty? blocks) (nil? target))
|
||||
(invalid-rebase-op! op {:args args}))
|
||||
(when (seq blocks)
|
||||
(outliner-core/move-blocks! conn blocks target (assoc opts :sibling? sibling?))))
|
||||
|
||||
:move-blocks-up-down
|
||||
(let [[ids up?] args
|
||||
ids' (replay-entity-id-coll @conn ids)
|
||||
blocks (keep #(d/entity @conn %) ids')]
|
||||
(when (seq blocks)
|
||||
(outliner-core/move-blocks-up-down! conn blocks up?)))
|
||||
|
||||
:indent-outdent-blocks
|
||||
(let [[ids indent? opts] args
|
||||
ids' (replay-entity-id-coll @conn ids)
|
||||
blocks (keep #(d/entity @conn %) ids')]
|
||||
(when (empty? blocks)
|
||||
(invalid-rebase-op! op {:args args}))
|
||||
(when (seq blocks)
|
||||
(outliner-core/indent-outdent-blocks! conn blocks indent? opts)))
|
||||
|
||||
:delete-blocks
|
||||
(let [[ids opts] args
|
||||
ids' (replay-entity-id-coll @conn ids)
|
||||
blocks (keep #(d/entity @conn %) ids')]
|
||||
;; Keep delete replay idempotent under concurrent edits where blocks may already
|
||||
;; be gone, but still leave a debug breadcrumb for malformed/missing targets.
|
||||
(when (empty? blocks)
|
||||
(log/debug :undo-redo/drop-delete-blocks-replay
|
||||
{:args args}))
|
||||
(when (seq blocks)
|
||||
(outliner-core/delete-blocks! conn blocks opts)))
|
||||
|
||||
:create-page
|
||||
(let [[title opts] args
|
||||
page-uuid (:uuid opts)
|
||||
existing-page (or (when (uuid? page-uuid)
|
||||
(d/entity @conn [:block/uuid page-uuid]))
|
||||
(ldb/get-page @conn title))]
|
||||
(if (and existing-page
|
||||
(not (ldb/recycled? existing-page)))
|
||||
[(:block/title existing-page) (:block/uuid existing-page)]
|
||||
(outliner-page/create! conn title opts)))
|
||||
|
||||
:delete-page
|
||||
(let [[page-uuid opts] args]
|
||||
(outliner-page/delete! conn page-uuid opts))
|
||||
|
||||
:upsert-property
|
||||
(apply outliner-property/upsert-property! conn args)
|
||||
|
||||
:restore-recycled
|
||||
(let [[root-id] args
|
||||
root-ref (cond
|
||||
(and (vector? root-id)
|
||||
(= :block/uuid (first root-id)))
|
||||
root-id
|
||||
|
||||
(uuid? root-id)
|
||||
[:block/uuid root-id]
|
||||
|
||||
:else
|
||||
root-id)
|
||||
root (d/entity @conn root-ref)
|
||||
tx-data (when root
|
||||
(seq (outliner-recycle/restore-tx-data @conn root)))]
|
||||
(when-not tx-data
|
||||
(invalid-rebase-op! op {:args args
|
||||
:reason :invalid-restore-target}))
|
||||
(ldb/transact! conn tx-data
|
||||
{:outliner-op :restore-recycled}))
|
||||
|
||||
:recycle-delete-permanently
|
||||
(let [[root-id] args
|
||||
root-ref (cond
|
||||
(and (vector? root-id)
|
||||
(= :block/uuid (first root-id)))
|
||||
root-id
|
||||
|
||||
(uuid? root-id)
|
||||
[:block/uuid root-id]
|
||||
|
||||
:else
|
||||
root-id)
|
||||
root (d/entity @conn root-ref)
|
||||
tx-data (when root
|
||||
(seq (outliner-recycle/permanently-delete-tx-data @conn root)))]
|
||||
;; Keep replay idempotent under concurrent edits where the recycled root may
|
||||
;; already be permanently removed by a preceding tx.
|
||||
(when (seq tx-data)
|
||||
(ldb/transact! conn tx-data
|
||||
{:outliner-op :recycle-delete-permanently})))
|
||||
|
||||
(let [[tx-data tx-meta] args
|
||||
tx-data (expand-block-retracts-to-descendants @conn tx-data)]
|
||||
(when-let [tx-data (seq tx-data)]
|
||||
(ldb/transact! conn tx-data tx-meta)))))
|
||||
|
||||
(defn ^:large-vars/cleanup-todo apply-history-action!
|
||||
"The default worker-undo-redo/*apply-history-action! implementation:
|
||||
replays a recorded entry's forward or inverse outliner ops (undo?
|
||||
selects which) against the *current* db state, since it may have moved
|
||||
on since the entry was recorded."
|
||||
[repo tx-id undo? tx-meta]
|
||||
(if-let [conn (worker-state/get-datascript-conn repo)]
|
||||
(if-let [action (or (local-tx-log/get-local-tx-entry repo tx-id)
|
||||
(inline-history-action tx-meta))]
|
||||
(if (= :fix (:outliner-op action))
|
||||
{:applied? false
|
||||
:reason :unsupported-history-action
|
||||
:action action}
|
||||
(let [{:keys [tx reversed-tx forward-outliner-ops inverse-outliner-ops]} action
|
||||
ops (->> (if undo? inverse-outliner-ops forward-outliner-ops)
|
||||
(filter (fn [op-entry]
|
||||
(and (sequential? op-entry)
|
||||
(contains? op-construct/semantic-outliner-ops
|
||||
(first op-entry)))))
|
||||
seq)
|
||||
tx-data (-> (if undo? reversed-tx tx)
|
||||
normalize-tx-data-for-rebase)
|
||||
ops' (if (seq ops)
|
||||
ops
|
||||
[[:transact [tx-data nil]]])
|
||||
history-tx-id (let [provided-history-tx-id (:db-sync/tx-id tx-meta)]
|
||||
(if (and (uuid? provided-history-tx-id)
|
||||
(not= provided-history-tx-id tx-id))
|
||||
provided-history-tx-id
|
||||
(random-uuid)))
|
||||
tx-meta' (cond-> {:outliner-op (:outliner-op action)
|
||||
:local-tx? true
|
||||
:gen-undo-ops? false
|
||||
:persist-op? true
|
||||
:undo? undo?
|
||||
:redo? (:redo? tx-meta)
|
||||
:db-sync/tx-id history-tx-id
|
||||
:db-sync/source-tx-id (or (:db-sync/source-tx-id tx-meta)
|
||||
tx-id)
|
||||
:db-sync/forward-outliner-ops (if undo?
|
||||
(:inverse-outliner-ops action)
|
||||
(:forward-outliner-ops action))
|
||||
:db-sync/inverse-outliner-ops (if undo?
|
||||
(:forward-outliner-ops action)
|
||||
(:inverse-outliner-ops action))})]
|
||||
(if (seq ops')
|
||||
(try
|
||||
(when (seq ops)
|
||||
(op-construct/assert-no-numeric-entity-ids! @conn ops :history-action-ops))
|
||||
(ldb/batch-transact-with-temp-conn!
|
||||
conn
|
||||
tx-meta'
|
||||
(fn [conn]
|
||||
(doseq [op ops']
|
||||
(replay-canonical-outliner-op! conn op nil))))
|
||||
{:applied? true
|
||||
:history-tx-id history-tx-id}
|
||||
(catch :default e
|
||||
(let [reason (history-action-error-reason e)]
|
||||
(when-not (expected-history-action-error-reason? reason)
|
||||
(log/error ::undo-redo-failed e))
|
||||
{:applied? false
|
||||
:reason reason
|
||||
:action action})))
|
||||
{:applied? false :reason :unsupported-history-action
|
||||
:action action})))
|
||||
{:applied? false
|
||||
:reason :missing-history-action
|
||||
:tx-id tx-id})
|
||||
(let [data {:repo repo :op :apply-history-action}]
|
||||
(log/error :undo-redo/missing-db data)
|
||||
(throw (ex-info "missing-db" data)))))
|
||||
@@ -21,10 +21,8 @@
|
||||
[frontend.worker.query-dsl :as worker-query-dsl]
|
||||
[frontend.worker.search :as search]
|
||||
[frontend.worker.shared-service :as shared-service]
|
||||
[frontend.worker.local-tx-log :as local-tx-log]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync :as db-sync]
|
||||
[frontend.worker.sync.client-op :as client-op]
|
||||
[frontend.worker.sync.download :as sync-download]
|
||||
[frontend.worker.undo-redo :as worker-undo-redo]
|
||||
[goog.object :as gobj]
|
||||
[logseq.common.config :as common-config]
|
||||
@@ -51,13 +49,7 @@
|
||||
(def ^:private expected-db-core-thread-apis
|
||||
(into #{}
|
||||
(concat
|
||||
[:thread-api/list-db :thread-api/init :thread-api/set-db-sync-config :thread-api/get-db-sync-config :thread-api/get-key-value
|
||||
:thread-api/db-sync-status :thread-api/db-sync-start :thread-api/db-sync-stop :thread-api/db-sync-update-presence
|
||||
:thread-api/db-sync-request-asset-download :thread-api/db-sync-grant-graph-access :thread-api/db-sync-ensure-user-rsa-keys
|
||||
:thread-api/db-sync-list-remote-graphs :thread-api/db-sync-upload-graph :thread-api/db-sync-create-remote-graph
|
||||
:thread-api/db-sync-stop-upload :thread-api/db-sync-resume-upload :thread-api/db-sync-upload-stopped?
|
||||
:thread-api/db-sync-get-all-block-conflicts :thread-api/db-sync-clear-block-conflicts
|
||||
:thread-api/db-sync-download-graph-by-id
|
||||
[:thread-api/list-db :thread-api/init :thread-api/get-key-value
|
||||
:thread-api/create-or-open-db :thread-api/q :thread-api/datoms :thread-api/pull :thread-api/task-spent-time :thread-api/get-blocks
|
||||
:thread-api/get-block-refs :thread-api/get-block-source
|
||||
:thread-api/get-block-parents :thread-api/set-context :thread-api/transact :thread-api/undo-redo-set-pending-editor-info
|
||||
@@ -66,9 +58,7 @@
|
||||
:thread-api/build-publishing-html :thread-api/reset-db
|
||||
:thread-api/get-file-content :thread-api/get-all-properties :thread-api/get-date-scheduled-or-deadlines
|
||||
:thread-api/unsafe-unlink-db :thread-api/close-db
|
||||
:thread-api/db-sync-close-db :thread-api/db-sync-invalidate-search-db :thread-api/db-sync-recreate-lock
|
||||
:thread-api/db-sync-rehydrate-large-titles :thread-api/db-sync-import-prepare :thread-api/db-sync-import-rows-chunk
|
||||
:thread-api/db-sync-import-finalize :thread-api/release-access-handles :thread-api/db-exists
|
||||
:thread-api/release-access-handles :thread-api/db-exists
|
||||
:thread-api/export-db-binary :thread-api/import-file-graph
|
||||
:thread-api/export-client-ops-db-binary :thread-api/backup-db-sqlite
|
||||
:thread-api/import-db-binary :thread-api/search-blocks :thread-api/search-upsert-blocks :thread-api/search-delete-blocks
|
||||
@@ -76,7 +66,7 @@
|
||||
:thread-api/search-build-pages-indice :thread-api/apply-outliner-ops :thread-api/sync-app-state
|
||||
:thread-api/markdown-mirror-set-enabled :thread-api/markdown-mirror-flush :thread-api/markdown-mirror-regenerate
|
||||
:thread-api/export-get-debug-datoms :thread-api/export-get-all-page->content :thread-api/validate-db
|
||||
:thread-api/recompute-checksum-diagnostics :thread-api/export-edn :thread-api/import-edn
|
||||
:thread-api/export-edn :thread-api/import-edn
|
||||
:thread-api/get-fsrs-due-card-block-ids :thread-api/get-view-data
|
||||
:thread-api/get-class-objects :thread-api/validate-block-tag
|
||||
:thread-api/convert-tag-to-page
|
||||
@@ -434,7 +424,6 @@
|
||||
(platform/set-platform! platform')
|
||||
(p/with-redefs
|
||||
[shared-service/*master-client? (atom true)
|
||||
db-sync/handle-local-tx! (fn [& _] nil)
|
||||
shared-service/broadcast-to-clients! (fn [& args]
|
||||
(swap! broadcasts conj args))
|
||||
db-listener/listen-db-changes!
|
||||
@@ -477,17 +466,6 @@
|
||||
(is false (str error))))
|
||||
(p/finally done))))
|
||||
|
||||
(deftest db-core-registers-db-sync-thread-apis
|
||||
(let [api-map @thread-api/*thread-apis]
|
||||
(is (contains? api-map :thread-api/set-db-sync-config))
|
||||
(is (contains? api-map :thread-api/db-sync-start))
|
||||
(is (contains? api-map :thread-api/db-sync-stop))
|
||||
(is (contains? api-map :thread-api/db-sync-update-presence))
|
||||
(is (contains? api-map :thread-api/db-sync-request-asset-download))
|
||||
(is (contains? api-map :thread-api/db-sync-grant-graph-access))
|
||||
(is (contains? api-map :thread-api/db-sync-ensure-user-rsa-keys))
|
||||
(is (contains? api-map :thread-api/db-sync-upload-graph))))
|
||||
|
||||
(deftest resolve-initial-config-falls-back-to-template-config-test
|
||||
(let [resolve-initial-config #'db-core/resolve-initial-config
|
||||
template-config (rc/inline "templates/config.edn")]
|
||||
@@ -532,53 +510,6 @@
|
||||
(is false (str "expected import to succeed with active pool, got " error))))))))
|
||||
(p/finally done))))
|
||||
|
||||
(deftest set-db-sync-config-keeps-only-non-auth-fields-test
|
||||
(let [set-config! (get @thread-api/*thread-apis :thread-api/set-db-sync-config)
|
||||
get-config (get @thread-api/*thread-apis :thread-api/get-db-sync-config)
|
||||
state-prev @worker-state/*state
|
||||
config-prev @worker-state/*db-sync-config]
|
||||
(try
|
||||
(reset! worker-state/*state (assoc state-prev
|
||||
:auth/id-token "existing-id-token"
|
||||
:auth/oauth-token-url "https://existing.example.com/oauth2/token"
|
||||
:auth/oauth-domain "existing.example.com"
|
||||
:auth/oauth-client-id "existing-client-id"))
|
||||
(set-config! {:ws-url "wss://example.com/sync/%s"
|
||||
:http-base "https://example.com"
|
||||
:enabled? true
|
||||
:auth-token "id-token-from-config"
|
||||
:oauth-token-url "https://auth.example.com/oauth2/token"
|
||||
:oauth-domain "auth.example.com"
|
||||
:oauth-client-id "worker-client-id"})
|
||||
(is (= {:ws-url "wss://example.com/sync/%s"
|
||||
:http-base "https://example.com"
|
||||
:enabled? true}
|
||||
@worker-state/*db-sync-config))
|
||||
(is (= {:ws-url "wss://example.com/sync/%s"
|
||||
:http-base "https://example.com"
|
||||
:enabled? true}
|
||||
(get-config)))
|
||||
(is (= "existing-id-token" (:auth/id-token @worker-state/*state)))
|
||||
(is (= "https://existing.example.com/oauth2/token"
|
||||
(:auth/oauth-token-url @worker-state/*state)))
|
||||
(is (= "existing.example.com" (:auth/oauth-domain @worker-state/*state)))
|
||||
(is (= "existing-client-id" (:auth/oauth-client-id @worker-state/*state)))
|
||||
(finally
|
||||
(reset! worker-state/*state state-prev)
|
||||
(reset! worker-state/*db-sync-config config-prev)))))
|
||||
|
||||
(deftest get-db-sync-config-strips-auth-fields-test
|
||||
(let [get-config (get @thread-api/*thread-apis :thread-api/get-db-sync-config)
|
||||
config-prev @worker-state/*db-sync-config]
|
||||
(try
|
||||
(reset! worker-state/*db-sync-config {:ws-url "wss://example.com/sync/%s"
|
||||
:auth-token "leaked-token"
|
||||
:oauth-client-id "leaked-client"})
|
||||
(is (= {:ws-url "wss://example.com/sync/%s"}
|
||||
(get-config)))
|
||||
(finally
|
||||
(reset! worker-state/*db-sync-config config-prev)))))
|
||||
|
||||
(deftest init-service-does-not-close-db-when-graph-unchanged
|
||||
(async done
|
||||
(let [service {:status {:ready (p/resolved true)}
|
||||
@@ -605,24 +536,6 @@
|
||||
(reset! *service old-service)
|
||||
(done))))))))
|
||||
|
||||
(deftest handle-migrate-result-local-txs-enqueues-upgrade-reports-test
|
||||
(let [conn (d/create-conn db-schema/schema)
|
||||
migration-tx-report {:tx-data [[:db/add 1 :block/title "Migrated"]]
|
||||
:db-before @conn
|
||||
:db-after @conn
|
||||
:tx-meta {:db-migrate? true}}
|
||||
handled-reports (atom [])]
|
||||
(with-redefs [db-sync/handle-local-tx! (fn [repo tx-report]
|
||||
(swap! handled-reports conj
|
||||
{:repo repo
|
||||
:tx-report tx-report}))]
|
||||
(#'db-core/handle-migrate-result-local-txs!
|
||||
test-repo
|
||||
{:upgrade-result-coll [migration-tx-report]})
|
||||
(is (= [{:repo test-repo
|
||||
:tx-report migration-tx-report}]
|
||||
@handled-reports)))))
|
||||
|
||||
(deftest built-in-sync-repair-tx-data-covers-65-26-through-65-28-test
|
||||
(let [tx-data (#'db-core/built-in-sync-repair-tx-data)
|
||||
tx-data-again (#'db-core/built-in-sync-repair-tx-data)
|
||||
@@ -663,12 +576,12 @@
|
||||
(deftest enqueue-built-in-sync-repair-queues-pending-fix-once-test
|
||||
(let [upserts (atom [])
|
||||
count-adjustments (atom [])]
|
||||
(with-redefs [client-op/get-local-tx-entry (fn [_repo _tx-id] nil)
|
||||
client-op/upsert-local-tx-entry! (fn [repo entry]
|
||||
(with-redefs [local-tx-log/get-local-tx-entry (fn [_repo _tx-id] nil)
|
||||
local-tx-log/upsert-local-tx-entry! (fn [repo entry]
|
||||
(swap! upserts conj {:repo repo
|
||||
:entry entry})
|
||||
{:should-inc-pending? true})
|
||||
client-op/adjust-pending-local-tx-count! (fn [repo delta]
|
||||
local-tx-log/adjust-pending-local-tx-count! (fn [repo delta]
|
||||
(swap! count-adjustments conj [repo delta]))]
|
||||
(#'db-core/enqueue-built-in-sync-repair! test-repo)
|
||||
(let [{:keys [repo entry]} (first @upserts)]
|
||||
@@ -683,10 +596,10 @@
|
||||
|
||||
(reset! upserts [])
|
||||
(reset! count-adjustments [])
|
||||
(with-redefs [client-op/get-local-tx-entry (fn [_repo _tx-id] {:tx-id (random-uuid)})
|
||||
client-op/upsert-local-tx-entry! (fn [_repo _entry]
|
||||
(with-redefs [local-tx-log/get-local-tx-entry (fn [_repo _tx-id] {:tx-id (random-uuid)})
|
||||
local-tx-log/upsert-local-tx-entry! (fn [_repo _entry]
|
||||
(swap! upserts conj :unexpected))
|
||||
client-op/adjust-pending-local-tx-count! (fn [_repo _delta]
|
||||
local-tx-log/adjust-pending-local-tx-count! (fn [_repo _delta]
|
||||
(swap! count-adjustments conj :unexpected))]
|
||||
(#'db-core/enqueue-built-in-sync-repair! test-repo)
|
||||
(is (empty? @upserts))
|
||||
@@ -700,12 +613,12 @@
|
||||
:kv/value true}])
|
||||
(d/transact! local-conn [{:db/ident :logseq.kv/graph-remote?
|
||||
:kv/value false}])
|
||||
(with-redefs [client-op/get-local-tx-entry (fn [_repo _tx-id] nil)
|
||||
client-op/upsert-local-tx-entry! (fn [repo entry]
|
||||
(with-redefs [local-tx-log/get-local-tx-entry (fn [_repo _tx-id] nil)
|
||||
local-tx-log/upsert-local-tx-entry! (fn [repo entry]
|
||||
(swap! upserts conj {:repo repo
|
||||
:entry entry})
|
||||
{:should-inc-pending? false})
|
||||
client-op/adjust-pending-local-tx-count! (fn [_repo _delta] nil)]
|
||||
local-tx-log/adjust-pending-local-tx-count! (fn [_repo _delta] nil)]
|
||||
(#'db-core/maybe-enqueue-built-in-sync-repair! test-repo remote-conn nil true)
|
||||
(is (= 1 (count @upserts)))
|
||||
|
||||
@@ -1237,68 +1150,27 @@
|
||||
(is (= "Alpha"
|
||||
(vector-embedding-title {:title "Alpha"})))))
|
||||
|
||||
(deftest release-access-handles-clears-active-import-state-test
|
||||
(deftest close-db-closes-all-conns-and-pauses-pool-test
|
||||
(restoring-worker-state
|
||||
(fn []
|
||||
(let [release-access-handles! (get @thread-api/*thread-apis :thread-api/release-access-handles)
|
||||
*import-state @#'sync-download/*import-state
|
||||
import-state-prev @*import-state
|
||||
closed (atom [])
|
||||
(let [closed (atom [])
|
||||
removed-pools (atom [])
|
||||
pause-calls (atom 0)]
|
||||
(try
|
||||
(platform/set-platform! (build-test-platform {:remove-vfs! (fn [pool]
|
||||
(swap! removed-pools conj pool)
|
||||
(p/resolved nil))}))
|
||||
(reset! worker-state/*opfs-pools {test-repo #js {:pauseVfs (fn [] (swap! pause-calls inc))}})
|
||||
(reset! *import-state {:repo test-repo
|
||||
:graph-id "graph-1"
|
||||
:import-id "import-1"
|
||||
:rows-db #js {:close (fn [] (swap! closed conj :rows-db))}
|
||||
:rows-pool :rows-pool})
|
||||
(platform/set-platform! (build-test-platform {:remove-vfs! (fn [pool]
|
||||
(swap! removed-pools conj pool)
|
||||
(p/resolved nil))}))
|
||||
(reset! worker-state/*sqlite-conns
|
||||
{test-repo {:db (fake-db {:close-calls closed :close-label :db})
|
||||
:search (fake-db {:close-calls closed :close-label :search})
|
||||
:client-ops (fake-db {:close-calls closed :close-label :client-ops})}})
|
||||
(reset! worker-state/*datascript-conns {test-repo :datascript})
|
||||
(reset! worker-state/*client-ops-conns {test-repo :client-ops})
|
||||
(reset! worker-state/*opfs-pools {test-repo #js {:pauseVfs (fn [] (swap! pause-calls inc))}})
|
||||
|
||||
(release-access-handles! test-repo)
|
||||
(db-core/close-db! test-repo)
|
||||
|
||||
(is (nil? @*import-state))
|
||||
(is (= [:rows-db] @closed))
|
||||
(is (= [:rows-pool] @removed-pools))
|
||||
(is (= 1 @pause-calls))
|
||||
(finally
|
||||
(reset! *import-state import-state-prev)))))))
|
||||
|
||||
(deftest close-db-clears-active-import-state-test
|
||||
(restoring-worker-state
|
||||
(fn []
|
||||
(let [*import-state @#'sync-download/*import-state
|
||||
import-state-prev @*import-state
|
||||
closed (atom [])
|
||||
removed-pools (atom [])
|
||||
pause-calls (atom 0)]
|
||||
(try
|
||||
(platform/set-platform! (build-test-platform {:remove-vfs! (fn [pool]
|
||||
(swap! removed-pools conj pool)
|
||||
(p/resolved nil))}))
|
||||
(reset! worker-state/*sqlite-conns
|
||||
{test-repo {:db (fake-db {:close-calls closed :close-label :db})
|
||||
:search (fake-db {:close-calls closed :close-label :search})
|
||||
:client-ops (fake-db {:close-calls closed :close-label :client-ops})}})
|
||||
(reset! worker-state/*datascript-conns {test-repo :datascript})
|
||||
(reset! worker-state/*client-ops-conns {test-repo :client-ops})
|
||||
(reset! worker-state/*opfs-pools {test-repo #js {:pauseVfs (fn [] (swap! pause-calls inc))}})
|
||||
(reset! *import-state {:repo test-repo
|
||||
:graph-id "graph-1"
|
||||
:import-id "import-1"
|
||||
:rows-db #js {:close (fn [] (swap! closed conj :rows-db))}
|
||||
:rows-pool :rows-pool})
|
||||
|
||||
(db-core/close-db! test-repo)
|
||||
|
||||
(is (nil? @*import-state))
|
||||
(is (= #{:rows-db :db :search :client-ops} (set @closed)))
|
||||
(is (= [:rows-pool] @removed-pools))
|
||||
(is (= 1 @pause-calls))
|
||||
(finally
|
||||
(reset! *import-state import-state-prev)))))))
|
||||
(is (= #{:db :search :client-ops} (set @closed)))
|
||||
(is (= 1 @pause-calls))))))
|
||||
|
||||
;; ---- ->uint8array tests ----
|
||||
|
||||
@@ -1612,25 +1484,6 @@
|
||||
|
||||
;; ---- checksum-diagnostics tests ----
|
||||
|
||||
(deftest checksum-diagnostics-returns-local-and-remote
|
||||
(restoring-worker-state
|
||||
(fn []
|
||||
(let [checksum-diagnostics #'worker-export-handler/checksum-diagnostics]
|
||||
(with-redefs [client-op/get-local-checksum (fn [_] "local-checksum-123")
|
||||
db-sync/*repo->latest-remote-checksum (atom {test-repo "remote-checksum-456"})]
|
||||
(is (= {:local-checksum "local-checksum-123"
|
||||
:remote-checksum "remote-checksum-456"}
|
||||
(checksum-diagnostics test-repo))))))))
|
||||
|
||||
(deftest checksum-diagnostics-handles-missing-remote
|
||||
(restoring-worker-state
|
||||
(fn []
|
||||
(let [checksum-diagnostics #'worker-export-handler/checksum-diagnostics]
|
||||
(with-redefs [client-op/get-local-checksum (fn [_] "local-checksum-123")
|
||||
db-sync/*repo->latest-remote-checksum (atom {})]
|
||||
(is (= {:local-checksum "local-checksum-123"
|
||||
:remote-checksum nil}
|
||||
(checksum-diagnostics test-repo))))))))
|
||||
|
||||
;; ---- get-all-page-titles tests ----
|
||||
|
||||
@@ -1687,12 +1540,9 @@
|
||||
(let [types db-core/broadcast-data-types]
|
||||
(is (set? types))
|
||||
(is (contains? types "sync-db-changes"))
|
||||
(is (contains? types "sync-conflicts-updated"))
|
||||
(is (contains? types "notification"))
|
||||
(is (contains? types "log"))
|
||||
(is (contains? types "add-repo"))
|
||||
(is (contains? types "rtc-log"))
|
||||
(is (contains? types "rtc-sync-state"))))
|
||||
(is (contains? types "add-repo"))))
|
||||
|
||||
;; ---- init-core! tests ----
|
||||
|
||||
@@ -2314,8 +2164,7 @@
|
||||
first
|
||||
:block)]
|
||||
(is (contains? hydrated :block.temp/display-properties))
|
||||
(is (contains? hydrated :block.temp/task-spent-time))
|
||||
(is (contains? hydrated :block.temp/sync-conflicts))))))))
|
||||
(is (contains? hydrated :block.temp/task-spent-time))))))))
|
||||
|
||||
(deftest get-blocks-includes-render-critical-property-data
|
||||
(restoring-worker-state
|
||||
@@ -2580,9 +2429,7 @@
|
||||
(reset! worker-state/*datascript-conns {test-repo conn})
|
||||
(ldb/register-transact-pipeline-fn! worker-pipeline/transact-pipeline)
|
||||
(p/with-redefs
|
||||
[db-sync/update-local-sync-checksum! (fn [& _] nil)
|
||||
db-sync/handle-local-tx! (fn [& _] nil)
|
||||
shared-service/broadcast-to-clients!
|
||||
[shared-service/broadcast-to-clients!
|
||||
(fn [event payload]
|
||||
(when (= :sync-db-changes event)
|
||||
(swap! renderer-payloads conj payload)))]
|
||||
@@ -3314,42 +3161,6 @@
|
||||
(is (empty? missing) (str "Missing thread apis: " missing))
|
||||
(is (empty? non-functions) (str "Non-function thread apis: " non-functions))))
|
||||
|
||||
(deftest db-sync-get-all-block-conflicts-thread-api-is-explicit-and-fail-fast-test
|
||||
(restoring-worker-state
|
||||
(fn []
|
||||
(let [repo "graph-a"
|
||||
block-uuid (random-uuid)
|
||||
conflicts [{:block-uuid block-uuid
|
||||
:value "remote"}]
|
||||
api (get @thread-api/*thread-apis
|
||||
:thread-api/db-sync-get-all-block-conflicts)]
|
||||
(is (nil? (get @thread-api/*thread-apis
|
||||
:thread-api/db-sync-get-block-conflicts))
|
||||
"Per-block reads must not coexist with graph hydration.")
|
||||
(is (fn? api)
|
||||
"The restore path needs one explicit worker API for graph conflict hydration.")
|
||||
(when (fn? api)
|
||||
(let [calls (atom [])]
|
||||
(with-redefs [client-op/get-all-sync-conflicts
|
||||
(fn [repo']
|
||||
(swap! calls conj repo')
|
||||
{block-uuid conflicts})]
|
||||
(is (= {(str block-uuid) conflicts}
|
||||
(api repo))
|
||||
"The API returns a graph snapshot keyed for direct renderer-state replacement."))
|
||||
(is (= [repo] @calls)))
|
||||
(with-redefs [client-op/get-all-sync-conflicts (constantly nil)]
|
||||
(let [error (try
|
||||
(api repo)
|
||||
nil
|
||||
(catch :default error
|
||||
error))]
|
||||
(is (instance? js/Error error)
|
||||
"A missing client-op store must fail instead of looking like an empty graph.")
|
||||
(is (= {:repo repo
|
||||
:conflicts nil}
|
||||
(ex-data error))))))))))
|
||||
|
||||
(deftest worker-export-replaces-block-refs-with-worker-db-test
|
||||
(let [conn (d/create-conn db-schema/schema)
|
||||
page-id -1
|
||||
@@ -3413,9 +3224,7 @@
|
||||
(reset! worker-state/*datascript-conns {test-repo dest-conn})
|
||||
(ldb/register-transact-pipeline-fn! worker-pipeline/transact-pipeline)
|
||||
(try
|
||||
(with-redefs [db-sync/update-local-sync-checksum! (fn [& _] nil)
|
||||
db-sync/handle-local-tx! (fn [& _] nil)
|
||||
shared-service/broadcast-to-clients!
|
||||
(with-redefs [shared-service/broadcast-to-clients!
|
||||
(fn [event payload]
|
||||
(when (= :sync-db-changes event)
|
||||
(swap! renderer-payloads conj payload)))]
|
||||
|
||||
@@ -7,7 +7,6 @@
|
||||
[frontend.worker.platform :as platform]
|
||||
[frontend.worker.render-delta :as render-delta]
|
||||
[frontend.worker.shared-service :as shared-service]
|
||||
[frontend.worker.sync :as db-sync]
|
||||
[logseq.db :as ldb]
|
||||
[logseq.db.test.helper :as db-test]
|
||||
[promesa.core :as p]))
|
||||
@@ -78,25 +77,6 @@
|
||||
(is (= [["repo" nil tx-report {:defer? true}]]
|
||||
@calls))))
|
||||
|
||||
(deftest db-listener-persists-local-tx-before-broadcasting-ui-refresh-test
|
||||
(let [conn (db-test/create-conn)
|
||||
calls (atom [])]
|
||||
(with-redefs [db-sync/update-local-sync-checksum! (fn [& _] nil)
|
||||
db-sync/handle-local-tx! (fn [& _]
|
||||
(swap! calls conj :persist-local-tx))
|
||||
worker-pipeline/invoke-hooks (fn [_conn tx-report _context]
|
||||
{:tx-report tx-report})
|
||||
render-delta/build (fn [_input]
|
||||
(swap! calls conj :build-ui-refresh)
|
||||
{:rev 1})
|
||||
shared-service/broadcast-to-clients! (fn [event _payload]
|
||||
(when (= :sync-db-changes event)
|
||||
(swap! calls conj :broadcast-ui-refresh)))]
|
||||
(db-listener/listen-db-changes! "repo" conn :handler-keys [:sync-db-to-main-thread :db-sync])
|
||||
(d/transact! conn [{:db/id -1 :block/title "b1"}] {:local-tx? true}))
|
||||
(is (= [:persist-local-tx :build-ui-refresh :broadcast-ui-refresh] @calls)
|
||||
"UI refresh work must wait until the local tx has been persisted.")))
|
||||
|
||||
(deftest db-listener-builds-one-render-delta-for-origin-and-broadcast-test
|
||||
(let [repo "repo"
|
||||
conn (db-test/create-conn)
|
||||
@@ -115,9 +95,7 @@
|
||||
post-pipeline-report (atom nil)
|
||||
build-inputs (atom [])
|
||||
broadcast-payloads (atom [])]
|
||||
(with-redefs [db-sync/update-local-sync-checksum! (fn [& _] nil)
|
||||
db-sync/handle-local-tx! (fn [& _] nil)
|
||||
worker-pipeline/invoke-hooks
|
||||
(with-redefs [worker-pipeline/invoke-hooks
|
||||
(fn [_conn tx-report _context]
|
||||
(let [block-id (some (fn [datom]
|
||||
(when (and (:added datom)
|
||||
@@ -138,7 +116,7 @@
|
||||
(when (= :sync-db-changes event)
|
||||
(swap! broadcast-payloads conj payload)))]
|
||||
(db-listener/listen-db-changes! repo conn
|
||||
:handler-keys [:sync-db-to-main-thread :db-sync])
|
||||
:handler-keys [:sync-db-to-main-thread])
|
||||
(d/transact! conn
|
||||
[{:db/id -1
|
||||
:block/uuid block-uuid
|
||||
@@ -249,9 +227,7 @@
|
||||
(let [conn (db-test/create-conn)
|
||||
build-inputs (atom [])
|
||||
broadcast-payloads (atom [])]
|
||||
(with-redefs [db-sync/update-local-sync-checksum! (fn [& _] nil)
|
||||
db-sync/handle-local-tx! (fn [& _] nil)
|
||||
worker-pipeline/invoke-hooks
|
||||
(with-redefs [worker-pipeline/invoke-hooks
|
||||
(fn [_conn tx-report _context]
|
||||
{:tx-report tx-report})
|
||||
render-delta/build
|
||||
@@ -263,7 +239,7 @@
|
||||
(when (= :sync-db-changes event)
|
||||
(swap! broadcast-payloads conj payload)))]
|
||||
(db-listener/listen-db-changes! "repo" conn
|
||||
:handler-keys [:sync-db-to-main-thread :db-sync])
|
||||
:handler-keys [:sync-db-to-main-thread])
|
||||
(d/transact! conn
|
||||
[{:db/id -1 :block/title "downloaded"}]
|
||||
tx-meta))
|
||||
@@ -287,9 +263,7 @@
|
||||
(fn [tx-report]
|
||||
(swap! formal-pipeline-calls conj tx-report)
|
||||
(worker-pipeline/transact-pipeline tx-report)))
|
||||
(with-redefs [db-sync/update-local-sync-checksum! (fn [& _] nil)
|
||||
db-sync/handle-local-tx! (fn [& _] nil)
|
||||
markdown-mirror/<handle-tx-report!
|
||||
(with-redefs [markdown-mirror/<handle-tx-report!
|
||||
(fn [_repo _conn tx-report _opts]
|
||||
(swap! mirrored-reports conj tx-report))
|
||||
worker-pipeline/invoke-hooks
|
||||
@@ -305,7 +279,6 @@
|
||||
(swap! broadcast-payloads conj payload)))]
|
||||
(db-listener/listen-db-changes! "repo" conn
|
||||
:handler-keys [:sync-db-to-main-thread
|
||||
:db-sync
|
||||
:markdown-mirror])
|
||||
(ldb/transact! conn
|
||||
[[:db/add block-id :block/title "after"]]
|
||||
@@ -328,59 +301,11 @@
|
||||
(is (identical? @conn (:db-after (first @mirrored-reports)))
|
||||
"Deferred handlers must receive the raw skipped transaction report.")))
|
||||
|
||||
(deftest db-listener-reports-post-commit-failures-without-blocking-ui-sync-test
|
||||
(doseq [failed-stage [:checksum :persist]]
|
||||
(let [conn (db-test/create-conn)
|
||||
calls (atom [])
|
||||
captured-errors (atom [])]
|
||||
(with-redefs [db-sync/update-local-sync-checksum!
|
||||
(fn [& _]
|
||||
(swap! calls conj :checksum)
|
||||
(when (= :checksum failed-stage)
|
||||
(throw (js/Error. "checksum failed"))))
|
||||
db-sync/handle-local-tx!
|
||||
(fn [& _]
|
||||
(swap! calls conj :persist)
|
||||
(when (= :persist failed-stage)
|
||||
(throw (js/Error. "persist failed"))))
|
||||
worker-pipeline/invoke-hooks
|
||||
(fn [_conn tx-report _context]
|
||||
{:tx-report tx-report})
|
||||
render-delta/build
|
||||
(fn [_input]
|
||||
(swap! calls conj :build-ui-refresh)
|
||||
{:rev 1})
|
||||
shared-service/broadcast-to-clients!
|
||||
(fn [event _payload]
|
||||
(when (= :sync-db-changes event)
|
||||
(swap! calls conj :broadcast-ui-refresh)))
|
||||
platform/post-message!
|
||||
(fn [_platform event payload]
|
||||
(when (= :capture-error event)
|
||||
(swap! captured-errors conj payload)))
|
||||
platform/current (constantly :test)]
|
||||
(db-listener/listen-db-changes! "repo" conn
|
||||
:handler-keys [:sync-db-to-main-thread :db-sync])
|
||||
(let [error (try
|
||||
(d/transact! conn [{:db/id -1 :block/title "hello"}]
|
||||
{:local-tx? true})
|
||||
nil
|
||||
(catch :default error
|
||||
error))]
|
||||
(is (nil? error) (str "Post-commit failure escaped: " failed-stage))
|
||||
(is (= 1 (d/q '[:find (count ?e) .
|
||||
:where [?e :block/title "hello"]]
|
||||
@conn)))
|
||||
(is (= 1 (count @captured-errors)))
|
||||
(is (= [:checksum :persist :build-ui-refresh :broadcast-ui-refresh]
|
||||
@calls)))))))
|
||||
|
||||
(deftest deferred-listener-failures-do-not-block-ui-sync-test
|
||||
(let [conn (db-test/create-conn)
|
||||
calls (atom [])
|
||||
captured-errors (atom [])]
|
||||
(with-redefs [db-sync/update-local-sync-checksum! (fn [& _] nil)
|
||||
worker-pipeline/invoke-hooks
|
||||
(with-redefs [worker-pipeline/invoke-hooks
|
||||
(fn [_conn tx-report _context]
|
||||
{:tx-report tx-report})
|
||||
render-delta/build (constantly {:rev 1})
|
||||
@@ -416,8 +341,7 @@
|
||||
captured-errors (atom [])
|
||||
broadcasts (atom 0)]
|
||||
(-> (p/with-redefs
|
||||
[db-sync/update-local-sync-checksum! (fn [& _] nil)
|
||||
worker-pipeline/invoke-hooks
|
||||
[worker-pipeline/invoke-hooks
|
||||
(fn [_conn tx-report _context]
|
||||
{:tx-report tx-report})
|
||||
render-delta/build (constantly {:rev 1})
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -930,14 +930,9 @@
|
||||
(let [repo-error #'db-worker-node/repo-error
|
||||
bound-repo "logseq_db_bound"]
|
||||
(is (nil? (repo-error :thread-api/list-db [] bound-repo)))
|
||||
(is (nil? (repo-error :thread-api/get-db-sync-config [] bound-repo)))
|
||||
(is (nil? (repo-error :thread-api/sync-app-state [{:auth/id-token "token"}] bound-repo)))
|
||||
(is (nil? (repo-error :thread-api/db-sync-list-remote-graphs [] bound-repo)))
|
||||
(is (nil? (repo-error "thread-api/list-db" [] bound-repo)))
|
||||
(is (nil? (repo-error :thread-api/set-context [{:repo "not-a-repo-arg"}] bound-repo)))
|
||||
(is (nil? (repo-error :thread-api/resolve-ui-request ["req-id" {:password "pw"}] bound-repo)))
|
||||
(is (nil? (repo-error :thread-api/reject-ui-request ["req-id" {:code :cancelled}] bound-repo)))
|
||||
(is (nil? (repo-error :thread-api/cancel-ui-requests [{:context :logout}] bound-repo)))
|
||||
(is (= {:status 400
|
||||
:error {:code :missing-repo
|
||||
:message "repo is required"}}
|
||||
@@ -963,13 +958,8 @@
|
||||
(start-daemon! {:root-dir data-dir
|
||||
:repo repo})
|
||||
_ (reset! daemon {:host host :port port :stop! stop!})
|
||||
_ (invoke host port "thread-api/set-db-sync-config"
|
||||
[{:ws-url "wss://example.com/sync/%s"}])
|
||||
_ (invoke host port "thread-api/sync-app-state"
|
||||
[{:auth/id-token "token-value"}])
|
||||
config (invoke host port "thread-api/get-db-sync-config" [])
|
||||
_ (is (= "wss://example.com/sync/%s" (:ws-url config)))
|
||||
_ (is (not (contains? config :auth-token)))
|
||||
result (invoke host port "thread-api/set-context" [{:app "desktop"}])]
|
||||
(is (nil? result)))
|
||||
(p/catch (fn [e]
|
||||
@@ -1003,64 +993,6 @@
|
||||
(-> (stop!) (p/finally (fn [] (done))))
|
||||
(done))))))))
|
||||
|
||||
(deftest db-worker-node-sync-status-requires-repo-and-returns-structured-status
|
||||
(async done
|
||||
(let [daemon (atom nil)
|
||||
data-dir (node-helper/create-tmp-dir "db-worker-sync-status")
|
||||
repo (str "logseq_db_sync_status_" (subs (str (random-uuid)) 0 8))]
|
||||
(-> (p/let [{:keys [host port stop!]}
|
||||
(start-daemon! {:root-dir data-dir
|
||||
:repo repo})
|
||||
_ (reset! daemon {:host host :port port :stop! stop!})
|
||||
{:keys [status body]} (invoke-raw host port "thread-api/db-sync-status" [])
|
||||
parsed (js->clj (js/JSON.parse body) :keywordize-keys true)
|
||||
_ (is (= 400 status))
|
||||
_ (is (= false (:ok parsed)))
|
||||
_ (is (= "missing-repo" (get-in parsed [:error :code])))
|
||||
_ (invoke host port "thread-api/create-or-open-db" [repo {}])
|
||||
status-result (invoke host port "thread-api/db-sync-status" [repo])]
|
||||
(is (= repo (:repo status-result)))
|
||||
(is (contains? status-result :ws-state))
|
||||
(is (contains? status-result :pending-local))
|
||||
(is (contains? status-result :pending-asset))
|
||||
(is (contains? status-result :pending-server))
|
||||
(is (contains? status-result :local-tx))
|
||||
(is (contains? status-result :remote-tx))
|
||||
(is (contains? status-result :graph-id)))
|
||||
(p/catch (fn [e]
|
||||
(is false (str "unexpected error: " e))))
|
||||
(p/finally (fn []
|
||||
(if-let [stop! (:stop! @daemon)]
|
||||
(-> (stop!) (p/finally (fn [] (done))))
|
||||
(done))))))))
|
||||
|
||||
(deftest db-worker-node-sync-start-and-status-invoke-path
|
||||
(async done
|
||||
(let [daemon (atom nil)
|
||||
data-dir (node-helper/create-tmp-dir "db-worker-sync-start")
|
||||
repo (str "logseq_db_sync_start_" (subs (str (random-uuid)) 0 8))]
|
||||
(-> (p/let [{:keys [host port stop!]}
|
||||
(start-daemon! {:root-dir data-dir
|
||||
:repo repo})
|
||||
_ (reset! daemon {:host host :port port :stop! stop!})
|
||||
_ (invoke host port "thread-api/create-or-open-db" [repo {}])
|
||||
_ (invoke host port "thread-api/set-db-sync-config"
|
||||
[{:ws-url nil
|
||||
:http-base "https://example.com"}])
|
||||
start-result (invoke host port "thread-api/db-sync-start" [repo])
|
||||
status-result (invoke host port "thread-api/db-sync-status" [repo])]
|
||||
(is (nil? start-result))
|
||||
(is (= repo (:repo status-result)))
|
||||
(is (= :inactive (:ws-state status-result)))
|
||||
(is (contains? status-result :pending-local))
|
||||
(is (contains? status-result :pending-server)))
|
||||
(p/catch (fn [e]
|
||||
(is false (str "unexpected error: " e))))
|
||||
(p/finally (fn []
|
||||
(if-let [stop! (:stop! @daemon)]
|
||||
(-> (stop!) (p/finally (fn [] (done))))
|
||||
(done))))))))
|
||||
|
||||
(deftest db-worker-node-daemon-smoke-test
|
||||
(async done
|
||||
(let [daemon (atom nil)
|
||||
|
||||
@@ -1,41 +1,23 @@
|
||||
(ns frontend.worker.db-worker-test
|
||||
(:require [cljs.test :refer [async deftest is]]
|
||||
[clojure.string :as string]
|
||||
[datascript.core :as d]
|
||||
[frontend.common.thread-api :as thread-api]
|
||||
[frontend.worker.a-test-env]
|
||||
[frontend.worker.db-core :as db-worker]
|
||||
[frontend.worker.local-tx-log :as local-tx-log]
|
||||
[frontend.worker.platform :as platform]
|
||||
[frontend.worker.shared-service :as shared-service]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync :as db-sync]
|
||||
[frontend.worker.sync.client-op :as client-op]
|
||||
[frontend.worker.sync.crypt :as sync-crypt]
|
||||
[frontend.worker.sync.download :as sync-download]
|
||||
[frontend.worker.sync.log-and-state :as rtc-log-and-state]
|
||||
[logseq.db.frontend.schema :as db-schema]
|
||||
[promesa.core :as p]))
|
||||
|
||||
(def ^:private test-repo "test-db-worker-repo")
|
||||
(def ^:private close-db!-orig db-worker/close-db!)
|
||||
(def ^:private decrypt-snapshot-datoms-batch-orig sync-crypt/<decrypt-snapshot-datoms-batch)
|
||||
(def ^:private fetch-graph-aes-key-for-download-orig sync-crypt/<fetch-graph-aes-key-for-download)
|
||||
(def ^:private rehydrate-large-titles-from-db-orig db-sync/rehydrate-large-titles-from-db!)
|
||||
(def ^:private rtc-log-orig rtc-log-and-state/rtc-log)
|
||||
(def ^:private update-local-tx-orig client-op/update-local-tx)
|
||||
(def ^:private update-local-tx-orig local-tx-log/update-local-tx)
|
||||
(def ^:private broadcast-to-clients-orig shared-service/broadcast-to-clients!)
|
||||
;; Keep this compact to satisfy lint:large-vars while still checking full coverage.
|
||||
(def ^:private expected-db-core-thread-apis
|
||||
(into #{}
|
||||
(concat
|
||||
[:thread-api/list-db :thread-api/init :thread-api/set-db-sync-config :thread-api/get-db-sync-config :thread-api/get-key-value
|
||||
:thread-api/db-sync-status :thread-api/db-sync-start :thread-api/db-sync-stop :thread-api/db-sync-update-presence
|
||||
:thread-api/db-sync-request-asset-download :thread-api/db-sync-download-missing-assets
|
||||
:thread-api/db-sync-grant-graph-access :thread-api/db-sync-ensure-user-rsa-keys
|
||||
:thread-api/db-sync-list-remote-graphs :thread-api/db-sync-upload-graph :thread-api/db-sync-create-remote-graph
|
||||
:thread-api/db-sync-stop-upload :thread-api/db-sync-resume-upload :thread-api/db-sync-upload-stopped?
|
||||
:thread-api/db-sync-get-all-block-conflicts :thread-api/db-sync-clear-block-conflicts
|
||||
:thread-api/db-sync-download-graph-by-id
|
||||
[:thread-api/list-db :thread-api/init :thread-api/get-key-value
|
||||
:thread-api/create-or-open-db :thread-api/q :thread-api/datoms :thread-api/pull :thread-api/task-spent-time :thread-api/get-blocks
|
||||
:thread-api/get-block-refs :thread-api/get-block-source
|
||||
:thread-api/get-block-parents :thread-api/set-context :thread-api/transact :thread-api/undo-redo-set-pending-editor-info
|
||||
@@ -44,9 +26,7 @@
|
||||
:thread-api/reset-db :thread-api/get-file-content :thread-api/get-all-properties
|
||||
:thread-api/get-date-scheduled-or-deadlines
|
||||
:thread-api/unsafe-unlink-db :thread-api/close-db
|
||||
:thread-api/db-sync-close-db :thread-api/db-sync-invalidate-search-db :thread-api/db-sync-recreate-lock
|
||||
:thread-api/db-sync-rehydrate-large-titles :thread-api/db-sync-import-prepare :thread-api/db-sync-import-rows-chunk
|
||||
:thread-api/db-sync-import-finalize :thread-api/release-access-handles :thread-api/db-exists
|
||||
:thread-api/release-access-handles :thread-api/db-exists
|
||||
:thread-api/export-db-binary :thread-api/import-file-graph
|
||||
:thread-api/export-client-ops-db-binary :thread-api/backup-db-sqlite
|
||||
:thread-api/import-db-binary :thread-api/search-blocks :thread-api/search-upsert-blocks :thread-api/search-delete-blocks
|
||||
@@ -54,7 +34,7 @@
|
||||
:thread-api/search-build-pages-indice :thread-api/apply-outliner-ops :thread-api/sync-app-state
|
||||
:thread-api/markdown-mirror-set-enabled :thread-api/markdown-mirror-flush :thread-api/markdown-mirror-regenerate
|
||||
:thread-api/export-get-debug-datoms :thread-api/export-get-all-page->content :thread-api/validate-db
|
||||
:thread-api/recompute-checksum-diagnostics :thread-api/export-edn :thread-api/import-edn
|
||||
:thread-api/export-edn :thread-api/import-edn
|
||||
:thread-api/get-fsrs-due-card-block-ids :thread-api/get-view-data
|
||||
:thread-api/get-class-objects :thread-api/validate-block-tag
|
||||
:thread-api/convert-tag-to-page
|
||||
@@ -133,11 +113,7 @@
|
||||
platform-prev @@#'platform/*platform
|
||||
cleanup (fn []
|
||||
(set! db-worker/close-db! close-db!-orig)
|
||||
(set! sync-crypt/<decrypt-snapshot-datoms-batch decrypt-snapshot-datoms-batch-orig)
|
||||
(set! sync-crypt/<fetch-graph-aes-key-for-download fetch-graph-aes-key-for-download-orig)
|
||||
(set! db-sync/rehydrate-large-titles-from-db! rehydrate-large-titles-from-db-orig)
|
||||
(set! rtc-log-and-state/rtc-log rtc-log-orig)
|
||||
(set! client-op/update-local-tx update-local-tx-orig)
|
||||
(set! local-tx-log/update-local-tx update-local-tx-orig)
|
||||
(set! shared-service/broadcast-to-clients! broadcast-to-clients-orig)
|
||||
(reset! worker-state/*sqlite-conns sqlite-conns-prev)
|
||||
(reset! worker-state/*datascript-conns datascript-prev)
|
||||
@@ -146,11 +122,7 @@
|
||||
(reset! worker-state/*sqlite sqlite-prev)
|
||||
(reset! @#'platform/*platform platform-prev))]
|
||||
(set! db-worker/close-db! close-db!-orig)
|
||||
(set! sync-crypt/<decrypt-snapshot-datoms-batch decrypt-snapshot-datoms-batch-orig)
|
||||
(set! sync-crypt/<fetch-graph-aes-key-for-download fetch-graph-aes-key-for-download-orig)
|
||||
(set! db-sync/rehydrate-large-titles-from-db! rehydrate-large-titles-from-db-orig)
|
||||
(set! rtc-log-and-state/rtc-log rtc-log-orig)
|
||||
(set! client-op/update-local-tx update-local-tx-orig)
|
||||
(set! local-tx-log/update-local-tx update-local-tx-orig)
|
||||
(set! shared-service/broadcast-to-clients! broadcast-to-clients-orig)
|
||||
(platform/set-platform! (build-test-platform))
|
||||
(reset! worker-state/*sqlite #js {})
|
||||
@@ -176,13 +148,13 @@
|
||||
(reset! worker-state/*client-ops-conns {test-repo :client-ops})
|
||||
(reset! worker-state/*opfs-pools
|
||||
{test-repo #js {:pauseVfs (fn [] (swap! pause-calls inc))}})
|
||||
(reset! client-op/*repo->pending-local-tx-count {test-repo 9})
|
||||
(reset! local-tx-log/*repo->pending-local-tx-count {test-repo 9})
|
||||
|
||||
(db-worker/close-db! test-repo)
|
||||
|
||||
(is (= #{:db :search :client-ops} (set @closed)))
|
||||
(is (= 1 @pause-calls))
|
||||
(is (nil? (get @client-op/*repo->pending-local-tx-count test-repo)))
|
||||
(is (nil? (get @local-tx-log/*repo->pending-local-tx-count test-repo)))
|
||||
(is (nil? (get @worker-state/*sqlite-conns test-repo)))))))
|
||||
|
||||
(deftest close-db-checkpoints-wal-before-closing-test
|
||||
@@ -248,328 +220,6 @@
|
||||
(set! js/setInterval original-set-interval)
|
||||
(set! js/clearInterval original-clear-interval)))))))
|
||||
|
||||
(deftest complete-datoms-import-invalidates-existing-search-db-test
|
||||
(async done
|
||||
(restoring-worker-state
|
||||
(fn []
|
||||
(let [thread-apis-prev @thread-api/*thread-apis]
|
||||
(vreset! thread-api/*thread-apis
|
||||
(assoc thread-apis-prev
|
||||
:thread-api/create-or-open-db (fn [_repo _opts] (p/resolved nil))
|
||||
:thread-api/db-sync-rehydrate-large-titles (fn [_repo _graph-id] (p/resolved nil))))
|
||||
(-> (p/with-redefs [rtc-log-and-state/rtc-log (fn [& _] nil)
|
||||
client-op/update-graph-uuid (fn [& _] nil)
|
||||
client-op/update-local-tx (fn [& _] nil)
|
||||
shared-service/broadcast-to-clients! (fn [& _] nil)]
|
||||
(sync-download/complete-datoms-import! test-repo "graph-1" 42))
|
||||
(p/then (fn [_]
|
||||
(is true)
|
||||
(vreset! thread-api/*thread-apis thread-apis-prev)
|
||||
(done)))
|
||||
(p/catch (fn [error]
|
||||
(vreset! thread-api/*thread-apis thread-apis-prev)
|
||||
(is false (str error))
|
||||
(done)))))))))
|
||||
|
||||
(defn- capture-outcome
|
||||
[thunk]
|
||||
(try
|
||||
(-> (thunk)
|
||||
(p/then (fn [value] {:value value}))
|
||||
(p/catch (fn [error] {:error error})))
|
||||
(catch :default error
|
||||
(p/resolved {:error error}))))
|
||||
|
||||
(defn- with-fake-create-or-open-db
|
||||
([repo conn f]
|
||||
(with-fake-create-or-open-db repo conn {} f))
|
||||
([repo conn
|
||||
{:keys [create-or-open-db-f close-db-f invalidate-search-db-f unlink-db-f recreate-lock-f]}
|
||||
f]
|
||||
(let [thread-apis-prev @thread-api/*thread-apis
|
||||
create-or-open-db-f (or create-or-open-db-f
|
||||
(fn [_repo _opts]
|
||||
(swap! worker-state/*datascript-conns assoc repo conn)
|
||||
(p/resolved nil)))
|
||||
close-db-f (or close-db-f (fn [_repo] nil))
|
||||
invalidate-search-db-f (or invalidate-search-db-f (fn [_repo] (p/resolved nil)))
|
||||
unlink-db-f (or unlink-db-f (fn [_repo] nil))
|
||||
recreate-lock-f (or recreate-lock-f (fn [_repo] (p/resolved nil)))]
|
||||
(vreset! thread-api/*thread-apis
|
||||
(assoc thread-apis-prev
|
||||
:thread-api/create-or-open-db create-or-open-db-f
|
||||
:thread-api/db-sync-close-db close-db-f
|
||||
:thread-api/db-sync-invalidate-search-db invalidate-search-db-f
|
||||
:thread-api/unsafe-unlink-db unlink-db-f
|
||||
:thread-api/db-sync-recreate-lock recreate-lock-f))
|
||||
(-> (f)
|
||||
(p/finally (fn []
|
||||
(vreset! thread-api/*thread-apis thread-apis-prev)))))))
|
||||
|
||||
(deftest db-sync-import-prepare-replaces-active-import-state-test
|
||||
(async done
|
||||
(restoring-worker-state
|
||||
(fn []
|
||||
(let [prepare (@thread-api/*thread-apis :thread-api/db-sync-import-prepare)
|
||||
conn-a (d/create-conn db-schema/schema)
|
||||
conn-b (d/create-conn db-schema/schema)]
|
||||
(with-fake-create-or-open-db
|
||||
test-repo conn-a
|
||||
(fn []
|
||||
(-> (p/with-redefs [db-worker/close-db! (fn [_] nil)]
|
||||
(p/let [first-import (prepare test-repo true "graph-1" false)
|
||||
_ (swap! worker-state/*datascript-conns assoc test-repo conn-b)
|
||||
second-import (prepare test-repo true "graph-1" false)]
|
||||
(is (map? first-import))
|
||||
(is (map? second-import))
|
||||
(is (not= (:import-id first-import) (:import-id second-import)))))
|
||||
(p/then (fn [_] (done)))
|
||||
(p/catch (fn [error]
|
||||
(is false (str error))
|
||||
(done)))))))))))
|
||||
|
||||
(deftest db-sync-import-prepare-reset-unlinks-db-before-reopen-test
|
||||
(async done
|
||||
(restoring-worker-state
|
||||
(fn []
|
||||
(let [prepare (@thread-api/*thread-apis :thread-api/db-sync-import-prepare)
|
||||
conn (d/create-conn db-schema/schema)
|
||||
calls (atom [])]
|
||||
(with-fake-create-or-open-db
|
||||
test-repo conn
|
||||
{:close-db-f (fn [repo]
|
||||
(swap! calls conj [:close repo])
|
||||
nil)
|
||||
:unlink-db-f (fn [repo]
|
||||
(swap! calls conj [:unlink repo])
|
||||
nil)
|
||||
:recreate-lock-f (fn [repo]
|
||||
(swap! calls conj [:recreate-lock repo])
|
||||
(p/resolved nil))
|
||||
:invalidate-search-db-f (fn [repo]
|
||||
(swap! calls conj [:invalidate-search repo])
|
||||
(p/resolved nil))
|
||||
:create-or-open-db-f (fn [repo opts]
|
||||
(swap! calls conj [:create-or-open repo opts])
|
||||
(swap! worker-state/*datascript-conns assoc repo conn)
|
||||
(p/resolved nil))}
|
||||
(fn []
|
||||
(-> (prepare test-repo true "graph-1" false)
|
||||
(p/then (fn [_]
|
||||
(let [ops (mapv first @calls)
|
||||
idx (fn [op]
|
||||
(first (keep-indexed (fn [i v]
|
||||
(when (= op v) i))
|
||||
ops)))]
|
||||
(is (some? (idx :close)))
|
||||
(is (some? (idx :unlink)))
|
||||
(is (some? (idx :recreate-lock)))
|
||||
(is (some? (idx :invalidate-search)))
|
||||
(is (some? (idx :create-or-open)))
|
||||
(is (< (idx :close) (idx :unlink)))
|
||||
(is (< (idx :unlink) (idx :recreate-lock)))
|
||||
(is (< (idx :recreate-lock) (idx :invalidate-search)))
|
||||
(is (< (idx :invalidate-search) (idx :create-or-open))))
|
||||
(done)))
|
||||
(p/catch (fn [error]
|
||||
(is false (str error))
|
||||
(done)))))))))))
|
||||
|
||||
(deftest db-sync-import-finalize-rejects-stale-import-id-test
|
||||
(async done
|
||||
(restoring-worker-state
|
||||
(fn []
|
||||
(let [prepare (@thread-api/*thread-apis :thread-api/db-sync-import-prepare)
|
||||
finalize (@thread-api/*thread-apis :thread-api/db-sync-import-finalize)
|
||||
conn (d/create-conn db-schema/schema)]
|
||||
(with-fake-create-or-open-db
|
||||
test-repo conn
|
||||
(fn []
|
||||
(-> (p/with-redefs [rtc-log-and-state/rtc-log (fn [& _] nil)
|
||||
client-op/update-local-tx (fn [& _] nil)
|
||||
shared-service/broadcast-to-clients! (fn [& _] nil)]
|
||||
(p/let [first-import (prepare test-repo true "graph-1" false)
|
||||
second-import (prepare test-repo true "graph-1" false)
|
||||
stale-outcome (capture-outcome #(finalize test-repo "graph-1" 42 (:import-id first-import)))]
|
||||
(is (= :db-sync/stale-import (some-> stale-outcome :error ex-data :type)))
|
||||
(-> (finalize test-repo "graph-1" 42 (:import-id second-import))
|
||||
(p/then (fn [_]
|
||||
(is true)
|
||||
(done))))))
|
||||
(p/catch (fn [error]
|
||||
(is false (str error))
|
||||
(done)))))))))))
|
||||
|
||||
(deftest db-sync-import-finalize-cleans-temp-pool-on-success-test
|
||||
(async done
|
||||
(restoring-worker-state
|
||||
(fn []
|
||||
(let [import-id "import-success-1"
|
||||
graph-id "graph-success-1"
|
||||
remote-tx 42
|
||||
rows-db-closed* (atom 0)
|
||||
removed-pools* (atom [])
|
||||
rows-pool #js {:name "temp-download-pool"}
|
||||
rows-db #js {:close (fn []
|
||||
(swap! rows-db-closed* inc))}]
|
||||
(reset! @#'sync-download/*import-state
|
||||
{:import-id import-id
|
||||
:repo test-repo
|
||||
:graph-id graph-id
|
||||
:rows-db rows-db
|
||||
:rows-pool rows-pool
|
||||
:rows-path "/download-import.sqlite"
|
||||
:rows-imported? false})
|
||||
(-> (p/with-redefs [sync-download/complete-datoms-import! (fn [_repo _graph-id _remote-tx]
|
||||
(p/resolved :ok))
|
||||
platform/remove-storage-pool! (fn [_platform pool]
|
||||
(swap! removed-pools* conj pool)
|
||||
(p/resolved true))]
|
||||
(sync-download/finalize-import! test-repo graph-id remote-tx import-id))
|
||||
(p/then (fn [result]
|
||||
(is (= :ok result))
|
||||
(is (= 1 @rows-db-closed*))
|
||||
(is (= [rows-pool] @removed-pools*))
|
||||
(is (nil? @@#'sync-download/*import-state))
|
||||
(done)))
|
||||
(p/catch (fn [error]
|
||||
(is false (str error))
|
||||
(done)))))))))
|
||||
|
||||
(deftest db-sync-download-graph-by-id-cleans-temp-pool-on-failure-test
|
||||
(async done
|
||||
(restoring-worker-state
|
||||
(fn []
|
||||
(let [original-fetch js/fetch
|
||||
import-id "import-failure-1"
|
||||
graph-id "graph-failure-1"
|
||||
rows-db-closed* (atom 0)
|
||||
removed-pools* (atom [])
|
||||
finalize-calls* (atom 0)
|
||||
rows-pool #js {:name "temp-download-pool"}
|
||||
rows-db #js {:close (fn []
|
||||
(swap! rows-db-closed* inc))}]
|
||||
(reset! worker-state/*db-sync-config {:http-base "https://sync.example.test"})
|
||||
(set! js/fetch (fn [_url _opts]
|
||||
(p/resolved #js {:ok true})))
|
||||
(-> (p/with-redefs [rtc-log-and-state/rtc-log (fn [& _] nil)
|
||||
sync-download/fetch-json (fn [_url _opts schema]
|
||||
(case schema
|
||||
:sync/pull (p/resolved {:t 77})
|
||||
:sync/snapshot-download (p/resolved {:url "https://snapshot.example.test"})
|
||||
(p/rejected (ex-info "unexpected schema" {:schema schema}))))
|
||||
sync-download/prepare-import! (fn [repo _reset? gid _graph-e2ee? & _]
|
||||
(reset! @#'sync-download/*import-state
|
||||
{:import-id import-id
|
||||
:repo repo
|
||||
:graph-id gid
|
||||
:rows-db rows-db
|
||||
:rows-pool rows-pool
|
||||
:rows-path "/download-import.sqlite"
|
||||
:rows-imported? false})
|
||||
(p/resolved {:import-id import-id}))
|
||||
sync-download/import-rows-chunk! (fn [_rows _graph-id _import-id]
|
||||
(p/resolved true))
|
||||
sync-download/finalize-import! (fn [& _]
|
||||
(swap! finalize-calls* inc)
|
||||
(p/resolved nil))
|
||||
sync-download/<stream-snapshot-row-batches! (fn [_resp _batch-size on-batch]
|
||||
(p/let [_ (on-batch [[1 "content" nil]])]
|
||||
(p/rejected (ex-info "stream failed" {:code :stream-failed}))))
|
||||
platform/remove-storage-pool! (fn [_platform pool]
|
||||
(swap! removed-pools* conj pool)
|
||||
(p/resolved true))]
|
||||
(sync-download/download-graph-by-id! test-repo graph-id false))
|
||||
(p/then (fn [_]
|
||||
(is false "expected download failure")
|
||||
(done)))
|
||||
(p/catch (fn [error]
|
||||
(is (string/includes? (or (ex-message error) "") "db-sync download failed"))
|
||||
(is (= 1 @rows-db-closed*))
|
||||
(is (= [rows-pool] @removed-pools*))
|
||||
(is (= 0 @finalize-calls*))
|
||||
(is (nil? @@#'sync-download/*import-state))
|
||||
(done)))
|
||||
(p/finally (fn []
|
||||
(set! js/fetch original-fetch)))))))))
|
||||
|
||||
(deftest snapshot-datoms-in-import-order-puts-schema-before-data-test
|
||||
(let [conn (d/create-conn db-schema/schema)]
|
||||
(d/transact! conn [{:db/ident :logseq.kv/schema-version
|
||||
:kv/value {:major 65 :minor 0}}
|
||||
{:db/ident :user.test/attr
|
||||
:db/valueType :db.type/string
|
||||
:db/cardinality :db.cardinality/one}
|
||||
{:db/id 100
|
||||
:user.test/attr "hello"}])
|
||||
(let [ordered (vec (#'sync-download/snapshot-datoms-in-import-order conn))
|
||||
data-idx (first (keep-indexed (fn [idx datom]
|
||||
(when (and (= 100 (:e datom))
|
||||
(= :user.test/attr (:a datom)))
|
||||
idx))
|
||||
ordered))
|
||||
attr-eid (:db/id (d/entity @conn :user.test/attr))
|
||||
ident-idx (first (keep-indexed (fn [idx datom]
|
||||
(when (and (= attr-eid (:e datom))
|
||||
(= :db/ident (:a datom)))
|
||||
idx))
|
||||
ordered))
|
||||
cardinality-idx (first (keep-indexed (fn [idx datom]
|
||||
(when (and (= attr-eid (:e datom))
|
||||
(= :db/cardinality (:a datom)))
|
||||
idx))
|
||||
ordered))
|
||||
schema-version-eid (:db/id (d/entity @conn :logseq.kv/schema-version))
|
||||
schema-version-idx (first (keep-indexed (fn [idx datom]
|
||||
(when (and (= schema-version-eid (:e datom))
|
||||
(= :db/ident (:a datom)))
|
||||
idx))
|
||||
ordered))]
|
||||
(is (number? data-idx))
|
||||
(is (number? ident-idx))
|
||||
(is (number? cardinality-idx))
|
||||
(is (number? schema-version-idx))
|
||||
(is (< schema-version-idx data-idx))
|
||||
(is (< ident-idx data-idx))
|
||||
(is (< cardinality-idx data-idx)))))
|
||||
|
||||
(deftest import-datoms-batch-transacts-all-db-schema-before-data-test
|
||||
(async done
|
||||
(let [conn (d/create-conn db-schema/schema)
|
||||
attr-eid 8001
|
||||
datoms [{:e attr-eid :a :db/ident :v :user.test/indexed}
|
||||
{:e 100 :a :user.test/indexed :v "hello"}
|
||||
{:e attr-eid :a :db/valueType :v :db.type/string}
|
||||
{:e attr-eid :a :db/cardinality :v :db.cardinality/one}
|
||||
{:e attr-eid :a :db/index :v true}]]
|
||||
(-> (#'sync-download/import-datoms-batch! conn nil false datoms)
|
||||
(p/then (fn [_]
|
||||
(is (= 1 (count (d/datoms @conn :avet :user.test/indexed "hello"))))
|
||||
(done)))
|
||||
(p/catch (fn [error]
|
||||
(is false (str error))
|
||||
(done)))))))
|
||||
|
||||
(deftest imported-snapshot-blocks-receive-local-revisions-test
|
||||
(async done
|
||||
(let [conn (d/create-conn db-schema/schema)
|
||||
block-uuid (random-uuid)
|
||||
datoms [{:e 100 :a :block/uuid :v block-uuid}
|
||||
{:e 100 :a :block/title :v "downloaded"}
|
||||
{:e 100 :a :block/tx-id :v 42}]]
|
||||
(-> (#'sync-download/import-datoms-batch! conn nil false datoms)
|
||||
(p/then (fn [_]
|
||||
(let [block (d/entity @conn [:block/uuid block-uuid])]
|
||||
(is (= "downloaded" (:block/title block)))
|
||||
(is (not= 42 (:block/tx-id block))
|
||||
"A remote snapshot cannot supply a renderer revision.")
|
||||
(is (= (:max-tx @conn) (:block/tx-id block))
|
||||
"Snapshot import stamps the local DataScript transaction."))
|
||||
(done)))
|
||||
(p/catch (fn [error]
|
||||
(is false (str error))
|
||||
(done)))))))
|
||||
|
||||
(deftest thread-api-export-client-ops-db-binary-checkpoints-and-exports-client-ops-file-test
|
||||
(async done
|
||||
(restoring-worker-state
|
||||
@@ -685,174 +335,13 @@
|
||||
(is (empty? missing) (str "Missing thread apis: " missing))
|
||||
(is (empty? non-functions) (str "Non-function thread apis: " non-functions))))
|
||||
|
||||
(defn- with-db-sync-core-wrapper-redefs
|
||||
[calls f]
|
||||
(let [results {:status {:state :connected}
|
||||
:ensure-keys {:ok true}
|
||||
:list-graphs [{:graph-id "g1"}]
|
||||
:upload-stopped? true
|
||||
:download-missing-assets {:total 2 :downloaded 2 :skipped-existing 0}}]
|
||||
(with-redefs [db-worker/db-sync-dbs-open? (fn [repo]
|
||||
(swap! calls conj [:db-sync-dbs-open? repo])
|
||||
true)
|
||||
db-sync/status (fn [repo]
|
||||
(swap! calls conj [:status repo])
|
||||
(:status results))
|
||||
db-sync/start! (fn [repo]
|
||||
(swap! calls conj [:start repo])
|
||||
:started)
|
||||
db-sync/stop! (fn []
|
||||
(swap! calls conj [:stop])
|
||||
:stopped)
|
||||
db-sync/update-presence! (fn [editing-block-uuid]
|
||||
(swap! calls conj [:update-presence editing-block-uuid])
|
||||
:presence-updated)
|
||||
db-sync/request-asset-download! (fn [repo asset-uuid]
|
||||
(swap! calls conj [:request-asset-download repo asset-uuid])
|
||||
:asset-requested)
|
||||
db-sync/download-missing-assets! (fn [repo graph-id]
|
||||
(swap! calls conj [:download-missing-assets repo graph-id])
|
||||
(:download-missing-assets results))
|
||||
sync-crypt/<grant-graph-access! (fn [repo graph-id target-email]
|
||||
(swap! calls conj [:grant-graph-access repo graph-id target-email])
|
||||
:granted)
|
||||
sync-crypt/ensure-user-rsa-keys! (fn [opts]
|
||||
(swap! calls conj [:ensure-user-rsa-keys opts])
|
||||
(:ensure-keys results))
|
||||
db-sync/list-remote-graphs! (fn []
|
||||
(swap! calls conj [:list-remote-graphs])
|
||||
(:list-graphs results))
|
||||
db-sync/upload-graph! (fn [repo]
|
||||
(swap! calls conj [:upload-graph repo])
|
||||
:uploading)
|
||||
db-sync/create-remote-graph! (fn [repo opts]
|
||||
(swap! calls conj [:create-remote-graph repo opts])
|
||||
:created)
|
||||
db-sync/stop-upload! (fn [repo]
|
||||
(swap! calls conj [:stop-upload repo])
|
||||
:stopped-upload)
|
||||
db-sync/resume-upload! (fn [repo]
|
||||
(swap! calls conj [:resume-upload repo])
|
||||
:resumed-upload)
|
||||
db-sync/upload-stopped? (fn [repo]
|
||||
(swap! calls conj [:upload-stopped? repo])
|
||||
(:upload-stopped? results))
|
||||
client-op/clear-sync-conflicts! (fn [repo block-uuid]
|
||||
(swap! calls conj [:clear-sync-conflicts repo block-uuid])
|
||||
nil)
|
||||
shared-service/broadcast-to-clients! (fn [event payload]
|
||||
(swap! calls conj [:broadcast event payload])
|
||||
nil)]
|
||||
(f results))))
|
||||
|
||||
(defn- assert-db-sync-core-wrapper-results!
|
||||
[calls results request]
|
||||
(let [{:keys [repo graph-id block asset email opts]} request]
|
||||
(is (= (:status results) ((get-thread-api :thread-api/db-sync-status) repo)))
|
||||
(is (= :started ((get-thread-api :thread-api/db-sync-start) repo)))
|
||||
(is (= :stopped ((get-thread-api :thread-api/db-sync-stop))))
|
||||
(is (= :presence-updated ((get-thread-api :thread-api/db-sync-update-presence) block)))
|
||||
(is (= :asset-requested ((get-thread-api :thread-api/db-sync-request-asset-download) repo asset)))
|
||||
(is (= (:download-missing-assets results)
|
||||
((get-thread-api :thread-api/db-sync-download-missing-assets) repo graph-id)))
|
||||
(is (= :granted ((get-thread-api :thread-api/db-sync-grant-graph-access) repo graph-id email)))
|
||||
(is (= (:ensure-keys results) ((get-thread-api :thread-api/db-sync-ensure-user-rsa-keys) opts)))
|
||||
(is (= (:ensure-keys results) ((get-thread-api :thread-api/db-sync-ensure-user-rsa-keys))))
|
||||
(is (= (:list-graphs results) ((get-thread-api :thread-api/db-sync-list-remote-graphs))))
|
||||
(is (= :uploading ((get-thread-api :thread-api/db-sync-upload-graph) repo)))
|
||||
(is (= :created ((get-thread-api :thread-api/db-sync-create-remote-graph) repo true false)))
|
||||
(is (= :stopped-upload ((get-thread-api :thread-api/db-sync-stop-upload) repo)))
|
||||
(is (= :resumed-upload ((get-thread-api :thread-api/db-sync-resume-upload) repo)))
|
||||
(is (= (:upload-stopped? results) ((get-thread-api :thread-api/db-sync-upload-stopped?) repo)))
|
||||
((get-thread-api :thread-api/db-sync-clear-block-conflicts) repo block)
|
||||
(is (some #(= [:broadcast
|
||||
:sync-conflicts-updated
|
||||
{:repo repo
|
||||
:block-uuid block
|
||||
:conflicts []}]
|
||||
%)
|
||||
@calls))
|
||||
(is (some #(= [:ensure-user-rsa-keys nil] %) @calls))
|
||||
(is (some #(= [:create-remote-graph repo
|
||||
{:graph-e2ee? true
|
||||
:graph-ready-for-use? false}]
|
||||
%)
|
||||
@calls))))
|
||||
|
||||
(deftest thread-api-db-sync-wrappers-delegate-core-ops-test
|
||||
(restoring-worker-state
|
||||
(fn []
|
||||
(let [calls (atom [])
|
||||
request {:repo "graph-a"
|
||||
:graph-id "remote-graph-id"
|
||||
:block "block-1"
|
||||
:asset "asset-1"
|
||||
:email "user@example.com"
|
||||
:opts {:force? true}}]
|
||||
(with-db-sync-core-wrapper-redefs
|
||||
calls
|
||||
#(assert-db-sync-core-wrapper-results! calls % request))))))
|
||||
|
||||
(deftest thread-api-db-sync-wrappers-delegate-import-ops-test
|
||||
(restoring-worker-state
|
||||
(fn []
|
||||
(let [calls (atom [])
|
||||
request-repo "graph-a"
|
||||
request-graph-id "remote-graph-id"
|
||||
download-result {:downloaded true}
|
||||
import-prepare-result {:import-id "import-1"}
|
||||
rows-result {:rows-imported 2}
|
||||
finalize-result {:ok true}
|
||||
rehydrate-result {:rehydrated 3}]
|
||||
(with-redefs [sync-download/download-graph-by-id! (fn [repo graph-id graph-e2ee?]
|
||||
(swap! calls conj [:download-graph-by-id repo graph-id graph-e2ee?])
|
||||
download-result)
|
||||
db-sync/rehydrate-large-titles-from-db! (fn [repo graph-id]
|
||||
(swap! calls conj [:rehydrate-large-titles repo graph-id])
|
||||
rehydrate-result)
|
||||
sync-download/prepare-import! (fn [repo reset? graph-id graph-e2ee? total-datoms]
|
||||
(swap! calls conj [:prepare-import repo reset? graph-id graph-e2ee? total-datoms])
|
||||
import-prepare-result)
|
||||
sync-download/import-rows-chunk! (fn [rows graph-id import-id]
|
||||
(swap! calls conj [:import-rows-chunk rows graph-id import-id])
|
||||
rows-result)
|
||||
sync-download/finalize-import! (fn [repo graph-id remote-tx import-id]
|
||||
(swap! calls conj [:finalize-import repo graph-id remote-tx import-id])
|
||||
finalize-result)]
|
||||
(is (= download-result ((get-thread-api :thread-api/db-sync-download-graph-by-id) request-repo request-graph-id true)))
|
||||
(is (= rehydrate-result ((get-thread-api :thread-api/db-sync-rehydrate-large-titles) request-repo request-graph-id)))
|
||||
(is (= import-prepare-result ((get-thread-api :thread-api/db-sync-import-prepare) request-repo true request-graph-id true 99)))
|
||||
(is (= rows-result ((get-thread-api :thread-api/db-sync-import-rows-chunk) [[1 "row" nil]] request-graph-id "import-1")))
|
||||
(is (= finalize-result ((get-thread-api :thread-api/db-sync-import-finalize) request-repo request-graph-id 77 "import-1")))
|
||||
(is (some #(= [:prepare-import request-repo true request-graph-id true 99] %) @calls)))))))
|
||||
|
||||
(deftest thread-api-set-and-get-db-sync-config-uses-sanitized-config-test
|
||||
(restoring-worker-state
|
||||
(fn []
|
||||
(let [sanitize-calls (atom [])
|
||||
input-config {:http-base "https://sync.example.test"
|
||||
:token "secret"}
|
||||
sanitized-config {:http-base "https://sync.example.test"
|
||||
:sanitized? true}]
|
||||
(with-redefs [worker-state/non-auth-db-sync-config (fn [config]
|
||||
(swap! sanitize-calls conj config)
|
||||
sanitized-config)]
|
||||
(is (nil? ((get-thread-api :thread-api/set-db-sync-config) input-config)))
|
||||
(is (= sanitized-config @worker-state/*db-sync-config))
|
||||
(is (= sanitized-config ((get-thread-api :thread-api/get-db-sync-config))))
|
||||
(is (= [input-config sanitized-config] @sanitize-calls)))))))
|
||||
|
||||
(deftest thread-api-release-access-handles-closes-import-state-and-pauses-vfs-test
|
||||
(deftest thread-api-release-access-handles-pauses-vfs-test
|
||||
(restoring-worker-state
|
||||
(fn []
|
||||
(let [repo test-repo
|
||||
pause-calls (atom [])
|
||||
close-import-calls (atom [])
|
||||
pool #js {:pauseVfs (fn []
|
||||
(swap! pause-calls conj :paused))}]
|
||||
(reset! worker-state/*opfs-pools {repo pool})
|
||||
(with-redefs [sync-download/close-import-state-for-repo! (fn [graph]
|
||||
(swap! close-import-calls conj graph))]
|
||||
(is (nil? ((get-thread-api :thread-api/release-access-handles) repo)))
|
||||
(is (= [repo] @close-import-calls))
|
||||
(is (= [:paused] @pause-calls)))))))
|
||||
(is (nil? ((get-thread-api :thread-api/release-access-handles) repo)))
|
||||
(is (= [:paused] @pause-calls))))))
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
(ns frontend.worker.sync.client-op-test
|
||||
(ns frontend.worker.local-tx-log-test
|
||||
(:require [cljs.test :refer [deftest is testing]]
|
||||
[frontend.worker.local-tx-log :as local-tx-log]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync.client-op :as client-op]
|
||||
[logseq.db.common.sqlite :as common-sqlite]
|
||||
[logseq.db.sqlite.gc :as sqlite-gc]))
|
||||
|
||||
@@ -32,46 +32,14 @@
|
||||
(aget row "count"))
|
||||
0)))
|
||||
|
||||
(deftest sqlite-sync-meta-roundtrip-test
|
||||
(deftest sqlite-local-tx-roundtrip-test
|
||||
(let [repo "repo-1"]
|
||||
(with-client-ops-db
|
||||
repo
|
||||
(fn [_db]
|
||||
(client-op/update-graph-uuid repo "graph-1")
|
||||
(client-op/update-local-tx repo 9)
|
||||
(client-op/update-local-checksum repo "checksum-1")
|
||||
|
||||
(client-op/update-graph-uuid repo "graph-2")
|
||||
(client-op/update-local-tx repo 12)
|
||||
(client-op/update-local-checksum repo "checksum-2")
|
||||
|
||||
(is (= "graph-2" (client-op/get-graph-uuid repo)))
|
||||
(is (= 12 (client-op/get-local-tx repo)))
|
||||
(is (= "checksum-2" (client-op/get-local-checksum repo)))))))
|
||||
|
||||
(deftest sqlite-asset-ops-coalescing-test
|
||||
(let [repo "repo-asset"
|
||||
asset-uuid (random-uuid)]
|
||||
(with-client-ops-db
|
||||
repo
|
||||
(fn [_db]
|
||||
(client-op/add-asset-ops repo [[:update-asset 10 {:block-uuid asset-uuid}]])
|
||||
(is (= 1 (client-op/get-unpushed-asset-ops-count repo)))
|
||||
(is (= [:update-asset 10 {:block-uuid asset-uuid}]
|
||||
(:update-asset (first (client-op/get-all-asset-ops repo)))))
|
||||
|
||||
;; older remove should be ignored because a newer update already exists
|
||||
(client-op/add-asset-ops repo [[:remove-asset 9 {:block-uuid asset-uuid}]])
|
||||
(is (= [:update-asset 10 {:block-uuid asset-uuid}]
|
||||
(:update-asset (first (client-op/get-all-asset-ops repo)))))
|
||||
|
||||
;; newer remove should replace update
|
||||
(client-op/add-asset-ops repo [[:remove-asset 11 {:block-uuid asset-uuid}]])
|
||||
(is (= [:remove-asset 11 {:block-uuid asset-uuid}]
|
||||
(:remove-asset (first (client-op/get-all-asset-ops repo)))))
|
||||
|
||||
(client-op/remove-asset-op repo asset-uuid)
|
||||
(is (= 0 (client-op/get-unpushed-asset-ops-count repo)))))))
|
||||
(local-tx-log/update-local-tx repo 9)
|
||||
(local-tx-log/update-local-tx repo 12)
|
||||
(is (= 12 (local-tx-log/get-local-tx repo)))))))
|
||||
|
||||
(deftest cleanup-finished-history-ops-removes-only-unreferenced-finished-txs-test
|
||||
(let [repo "repo-cleanup"
|
||||
@@ -81,8 +49,8 @@
|
||||
(with-client-ops-db
|
||||
repo
|
||||
(fn [db]
|
||||
(client-op/update-local-tx repo 99)
|
||||
(client-op/upsert-local-tx-entry!
|
||||
(local-tx-log/update-local-tx repo 99)
|
||||
(local-tx-log/upsert-local-tx-entry!
|
||||
repo
|
||||
{:tx-id keep-tx-id
|
||||
:created-at 1
|
||||
@@ -90,7 +58,7 @@
|
||||
:failed? false
|
||||
:normalized-tx-data []
|
||||
:reversed-tx-data []})
|
||||
(client-op/upsert-local-tx-entry!
|
||||
(local-tx-log/upsert-local-tx-entry!
|
||||
repo
|
||||
{:tx-id remove-tx-id
|
||||
:created-at 2
|
||||
@@ -98,7 +66,7 @@
|
||||
:failed? false
|
||||
:normalized-tx-data []
|
||||
:reversed-tx-data []})
|
||||
(client-op/upsert-local-tx-entry!
|
||||
(local-tx-log/upsert-local-tx-entry!
|
||||
repo
|
||||
{:tx-id pending-tx-id
|
||||
:created-at 3
|
||||
@@ -107,11 +75,11 @@
|
||||
:normalized-tx-data []
|
||||
:reversed-tx-data []})
|
||||
|
||||
(is (= 1 (client-op/cleanup-finished-history-ops! repo #{keep-tx-id})))
|
||||
(is (= 1 (local-tx-log/cleanup-finished-history-ops! repo #{keep-tx-id})))
|
||||
(is (= 1 (sqlite-count db "select count(*) as c from client_ops where tx_id = ?" (str keep-tx-id))))
|
||||
(is (= 0 (sqlite-count db "select count(*) as c from client_ops where tx_id = ?" (str remove-tx-id))))
|
||||
(is (= 1 (sqlite-count db "select count(*) as c from client_ops where tx_id = ?" (str pending-tx-id))))
|
||||
(is (= 99 (client-op/get-local-tx repo)))))))
|
||||
(is (= 99 (local-tx-log/get-local-tx repo)))))))
|
||||
|
||||
(deftest cleanup-finished-history-ops-no-conn-is-noop-test
|
||||
(let [repo "repo-no-conn"
|
||||
@@ -119,11 +87,11 @@
|
||||
(reset! worker-state/*client-ops-conns {})
|
||||
(try
|
||||
(testing "cleanup should be safe when client-ops conn is missing"
|
||||
(is (= 0 (client-op/cleanup-finished-history-ops! repo #{}))))
|
||||
(is (= 0 (local-tx-log/cleanup-finished-history-ops! repo #{}))))
|
||||
(finally
|
||||
(reset! worker-state/*client-ops-conns prev-client-ops-conns)))))
|
||||
|
||||
(deftest gc-on-client-ops-db-preserves-sync-metadata-test
|
||||
(deftest gc-on-client-ops-db-preserves-local-tx-test
|
||||
"Reproduces the bug where GC on client-ops db could reset local-tx to 0.
|
||||
The client-ops db has a kvs table (created by create-kvs-table!) but it's
|
||||
empty because client-ops doesn't store Datascript data. Running GC on an
|
||||
@@ -135,9 +103,9 @@
|
||||
(fn [db]
|
||||
;; client-ops db gets a kvs table during db open, but it's empty
|
||||
(common-sqlite/create-kvs-table! db)
|
||||
(client-op/ensure-sqlite-schema! db)
|
||||
(client-op/update-local-tx repo 42)
|
||||
(is (= 42 (client-op/get-local-tx repo))
|
||||
(local-tx-log/ensure-sqlite-schema! db)
|
||||
(local-tx-log/update-local-tx repo 42)
|
||||
(is (= 42 (local-tx-log/get-local-tx repo))
|
||||
"local-tx should be 42 before GC")
|
||||
|
||||
;; GC on client-ops db: kvs table is empty because client-ops db
|
||||
@@ -153,10 +121,10 @@
|
||||
(sqlite-gc/gc-kvs-table! db {:full-gc? false}))))
|
||||
|
||||
;; After GC, local-tx should still be intact
|
||||
(is (= 42 (client-op/get-local-tx repo))
|
||||
(is (= 42 (local-tx-log/get-local-tx repo))
|
||||
"local-tx should still be 42 after GC")
|
||||
|
||||
;; VACUUM should also be safe
|
||||
(.exec db "VACUUM")
|
||||
(is (= 42 (client-op/get-local-tx repo))
|
||||
(is (= 42 (local-tx-log/get-local-tx repo))
|
||||
"local-tx should still be 42 after VACUUM")))))
|
||||
@@ -1,69 +0,0 @@
|
||||
(ns frontend.worker.node-sync-test
|
||||
(:require
|
||||
[cljs.test :refer [async deftest is]]
|
||||
[frontend.worker.platform :as platform]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync :as db-sync]
|
||||
[promesa.core :as p]))
|
||||
|
||||
(def ^:private test-repo "test-db-sync-repo")
|
||||
|
||||
(deftest resolve-ws-token-cli-owner-source-reads-state-token-test
|
||||
(async done
|
||||
(let [refresh-calls (atom 0)
|
||||
config-prev @worker-state/*db-sync-config
|
||||
state-prev @worker-state/*state
|
||||
main-thread-prev @worker-state/*main-thread]
|
||||
(reset! worker-state/*db-sync-config {:ws-url "wss://example.com/sync/%s"})
|
||||
(reset! worker-state/*state (assoc state-prev :auth/id-token "state-token"))
|
||||
(reset! worker-state/*main-thread
|
||||
(fn [qkw & _args]
|
||||
(when (= qkw :thread-api/ensure-id&access-token)
|
||||
(swap! refresh-calls inc))
|
||||
(p/resolved {:id-token "refreshed-token"})))
|
||||
(with-redefs [platform/current (fn [] {:env {:runtime :node
|
||||
:owner-source :cli}})
|
||||
db-sync/id-token-expired? (fn [_token] true)]
|
||||
(-> (#'db-sync/<resolve-ws-token)
|
||||
(p/then (fn [token]
|
||||
(is (= 0 @refresh-calls))
|
||||
(is (= "state-token" token))
|
||||
(is (= "state-token" (worker-state/get-id-token)))
|
||||
(reset! worker-state/*main-thread main-thread-prev)
|
||||
(reset! worker-state/*db-sync-config config-prev)
|
||||
(reset! worker-state/*state state-prev)
|
||||
(done)))
|
||||
(p/catch (fn [error]
|
||||
(reset! worker-state/*main-thread main-thread-prev)
|
||||
(reset! worker-state/*db-sync-config config-prev)
|
||||
(reset! worker-state/*state state-prev)
|
||||
(is nil (str error))
|
||||
(done))))))))
|
||||
|
||||
(deftest connect-uses-platform-websocket-adapter-test
|
||||
(let [ws-ctor-prev js/WebSocket
|
||||
state-prev @worker-state/*state
|
||||
platform-map {:runtime :test}
|
||||
ws-calls (atom [])
|
||||
attach-calls (atom [])]
|
||||
(set! js/WebSocket (js* "(function(_url){ this.readyState = 1; })"))
|
||||
(reset! worker-state/*state (assoc state-prev :auth/id-token "token-123"))
|
||||
(try
|
||||
(with-redefs [platform/current (fn [] platform-map)
|
||||
platform/websocket-connect (fn [platform' url]
|
||||
(swap! ws-calls conj {:platform platform' :url url})
|
||||
(js-obj))
|
||||
db-sync/attach-ws-handlers! (fn [repo _client ws url]
|
||||
(swap! attach-calls conj {:repo repo :ws ws :url url}))]
|
||||
(let [connected (#'db-sync/connect! test-repo {:repo test-repo} "wss://example.com/sync/graph-1")
|
||||
ws (:ws connected)]
|
||||
(is (= [{:platform platform-map
|
||||
:url "wss://example.com/sync/graph-1?token=token-123"}]
|
||||
@ws-calls))
|
||||
(is (= [{:repo test-repo
|
||||
:ws ws
|
||||
:url "wss://example.com/sync/graph-1"}]
|
||||
@attach-calls))))
|
||||
(finally
|
||||
(reset! worker-state/*state state-prev)
|
||||
(set! js/WebSocket ws-ctor-prev)))))
|
||||
@@ -1,336 +0,0 @@
|
||||
(ns frontend.worker.sync.assets-test
|
||||
(:require [cljs.test :refer [async deftest is]]
|
||||
[datascript.core :as d]
|
||||
[frontend.common.crypt :as crypt]
|
||||
[frontend.worker.platform :as platform]
|
||||
[frontend.worker.shared-service :as shared-service]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync.assets :as sync-assets]
|
||||
[logseq.db :as ldb]
|
||||
[logseq.db.frontend.schema :as db-schema]
|
||||
[promesa.core :as p]))
|
||||
|
||||
(defn- asset-conn
|
||||
[asset-uuid]
|
||||
(let [conn (d/create-conn db-schema/schema)]
|
||||
(ldb/transact! conn [{:block/uuid asset-uuid
|
||||
:logseq.property.asset/type "png"
|
||||
:logseq.property.asset/checksum "sha-256-value"
|
||||
:logseq.property.asset/remote-metadata {:checksum "sha-256-value"
|
||||
:type "png"}}])
|
||||
conn))
|
||||
|
||||
(defn- execute-enqueued-asset-task!
|
||||
[task]
|
||||
(if (fn? task)
|
||||
(task)
|
||||
(p/resolved nil)))
|
||||
|
||||
(deftest request-asset-download-skips-existing-local-asset-test
|
||||
(async done
|
||||
(let [repo "asset-download-repo"
|
||||
graph-id "graph-1"
|
||||
asset-uuid (random-uuid)
|
||||
conn (asset-conn asset-uuid)
|
||||
download-calls (atom [])
|
||||
asset-stat-calls (atom [])
|
||||
enqueued-task (atom nil)
|
||||
broadcast-calls (atom [])]
|
||||
(-> (p/with-redefs [worker-state/get-datascript-conn (fn [_repo]
|
||||
conn)
|
||||
platform/current (fn [] {})
|
||||
platform/asset-stat (fn [_platform repo' file-name]
|
||||
(swap! asset-stat-calls conj [repo' file-name])
|
||||
(p/resolved {:size 10}))
|
||||
sync-assets/download-remote-asset! (fn [& args]
|
||||
(swap! download-calls conj args)
|
||||
(p/resolved nil))]
|
||||
(sync-assets/request-asset-download!
|
||||
repo
|
||||
asset-uuid
|
||||
{:current-client-f (fn [_repo]
|
||||
{:graph-id graph-id})
|
||||
:enqueue-asset-task-f (fn [_client task]
|
||||
(reset! enqueued-task task)
|
||||
(execute-enqueued-asset-task! task))
|
||||
:broadcast-rtc-state!-f (fn [& args]
|
||||
(swap! broadcast-calls conj args))}))
|
||||
(p/then (fn [_]
|
||||
(is (= [[repo (str asset-uuid ".png")]] @asset-stat-calls))
|
||||
(is (fn? @enqueued-task))
|
||||
(is (= [] @download-calls))
|
||||
(is (= [] @broadcast-calls))))
|
||||
(p/catch (fn [error]
|
||||
(is false (str "unexpected error: " error))))
|
||||
(p/finally done)))))
|
||||
|
||||
(deftest request-asset-download-downloads-missing-local-asset-test
|
||||
(async done
|
||||
(let [repo "asset-download-repo"
|
||||
graph-id "graph-1"
|
||||
asset-uuid (random-uuid)
|
||||
conn (asset-conn asset-uuid)
|
||||
download-calls (atom [])
|
||||
asset-stat-calls (atom [])
|
||||
broadcast-calls (atom [])]
|
||||
(-> (p/with-redefs [worker-state/get-datascript-conn (fn [_repo]
|
||||
conn)
|
||||
platform/current (fn [] {})
|
||||
platform/asset-stat (fn [_platform repo' file-name]
|
||||
(swap! asset-stat-calls conj [repo' file-name])
|
||||
(p/resolved nil))
|
||||
sync-assets/download-remote-asset! (fn [& args]
|
||||
(swap! download-calls conj args)
|
||||
(p/resolved nil))]
|
||||
(sync-assets/request-asset-download!
|
||||
repo
|
||||
asset-uuid
|
||||
{:current-client-f (fn [_repo]
|
||||
{:graph-id graph-id})
|
||||
:enqueue-asset-task-f (fn [_client task]
|
||||
(execute-enqueued-asset-task! task))
|
||||
:broadcast-rtc-state!-f (fn [& args]
|
||||
(swap! broadcast-calls conj args))}))
|
||||
(p/then (fn [_]
|
||||
(is (= [[repo (str asset-uuid ".png")]] @asset-stat-calls))
|
||||
(is (= [[repo graph-id asset-uuid "png"]] @download-calls))
|
||||
(is (= 1 (count @broadcast-calls)))))
|
||||
(p/catch (fn [error]
|
||||
(is false (str "unexpected error: " error))))
|
||||
(p/finally done)))))
|
||||
|
||||
(deftest request-asset-download-propagates-and-logs-download-failure-test
|
||||
(async done
|
||||
(let [repo "asset-download-repo"
|
||||
graph-id "graph-1"
|
||||
asset-uuid (random-uuid)
|
||||
conn (asset-conn asset-uuid)
|
||||
download-error (ex-info "download failed" {:type :rtc.exception/download-asset-failed})
|
||||
log-calls (atom [])]
|
||||
(-> (p/with-redefs [worker-state/get-datascript-conn (fn [_repo]
|
||||
conn)
|
||||
platform/current (fn [] {})
|
||||
platform/asset-stat (fn [_platform _repo _file-name]
|
||||
(p/resolved nil))
|
||||
sync-assets/download-remote-asset! (fn [& _args]
|
||||
(p/rejected download-error))
|
||||
sync-assets/log-request-asset-download-failed!
|
||||
(fn [repo' asset-uuid' error']
|
||||
(swap! log-calls conj [repo' asset-uuid' error']))]
|
||||
(sync-assets/request-asset-download!
|
||||
repo
|
||||
asset-uuid
|
||||
{:current-client-f (fn [_repo]
|
||||
{:graph-id graph-id})
|
||||
:enqueue-asset-task-f (fn [_client task]
|
||||
(execute-enqueued-asset-task! task))
|
||||
:broadcast-rtc-state!-f (fn [& _args] nil)}))
|
||||
(p/then (fn [_]
|
||||
(is false "expected download failure to reject")))
|
||||
(p/catch (fn [error]
|
||||
(is (= download-error error))
|
||||
(is (= [[repo asset-uuid download-error]] @log-calls))))
|
||||
(p/finally done)))))
|
||||
|
||||
(deftest upload-remote-asset-serializes-resolved-encrypted-payload-test
|
||||
(async done
|
||||
(let [repo "asset-upload-repo"
|
||||
graph-id "graph-1"
|
||||
asset-uuid (random-uuid)
|
||||
checksum "sha-256-value"
|
||||
asset-bytes (js/Uint8Array. #js [1 2 3])
|
||||
encrypted-payload {:cipher "encrypted-payload"}
|
||||
expected-body (ldb/write-transit-str encrypted-payload)
|
||||
fetch-call* (atom nil)
|
||||
encrypt-input* (atom nil)
|
||||
original-fetch js/fetch
|
||||
db-sync-config @worker-state/*db-sync-config]
|
||||
(reset! worker-state/*db-sync-config
|
||||
{:http-base "https://sync.example.test"})
|
||||
(set! js/fetch
|
||||
(fn [url opts]
|
||||
(reset! fetch-call*
|
||||
{:url url
|
||||
:body (.-body opts)})
|
||||
(p/resolved #js {:ok true
|
||||
:status 200})))
|
||||
(-> (p/with-redefs [sync-assets/graph-aes-key
|
||||
(fn [_repo _graph-id _fail-fast-f]
|
||||
(p/resolved "aes-key"))
|
||||
platform/current
|
||||
(fn [] {})
|
||||
platform/asset-read-bytes!
|
||||
(fn [_platform _repo _file-name]
|
||||
(p/resolved asset-bytes))
|
||||
crypt/<encrypt-uint8array
|
||||
(fn [_aes-key payload]
|
||||
(reset! encrypt-input* payload)
|
||||
(p/resolved encrypted-payload))
|
||||
shared-service/broadcast-to-clients!
|
||||
(fn [& _] nil)]
|
||||
(sync-assets/upload-remote-asset!
|
||||
repo graph-id asset-uuid "png" checksum))
|
||||
(p/then
|
||||
(fn [_]
|
||||
(is (instance? js/Uint8Array @encrypt-input*))
|
||||
(is (= expected-body (:body @fetch-call*)))))
|
||||
(p/catch
|
||||
(fn [error]
|
||||
(is false (str "unexpected error: " error))))
|
||||
(p/finally
|
||||
(fn []
|
||||
(set! js/fetch original-fetch)
|
||||
(reset! worker-state/*db-sync-config db-sync-config)
|
||||
(done)))))))
|
||||
|
||||
(deftest upload-remote-asset-records-missing-local-file-test
|
||||
(async done
|
||||
(let [repo "asset-upload-repo"
|
||||
graph-id "graph-1"
|
||||
asset-uuid (random-uuid)
|
||||
checksum "sha-256-value"
|
||||
missing-file (str "assets/" asset-uuid ".pdf")
|
||||
read-error (js/Error. "ENOENT: no such file or directory")
|
||||
original-fetch js/fetch
|
||||
db-sync-config @worker-state/*db-sync-config
|
||||
fetch-called? (atom false)
|
||||
broadcasts (atom [])]
|
||||
(reset! worker-state/*db-sync-config
|
||||
{:http-base "https://sync.example.test"})
|
||||
(set! js/fetch
|
||||
(fn [& _args]
|
||||
(reset! fetch-called? true)
|
||||
(p/resolved #js {:ok true
|
||||
:status 200})))
|
||||
(-> (p/with-redefs [sync-assets/graph-aes-key
|
||||
(fn [_repo _graph-id _fail-fast-f]
|
||||
(p/resolved nil))
|
||||
platform/current
|
||||
(fn [] {})
|
||||
platform/asset-read-bytes!
|
||||
(fn [_platform _repo _file-name]
|
||||
(p/rejected read-error))
|
||||
shared-service/broadcast-to-clients!
|
||||
(fn [event payload]
|
||||
(swap! broadcasts conj [event payload]))]
|
||||
(sync-assets/clear-missing-asset-upload-files! repo)
|
||||
(sync-assets/upload-remote-asset!
|
||||
repo graph-id asset-uuid "pdf" checksum))
|
||||
(p/then
|
||||
(fn [_]
|
||||
(is false "expected missing local file to reject")))
|
||||
(p/catch
|
||||
(fn [error]
|
||||
(is (= :rtc.exception/read-asset-failed (:type (ex-data error))))
|
||||
(is (false? @fetch-called?))
|
||||
(is (= [] @broadcasts))
|
||||
(is (= [{:asset-id (str asset-uuid)
|
||||
:asset-type "pdf"
|
||||
:file missing-file}]
|
||||
(sync-assets/get-missing-asset-upload-files repo)))))
|
||||
(p/finally
|
||||
(fn []
|
||||
(set! js/fetch original-fetch)
|
||||
(reset! worker-state/*db-sync-config db-sync-config)
|
||||
(sync-assets/clear-missing-asset-upload-files! repo)
|
||||
(done)))))))
|
||||
|
||||
(deftest download-missing-remote-assets-downloads-only-missing-sync-assets-test
|
||||
(async done
|
||||
(let [repo "asset-prefetch-repo"
|
||||
graph-id "graph-1"
|
||||
missing-uuid (random-uuid)
|
||||
existing-uuid (random-uuid)
|
||||
local-uuid (random-uuid)
|
||||
external-uuid (random-uuid)
|
||||
conn (d/create-conn db-schema/schema)
|
||||
stat-calls (atom [])
|
||||
download-calls (atom [])]
|
||||
(ldb/transact!
|
||||
conn
|
||||
[{:db/ident :logseq.class/Asset}
|
||||
{:block/uuid missing-uuid
|
||||
:block/tags #{:logseq.class/Asset}
|
||||
:logseq.property.asset/type "png"
|
||||
:logseq.property.asset/checksum "missing-checksum"
|
||||
:logseq.property.asset/remote-metadata {:checksum "missing-checksum"
|
||||
:type "png"}}
|
||||
{:block/uuid existing-uuid
|
||||
:block/tags #{:logseq.class/Asset}
|
||||
:logseq.property.asset/type "pdf"
|
||||
:logseq.property.asset/checksum "existing-checksum"
|
||||
:logseq.property.asset/remote-metadata {:checksum "existing-checksum"
|
||||
:type "pdf"}}
|
||||
{:block/uuid local-uuid
|
||||
:block/tags #{:logseq.class/Asset}
|
||||
:logseq.property.asset/type "jpg"
|
||||
:logseq.property.asset/checksum "local-checksum"}
|
||||
{:block/uuid external-uuid
|
||||
:block/tags #{:logseq.class/Asset}
|
||||
:logseq.property.asset/type "gif"
|
||||
:logseq.property.asset/checksum "external-checksum"
|
||||
:logseq.property.asset/remote-metadata {:checksum "external-checksum"
|
||||
:type "gif"}
|
||||
:logseq.property.asset/external-url "https://example.com/asset.gif"}])
|
||||
(-> (p/with-redefs [worker-state/get-datascript-conn (fn [_repo]
|
||||
conn)
|
||||
platform/current (fn [] {})
|
||||
platform/asset-stat
|
||||
(fn [_platform repo' file-name]
|
||||
(swap! stat-calls conj [repo' file-name])
|
||||
(p/resolved (when (= file-name (str existing-uuid ".pdf"))
|
||||
{:size 10})))
|
||||
sync-assets/download-remote-asset!
|
||||
(fn [& args]
|
||||
(swap! download-calls conj args)
|
||||
(p/resolved nil))]
|
||||
(sync-assets/download-missing-remote-assets! repo graph-id))
|
||||
(p/then (fn [result]
|
||||
(is (= {:total 2
|
||||
:downloaded 1
|
||||
:skipped-existing 1}
|
||||
result))
|
||||
(is (= #{[repo (str existing-uuid ".pdf")]
|
||||
[repo (str missing-uuid ".png")]}
|
||||
(set @stat-calls)))
|
||||
(is (= [[repo graph-id missing-uuid "png"]]
|
||||
@download-calls))))
|
||||
(p/catch (fn [error]
|
||||
(is false (str "unexpected error: " error))))
|
||||
(p/finally done)))))
|
||||
|
||||
(deftest download-remote-assets-if-missing-bounds-download-concurrency-test
|
||||
(async done
|
||||
(let [repo "asset-prefetch-repo"
|
||||
graph-id "graph-1"
|
||||
candidates (mapv (fn [_]
|
||||
{:asset-uuid (random-uuid)
|
||||
:asset-type "png"})
|
||||
(range 12))
|
||||
active-downloads (atom 0)
|
||||
max-active-downloads (atom 0)
|
||||
download-calls (atom [])]
|
||||
(-> (p/with-redefs [platform/current (fn [] {})
|
||||
platform/asset-stat
|
||||
(fn [_platform _repo _file-name]
|
||||
(p/resolved nil))
|
||||
sync-assets/download-remote-asset!
|
||||
(fn [repo' graph-id' asset-uuid asset-type]
|
||||
(swap! download-calls conj [repo' graph-id' asset-uuid asset-type])
|
||||
(let [active (swap! active-downloads inc)]
|
||||
(swap! max-active-downloads max active))
|
||||
(p/let [_ (p/delay 20)]
|
||||
(swap! active-downloads dec)
|
||||
nil))]
|
||||
(sync-assets/download-remote-assets-if-missing!
|
||||
repo graph-id candidates))
|
||||
(p/then (fn [result]
|
||||
(is (= {:total 12
|
||||
:downloaded 12
|
||||
:skipped-existing 0}
|
||||
result))
|
||||
(is (= 12 (count @download-calls)))
|
||||
(is (= 10 @max-active-downloads))))
|
||||
(p/catch (fn [error]
|
||||
(is false (str "unexpected error: " error))))
|
||||
(p/finally done)))))
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,111 +0,0 @@
|
||||
(ns frontend.worker.sync.download-test
|
||||
(:require [cljs.test :refer [async deftest is]]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync.crypt :as sync-crypt]
|
||||
[frontend.worker.sync.download :as sync-download]
|
||||
[frontend.worker.sync.log-and-state :as rtc-log-and-state]
|
||||
[logseq.db-sync.snapshot :as snapshot]
|
||||
[promesa.core :as p]))
|
||||
|
||||
(defn- frame-bytes
|
||||
[^js data]
|
||||
(let [len (.-byteLength data)
|
||||
out (js/Uint8Array. (+ 4 len))
|
||||
view (js/DataView. (.-buffer out))]
|
||||
(.setUint32 view 0 len false)
|
||||
(.set out data 4)
|
||||
out))
|
||||
|
||||
(defn- stream-from-payload
|
||||
[^js payload]
|
||||
(js/ReadableStream.
|
||||
#js {:start (fn [controller]
|
||||
(.enqueue controller payload)
|
||||
(.close controller))}))
|
||||
|
||||
(deftest stream-snapshot-row-batches-ignores-stale-gzip-header-test
|
||||
(async done
|
||||
(let [rows [[1 "row-1" nil]
|
||||
[2 "row-2" nil]]
|
||||
payload (frame-bytes (snapshot/encode-rows rows))
|
||||
resp (js/Response.
|
||||
(stream-from-payload payload)
|
||||
#js {:status 200
|
||||
:headers #js {"content-encoding" "gzip"}})
|
||||
batches* (atom [])]
|
||||
(-> (#'sync-download/<stream-snapshot-row-batches!
|
||||
resp
|
||||
1000
|
||||
(fn [batch]
|
||||
(swap! batches* conj batch)
|
||||
(p/resolved true)))
|
||||
(p/then (fn [_]
|
||||
(is (= [rows] @batches*))
|
||||
(done)))
|
||||
(p/catch (fn [error]
|
||||
(is false (str error))
|
||||
(done)))))))
|
||||
|
||||
(deftest encrypted-download-preflights-e2ee-before-fetching-snapshot-stream-test
|
||||
(async done
|
||||
(let [config-prev @worker-state/*db-sync-config
|
||||
fetch-prev js/fetch
|
||||
calls (atom [])]
|
||||
(reset! worker-state/*db-sync-config {:http-base "https://sync.example.test"})
|
||||
(set! js/fetch
|
||||
(fn [_url _opts]
|
||||
(swap! calls conj :snapshot-stream)
|
||||
(js/Promise.resolve #js {:ok true})))
|
||||
(-> (p/with-redefs [sync-download/fetch-json (fn [_url _opts schema]
|
||||
(case schema
|
||||
:sync/pull
|
||||
(p/resolved {:t 42})
|
||||
|
||||
:sync/snapshot-download
|
||||
(p/resolved {:url "https://sync.example.test/snapshot"})
|
||||
|
||||
(p/rejected (ex-info "unexpected schema" {:schema schema}))))
|
||||
sync-crypt/<fetch-graph-aes-key-for-download (fn [_graph-id]
|
||||
(swap! calls conj :e2ee-preflight)
|
||||
(p/resolved :aes-key))
|
||||
sync-download/<stream-snapshot-row-batches! (fn [_resp _batch-size _on-batch]
|
||||
(p/resolved {:chunk-count 0}))]
|
||||
(sync-download/download-graph-by-id! "repo" "graph-1" true))
|
||||
(p/then (fn [_]
|
||||
(is (= [:e2ee-preflight :snapshot-stream] @calls))))
|
||||
(p/catch (fn [error]
|
||||
(is false (str error))))
|
||||
(p/finally (fn []
|
||||
(set! js/fetch fetch-prev)
|
||||
(reset! worker-state/*db-sync-config config-prev)
|
||||
(done)))))))
|
||||
|
||||
(deftest encrypted-download-failure-emits-completed-log-test
|
||||
(async done
|
||||
(let [config-prev @worker-state/*db-sync-config
|
||||
log-events (atom [])]
|
||||
(reset! worker-state/*db-sync-config {:http-base "https://sync.example.test"})
|
||||
(-> (p/with-redefs [sync-download/fetch-json (fn [_url _opts schema]
|
||||
(case schema
|
||||
:sync/pull
|
||||
(p/resolved {:t 42})
|
||||
|
||||
:sync/snapshot-download
|
||||
(p/resolved {:url "https://sync.example.test/snapshot"})
|
||||
|
||||
(p/rejected (ex-info "unexpected schema" {:schema schema}))))
|
||||
sync-crypt/<fetch-graph-aes-key-for-download (fn [_graph-id]
|
||||
(p/rejected (ex-info "decrypt-private-key" {})))
|
||||
rtc-log-and-state/rtc-log (fn [type payload]
|
||||
(swap! log-events conj (assoc payload :type type))
|
||||
nil)]
|
||||
(sync-download/download-graph-by-id! "repo" "graph-1" true))
|
||||
(p/then (fn [_]
|
||||
(is false "expected download failure")))
|
||||
(p/catch (fn [error]
|
||||
(is (= "db-sync download failed" (ex-message error)))
|
||||
(is (= [:download-progress :download-completed]
|
||||
(mapv :sub-type @log-events)))))
|
||||
(p/finally (fn []
|
||||
(reset! worker-state/*db-sync-config config-prev)
|
||||
(done)))))))
|
||||
@@ -1,186 +0,0 @@
|
||||
(ns frontend.worker.sync.restart-test
|
||||
(:require [cljs.test :refer [async deftest is]]
|
||||
[frontend.worker.platform :as platform]
|
||||
[frontend.worker.shared-service :as shared-service]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync :as sync]
|
||||
[frontend.worker.sync.client-op :as client-op]
|
||||
[frontend.worker.sync.util :as sync-util]
|
||||
[promesa.core :as p]))
|
||||
|
||||
(deftest start-reconnects-closed-ws-with-stale-open-state-test
|
||||
(async done
|
||||
(let [repo "stale-repo"
|
||||
graph-id "graph-1"
|
||||
prev-client @worker-state/*db-sync-client
|
||||
prev-config @worker-state/*db-sync-config
|
||||
prev-platform (try
|
||||
(platform/current)
|
||||
(catch :default _ nil))
|
||||
connect-calls (atom 0)
|
||||
stale-ws #js {:readyState 3
|
||||
:close (fn [] nil)}
|
||||
stale-client {:repo repo
|
||||
:graph-id graph-id
|
||||
:ws stale-ws
|
||||
:ws-state (atom :open)
|
||||
:online-users (atom [])
|
||||
:reconnect (atom {:attempt 0 :timer nil})
|
||||
:stale-kill-timer (atom nil)}]
|
||||
(reset! worker-state/*db-sync-config {:ws-url "wss://sync.example.test/sync/%s"})
|
||||
(reset! worker-state/*db-sync-client stale-client)
|
||||
(platform/set-platform!
|
||||
{:env {:runtime :node}
|
||||
:storage {}
|
||||
:kv {}
|
||||
:broadcast {}
|
||||
:websocket {:connect (fn [_url]
|
||||
(swap! connect-calls inc)
|
||||
#js {:readyState 0
|
||||
:close (fn [] nil)})}
|
||||
:crypto {}
|
||||
:timers {}
|
||||
:sqlite {}})
|
||||
(-> (p/with-redefs [worker-state/get-client-ops-conn (fn [_repo] true)
|
||||
client-op/get-local-tx (fn [_repo] 0)
|
||||
client-op/get-pending-local-tx-count (fn [_repo] 0)
|
||||
client-op/get-unpushed-asset-ops-count (fn [_repo] 0)
|
||||
client-op/get-all-asset-ops (fn [_repo] [])
|
||||
client-op/get-local-checksum (fn [_repo] nil)
|
||||
client-op/get-graph-uuid (fn [_repo] graph-id)
|
||||
client-op/update-graph-uuid (fn [_repo _graph-id] nil)
|
||||
sync-util/get-graph-id (fn [_repo] graph-id)
|
||||
sync/<resolve-ws-token (fn [] (p/resolved "token"))
|
||||
shared-service/broadcast-to-clients! (fn [& _] nil)]
|
||||
(sync/start! repo))
|
||||
(p/then
|
||||
(fn [_]
|
||||
(is (= 1 @connect-calls)
|
||||
"start! should reconnect when the cached websocket is already closed")))
|
||||
(p/catch
|
||||
(fn [error]
|
||||
(is false (str "unexpected error: " error))))
|
||||
(p/finally
|
||||
(fn []
|
||||
(reset! worker-state/*db-sync-client prev-client)
|
||||
(reset! worker-state/*db-sync-config prev-config)
|
||||
(when prev-platform
|
||||
(platform/set-platform! prev-platform))
|
||||
(done)))))))
|
||||
|
||||
(deftest start-skips-when-client-op-local-tx-is-missing-test
|
||||
(async done
|
||||
(let [repo "missing-local-tx-repo"
|
||||
graph-id "graph-1"
|
||||
prev-client @worker-state/*db-sync-client
|
||||
prev-config @worker-state/*db-sync-config
|
||||
prev-platform (try
|
||||
(platform/current)
|
||||
(catch :default _ nil))
|
||||
connect-calls (atom 0)]
|
||||
(reset! worker-state/*db-sync-config {:ws-url "wss://sync.example.test/sync/%s"})
|
||||
(reset! worker-state/*db-sync-client nil)
|
||||
(platform/set-platform!
|
||||
{:env {:runtime :node}
|
||||
:storage {}
|
||||
:kv {}
|
||||
:broadcast {}
|
||||
:websocket {:connect (fn [_url]
|
||||
(swap! connect-calls inc)
|
||||
#js {:readyState 0
|
||||
:close (fn [] nil)})}
|
||||
:crypto {}
|
||||
:timers {}
|
||||
:sqlite {}})
|
||||
(-> (p/with-redefs [worker-state/get-client-ops-conn (fn [_repo] true)
|
||||
client-op/get-local-tx (fn [_repo] nil)
|
||||
client-op/update-local-tx (fn [_repo _tx]
|
||||
(throw (js/Error. "must not initialize missing local-tx to 0")))
|
||||
sync-util/get-graph-id (fn [_repo] graph-id)]
|
||||
(sync/start! repo))
|
||||
(p/then
|
||||
(fn [_]
|
||||
(is (zero? @connect-calls)
|
||||
"start! should wait for valid client-op local-tx instead of syncing from 0")))
|
||||
(p/catch
|
||||
(fn [error]
|
||||
(is false (str "unexpected error: " error))))
|
||||
(p/finally
|
||||
(fn []
|
||||
(reset! worker-state/*db-sync-client prev-client)
|
||||
(reset! worker-state/*db-sync-config prev-config)
|
||||
(when prev-platform
|
||||
(platform/set-platform! prev-platform))
|
||||
(done)))))))
|
||||
|
||||
(deftest stale-loop-marks-non-open-ws-closed-test
|
||||
(let [broadcasts (atom [])
|
||||
clear-interval-calls (atom [])
|
||||
interval-f* (atom nil)
|
||||
original-clear-interval js/clearInterval
|
||||
original-set-interval js/setInterval
|
||||
prev-client @worker-state/*db-sync-client
|
||||
ws #js {:readyState 3
|
||||
:close (fn [] nil)}
|
||||
client {:repo "stale-repo"
|
||||
:graph-id "graph-1"
|
||||
:ws ws
|
||||
:ws-state (atom :open)
|
||||
:inflight (atom [(random-uuid)])
|
||||
:online-users (atom [])
|
||||
:stale-kill-timer (atom nil)}]
|
||||
(set! js/setInterval
|
||||
(fn [f _ms]
|
||||
(reset! interval-f* f)
|
||||
:interval-id))
|
||||
(set! js/clearInterval
|
||||
(fn [interval-id]
|
||||
(swap! clear-interval-calls conj interval-id)))
|
||||
(reset! worker-state/*db-sync-client client)
|
||||
(with-redefs [shared-service/broadcast-to-clients!
|
||||
(fn [topic payload]
|
||||
(swap! broadcasts conj [topic payload]))]
|
||||
(try
|
||||
(#'sync/close-stale-ws-loop client ws "wss://sync.example.test/sync/graph-1")
|
||||
(@interval-f*)
|
||||
(is (= :closed @(:ws-state client)))
|
||||
(is (empty? @(:inflight client)))
|
||||
(is (= [:interval-id] @clear-interval-calls))
|
||||
(is (some #(= :rtc-sync-state (first %)) @broadcasts))
|
||||
(finally
|
||||
(reset! worker-state/*db-sync-client prev-client)
|
||||
(set! js/setInterval original-set-interval)
|
||||
(set! js/clearInterval original-clear-interval))))))
|
||||
|
||||
(deftest ws-close-clears-inflight-before-reconnect-test
|
||||
(let [broadcasts (atom [])
|
||||
original-set-timeout js/setTimeout
|
||||
prev-client @worker-state/*db-sync-client
|
||||
ws #js {:readyState 3
|
||||
:close (fn [] nil)}
|
||||
client {:repo "stale-repo"
|
||||
:graph-id "graph-1"
|
||||
:ws ws
|
||||
:ws-state (atom :open)
|
||||
:inflight (atom [(random-uuid)])
|
||||
:online-users (atom [:user])
|
||||
:reconnect (atom {:attempt 0 :timer nil})
|
||||
:stale-kill-timer (atom nil)}]
|
||||
(set! js/setTimeout
|
||||
(fn [_f _ms]
|
||||
:timeout-id))
|
||||
(reset! worker-state/*db-sync-client client)
|
||||
(with-redefs [shared-service/broadcast-to-clients!
|
||||
(fn [topic payload]
|
||||
(swap! broadcasts conj [topic payload]))]
|
||||
(try
|
||||
(#'sync/attach-ws-handlers! "stale-repo" client ws "wss://sync.example.test/sync/graph-1")
|
||||
((.-onclose ws) #js {})
|
||||
(is (= :closed @(:ws-state client)))
|
||||
(is (empty? @(:inflight client)))
|
||||
(is (empty? @(:online-users client)))
|
||||
(is (= :timeout-id (:timer @(:reconnect client))))
|
||||
(is (some #(= :rtc-sync-state (first %)) @broadcasts))
|
||||
(finally
|
||||
(reset! worker-state/*db-sync-client prev-client)
|
||||
(set! js/setTimeout original-set-timeout))))))
|
||||
@@ -1,269 +0,0 @@
|
||||
(ns frontend.worker.sync.upload-test
|
||||
(:require [cljs.test :refer [async deftest is]]
|
||||
[datascript.core :as d]
|
||||
[frontend.worker.sync.crypt :as sync-crypt]
|
||||
[frontend.worker.sync.temp-sqlite :as sync-temp-sqlite]
|
||||
[frontend.worker.sync.util :as sync-util]
|
||||
[frontend.worker.sync.upload :as sync-upload]
|
||||
[logseq.db.frontend.schema :as db-schema]
|
||||
[promesa.core :as p]
|
||||
[clojure.string :as string]))
|
||||
|
||||
(deftest split-snapshot-rows-by-max-bytes-splits-rows-into-byte-capped-batches-test
|
||||
(let [sizes {:a 4
|
||||
:b 4
|
||||
:c 4
|
||||
:d 4}
|
||||
rows [[:a] [:b] [:c] [:d]]]
|
||||
(with-redefs [sync-upload/snapshot-rows-byte-length
|
||||
(fn [rows']
|
||||
(reduce + (map (fn [[addr]] (get sizes addr 0)) rows')))]
|
||||
(is (= [[[:a] [:b]]
|
||||
[[:c] [:d]]]
|
||||
(#'sync-upload/split-snapshot-rows-by-max-bytes rows 8))))))
|
||||
|
||||
(deftest split-snapshot-rows-by-max-bytes-fails-fast-for-oversized-single-row-test
|
||||
(let [sizes {:ok 3
|
||||
:too-big 11}
|
||||
rows [[:ok] [:too-big]]]
|
||||
(with-redefs [sync-upload/snapshot-rows-byte-length
|
||||
(fn [rows']
|
||||
(reduce + (map (fn [[addr]] (get sizes addr 0)) rows')))]
|
||||
(try
|
||||
(#'sync-upload/split-snapshot-rows-by-max-bytes rows 10)
|
||||
(is false "expected snapshot row too large error")
|
||||
(catch :default error
|
||||
(let [data (ex-data error)]
|
||||
(is (= "snapshot-row-too-large" (ex-message error)))
|
||||
(is (= 10 (:max-bytes data)))
|
||||
(is (= 11 (:row-size data)))
|
||||
(is (= :too-big (:addr data)))))))))
|
||||
|
||||
(deftest upload-snapshot-rows-batches-sets-reset-and-finished-flags-correctly-test
|
||||
(async done
|
||||
(let [calls* (atom [])
|
||||
rows-batches [[[1 "a" nil]]
|
||||
[[2 "b" nil]]
|
||||
[[3 "c" nil]]]]
|
||||
(-> (p/with-redefs [sync-upload/<snapshot-upload-body
|
||||
(fn [rows]
|
||||
(p/resolved {:body rows
|
||||
:encoding nil}))]
|
||||
(#'sync-upload/<upload-snapshot-rows-batches!
|
||||
rows-batches
|
||||
{:base "https://sync.example.test"
|
||||
:graph-id "graph-1"
|
||||
:first-batch? true
|
||||
:finished? true
|
||||
:checksum "abc+123="
|
||||
:auth-fetch-f
|
||||
(fn [url headers body]
|
||||
(swap! calls* conj {:url url
|
||||
:headers headers
|
||||
:body body})
|
||||
(p/resolved true))}))
|
||||
(p/then
|
||||
(fn [_]
|
||||
(is (= 3 (count @calls*)))
|
||||
(is (string/includes? (:url (nth @calls* 0)) "reset=true"))
|
||||
(is (string/includes? (:url (nth @calls* 0)) "finished=false"))
|
||||
(is (string/includes? (:url (nth @calls* 1)) "reset=false"))
|
||||
(is (string/includes? (:url (nth @calls* 1)) "finished=false"))
|
||||
(is (string/includes? (:url (nth @calls* 2)) "reset=false"))
|
||||
(is (string/includes? (:url (nth @calls* 2)) "finished=true"))
|
||||
(is (string/includes? (:url (nth @calls* 2)) "checksum=abc%2B123%3D"))
|
||||
(done)))
|
||||
(p/catch
|
||||
(fn [error]
|
||||
(is false (str error))
|
||||
(done)))))))
|
||||
|
||||
(deftest drop-oversized-upload-datoms-drops-large-tldraw-page-values-test
|
||||
(let [datoms [{:e 1 :a :block/title :v "safe"}
|
||||
{:e 2 :a :logseq.property.tldraw/page :v {:id "small"}}
|
||||
{:e 3 :a :logseq.property.tldraw/page :v {:id "huge"}}]]
|
||||
(with-redefs [sync-upload/datom-value-byte-length
|
||||
(fn [value]
|
||||
(case (:id value)
|
||||
"small" 32
|
||||
"huge" 1500000
|
||||
0))]
|
||||
(let [{:keys [kept dropped]} (#'sync-upload/drop-oversized-upload-datoms datoms)]
|
||||
(is (= 2 (count kept)))
|
||||
(is (= [1 2] (mapv :e kept)))
|
||||
(is (= 1 (count dropped)))
|
||||
(is (= {:a :logseq.property.tldraw/page
|
||||
:e 3
|
||||
:bytes 1500000}
|
||||
(first dropped)))))))
|
||||
|
||||
(deftest snapshot-upload-excludes-local-block-revisions-test
|
||||
(async done
|
||||
(let [source-conn (d/create-conn db-schema/schema)
|
||||
first-uuid (random-uuid)
|
||||
second-uuid (random-uuid)]
|
||||
(d/transact! source-conn
|
||||
[{:block/uuid first-uuid
|
||||
:block/title "first"
|
||||
:block/tx-id 41}
|
||||
{:block/uuid second-uuid
|
||||
:block/title "second"
|
||||
:block/tx-id 42}])
|
||||
(-> (p/with-redefs [sync-temp-sqlite/<create-temp-sqlite-conn
|
||||
(fn [schema _datoms]
|
||||
(p/resolved {:conn (d/create-conn schema)}))]
|
||||
(#'sync-upload/<prepare-upload-temp-sqlite!
|
||||
"repo"
|
||||
"graph-id"
|
||||
source-conn
|
||||
nil
|
||||
(fn [& _] nil)))
|
||||
(p/then
|
||||
(fn [{:keys [conn]}]
|
||||
(is (= "first"
|
||||
(:block/title (d/entity @conn [:block/uuid first-uuid]))))
|
||||
(is (= "second"
|
||||
(:block/title (d/entity @conn [:block/uuid second-uuid]))))
|
||||
(is (empty? (filter #(= :block/tx-id (:a %))
|
||||
(d/datoms @conn :eavt)))
|
||||
"Snapshot transport must not copy worker-local block revisions.")))
|
||||
(p/catch
|
||||
(fn [error]
|
||||
(is false (str error))))
|
||||
(p/finally done)))))
|
||||
|
||||
(deftest create-remote-graph-creates-new-remote-graph-when-no-remote-match-test
|
||||
(async done
|
||||
(let [calls* (atom [])]
|
||||
(-> (p/with-redefs [sync-crypt/<preflight-upload-e2ee! (fn [_repo graph-e2ee?]
|
||||
(swap! calls* conj [:preflight graph-e2ee?])
|
||||
(p/resolved nil))
|
||||
sync-upload/list-remote-graphs! (fn []
|
||||
(swap! calls* conj [:list-remote-graphs])
|
||||
(p/resolved []))
|
||||
sync-upload/<create-remote-graph-aux! (fn [repo opts]
|
||||
(swap! calls* conj [:create-aux repo opts])
|
||||
(p/resolved {:graph-id "new-graph-id"
|
||||
:graph-e2ee? (:graph-e2ee? opts)}))]
|
||||
(sync-upload/create-remote-graph! "repo-1" {:graph-e2ee? true
|
||||
:graph-ready-for-use? false}))
|
||||
(p/then (fn [identity]
|
||||
(is (= {:graph-id "new-graph-id" :graph-e2ee? true}
|
||||
identity))
|
||||
(is (= [[:list-remote-graphs]
|
||||
[:preflight true]
|
||||
[:create-aux "repo-1" {:graph-e2ee? true
|
||||
:graph-ready-for-use? false}]]
|
||||
@calls*))))
|
||||
(p/catch (fn [error]
|
||||
(is false (str "unexpected error: " error))))
|
||||
(p/finally done)))))
|
||||
|
||||
(deftest create-remote-graph-aux-skips-rsa-keys-for-non-e2ee-graph-test
|
||||
(async done
|
||||
(let [calls* (atom [])]
|
||||
(-> (p/with-redefs [sync-crypt/<preflight-upload-e2ee! (fn [_repo graph-e2ee?]
|
||||
(swap! calls* conj [:preflight graph-e2ee?])
|
||||
(p/resolved nil))
|
||||
sync-upload/list-remote-graphs! (fn []
|
||||
(swap! calls* conj [:list-remote-graphs])
|
||||
(p/resolved []))
|
||||
sync-upload/http-base-url (fn [] "https://sync.example.test")
|
||||
sync-util/require-auth-token! (fn [context]
|
||||
(swap! calls* conj [:require-auth context]))
|
||||
sync-crypt/ensure-user-rsa-keys! (fn [opts]
|
||||
(swap! calls* conj [:ensure-rsa opts])
|
||||
(p/resolved nil))
|
||||
sync-util/fetch-json (fn [url request _opts]
|
||||
(swap! calls* conj [:fetch url request])
|
||||
(p/resolved {:graph-id "new-graph-id"
|
||||
:graph-e2ee? false}))
|
||||
sync-upload/persist-upload-graph-identity! (fn [repo graph-id graph-e2ee?]
|
||||
(swap! calls* conj [:persist repo graph-id graph-e2ee?])
|
||||
{:graph-id graph-id
|
||||
:graph-e2ee? graph-e2ee?})]
|
||||
(sync-upload/create-remote-graph! "repo-1" {:graph-e2ee? false
|
||||
:graph-ready-for-use? false}))
|
||||
(p/then (fn [identity]
|
||||
(is (= {:graph-id "new-graph-id"
|
||||
:graph-e2ee? false}
|
||||
identity))
|
||||
(is (not-any? #(= :ensure-rsa (first %)) @calls*))
|
||||
(is (= [[:list-remote-graphs]
|
||||
[:preflight false]
|
||||
[:require-auth {:repo "repo-1" :field :auth-token}]
|
||||
[:fetch "https://sync.example.test/graphs"
|
||||
{:method "POST"
|
||||
:headers {"content-type" "application/json"}
|
||||
:body "{\"graph-name\":\"repo-1\",\"schema-version\":null,\"graph-e2ee?\":false,\"graph-ready-for-use?\":false}"}]
|
||||
[:persist "repo-1" "new-graph-id" false]]
|
||||
@calls*))))
|
||||
(p/catch (fn [error]
|
||||
(is false (str "unexpected error: " error))))
|
||||
(p/finally done)))))
|
||||
|
||||
(deftest create-remote-graph-rejects-matching-remote-graph-test
|
||||
(async done
|
||||
(let [create-aux-called? (atom false)]
|
||||
(-> (p/with-redefs [sync-crypt/<preflight-upload-e2ee! (fn [_repo _graph-e2ee?]
|
||||
(p/resolved nil))
|
||||
sync-upload/list-remote-graphs! (fn []
|
||||
(p/resolved [{:graph-name "repo-1"}]))
|
||||
sync-upload/<create-remote-graph-aux! (fn [_repo _opts]
|
||||
(reset! create-aux-called? true)
|
||||
(p/resolved {:graph-id "new-graph-id"}))]
|
||||
(sync-upload/create-remote-graph! "repo-1" {:graph-e2ee? true
|
||||
:graph-ready-for-use? false}))
|
||||
(p/then (fn [_]
|
||||
(is false "expected graph-already-exists error")))
|
||||
(p/catch (fn [error]
|
||||
(is (= "remote graph already exists; delete it before uploading again"
|
||||
(ex-message error)))
|
||||
(is (= :db-sync/graph-already-exists (:code (ex-data error))))
|
||||
(is (= "repo-1" (:graph-name (ex-data error))))
|
||||
(is (false? @create-aux-called?))))
|
||||
(p/finally done)))))
|
||||
|
||||
(deftest create-remote-graph-missing-e2ee-password-does-not-create-remote-graph-test
|
||||
(async done
|
||||
(let [calls* (atom [])]
|
||||
(-> (p/with-redefs [sync-upload/list-remote-graphs! (fn []
|
||||
(swap! calls* conj :list-remote-graphs)
|
||||
(p/resolved []))
|
||||
sync-crypt/<preflight-upload-e2ee! (fn [_repo _graph-e2ee?]
|
||||
(swap! calls* conj :preflight)
|
||||
(p/rejected (ex-info "missing-e2ee-password"
|
||||
{:code :db-sync/missing-e2ee-password
|
||||
:field :e2ee-password
|
||||
:reason :missing-persisted-password
|
||||
:hint "Provide --e2ee-password to persist it."})))
|
||||
sync-upload/<create-remote-graph-aux! (fn [_repo _opts]
|
||||
(swap! calls* conj :create-remote-graph)
|
||||
(p/resolved {:graph-id "new-graph-id"}))]
|
||||
(sync-upload/create-remote-graph! "repo-1" {:graph-e2ee? true
|
||||
:graph-ready-for-use? false}))
|
||||
(p/then (fn [_]
|
||||
(is false "expected missing e2ee password error")))
|
||||
(p/catch (fn [error]
|
||||
(is (= "missing-e2ee-password" (ex-message error)))
|
||||
(is (= :db-sync/missing-e2ee-password (:code (ex-data error))))
|
||||
(is (= [:list-remote-graphs :preflight] @calls*))))
|
||||
(p/finally done)))))
|
||||
|
||||
(deftest create-remote-graph-missing-e2ee-password-does-not-run-create-aux-test
|
||||
(async done
|
||||
(let [create-aux-called? (atom false)]
|
||||
(-> (p/with-redefs [sync-upload/list-remote-graphs! (fn [] (p/resolved []))
|
||||
sync-crypt/<preflight-upload-e2ee! (fn [_repo _graph-e2ee?]
|
||||
(p/rejected (ex-info "missing-e2ee-password"
|
||||
{:code :db-sync/missing-e2ee-password})))
|
||||
sync-upload/<create-remote-graph-aux! (fn [_repo _opts]
|
||||
(reset! create-aux-called? true)
|
||||
(p/resolved {:graph-id "new-graph-id"}))]
|
||||
(sync-upload/create-remote-graph! "repo-1" {:graph-e2ee? true
|
||||
:graph-ready-for-use? false}))
|
||||
(p/then (fn [_]
|
||||
(is false "expected missing e2ee password error")))
|
||||
(p/catch (fn [_error]
|
||||
(is (false? @create-aux-called?))))
|
||||
(p/finally done)))))
|
||||
@@ -1,24 +0,0 @@
|
||||
(ns frontend.worker.sync.util-test
|
||||
(:require [cljs.test :refer [deftest is]]
|
||||
[frontend.worker.sync.util :as sync-util]
|
||||
[logseq.common.version :as build-version]))
|
||||
|
||||
(deftest coerce-http-request-adds-client-revision-to-tx-batch-test
|
||||
(is (= (build-version/revision)
|
||||
(:client-revision
|
||||
(sync-util/coerce-http-request :sync/tx-batch
|
||||
{:t-before 0
|
||||
:txs []})))))
|
||||
|
||||
(deftest coerce-http-request-does-not-add-client-revision-to-other-requests-test
|
||||
(is (= {:graph-name "Demo"}
|
||||
(sync-util/coerce-http-request :graphs/create
|
||||
{:graph-name "Demo"}))))
|
||||
|
||||
(deftest coerce-http-request-preserves-explicit-client-revision-test
|
||||
(is (= "explicit-revision"
|
||||
(:client-revision
|
||||
(sync-util/coerce-http-request :sync/tx-batch
|
||||
{:t-before 0
|
||||
:txs []
|
||||
:client-revision "explicit-revision"})))))
|
||||
@@ -2,11 +2,11 @@
|
||||
(:require [cljs.test :refer [deftest is testing use-fixtures]]
|
||||
[datascript.core :as d]
|
||||
[frontend.worker.a-test-env]
|
||||
[frontend.worker.sync.apply-txs :as sync-apply]
|
||||
[frontend.worker.db-listener :as db-listener]
|
||||
[frontend.worker.local-tx-log :as local-tx-log]
|
||||
[frontend.worker.state :as worker-state]
|
||||
[frontend.worker.sync :as db-sync]
|
||||
[frontend.worker.sync.client-op :as client-op]
|
||||
[frontend.worker.undo-redo :as worker-undo-redo]
|
||||
[frontend.worker.undo-redo.replay :as replay]
|
||||
[logseq.common.util.date-time :as date-time-util]
|
||||
[logseq.db :as ldb]
|
||||
[logseq.db.test.helper :as db-test]
|
||||
@@ -18,7 +18,7 @@
|
||||
[]
|
||||
(let [Database (js/require "better-sqlite3")
|
||||
db (new Database ":memory:")]
|
||||
(client-op/ensure-sqlite-schema! db)
|
||||
(local-tx-log/ensure-sqlite-schema! db)
|
||||
db))
|
||||
|
||||
(defn- delete-client-op-tx-row!
|
||||
@@ -52,15 +52,15 @@
|
||||
client-ops-conn (new-client-ops-db)]
|
||||
(reset! worker-state/*datascript-conns {test-repo conn})
|
||||
(reset! worker-state/*client-ops-conns {test-repo client-ops-conn})
|
||||
(reset! worker-undo-redo/*apply-history-action! sync-apply/apply-history-action!)
|
||||
(d/listen! conn ::gen-undo-ops
|
||||
(fn [tx-report]
|
||||
(db-sync/enqueue-local-tx! test-repo tx-report)))
|
||||
(reset! worker-undo-redo/*apply-history-action! replay/apply-history-action!)
|
||||
;; Real production dispatch (db_listener.cljs), scoped to just the
|
||||
;; :persist-local-tx handler -- this is what actually wires
|
||||
;; persist-local-tx! to every committed local tx outside tests.
|
||||
(db-listener/listen-db-changes! test-repo conn :handler-keys [:persist-local-tx])
|
||||
(worker-undo-redo/clear-history! test-repo)
|
||||
(try
|
||||
(f)
|
||||
(finally
|
||||
(d/unlisten! conn ::gen-undo-ops)
|
||||
(worker-undo-redo/clear-history! test-repo)
|
||||
(.close client-ops-conn)
|
||||
(reset! worker-undo-redo/*apply-history-action! apply-history-action-prev)
|
||||
@@ -286,8 +286,8 @@
|
||||
|
||||
(defn- poison-history-tx-order!
|
||||
[tx-id]
|
||||
(when-let [entry (client-op/get-local-tx-entry test-repo tx-id)]
|
||||
(client-op/upsert-local-tx-entry!
|
||||
(when-let [entry (local-tx-log/get-local-tx-entry test-repo tx-id)]
|
||||
(local-tx-log/upsert-local-tx-entry!
|
||||
test-repo
|
||||
{:tx-id tx-id
|
||||
:pending? true
|
||||
|
||||
Reference in New Issue
Block a user