dasemu/Synapse
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fase 8 (1/N): delete the worker-side remote sync protocol

Browse Source 
Deletes frontend.worker.sync/* wholesale (apply_txs, asset_db_listener,
assets, auth, client_op, const, crypt, download, handle_message,
large_title, log_and_state, presence, temp_sqlite, transport, upload,
util), worker/sync.cljs, worker/handler/sync.cljs, and
worker/ui_request.cljs -- the remote CRDT sync protocol and its E2EE
password-request plumbing, both made obsolete by the server-client
daemon architecture built in Fases 1-7. Deviates from the plan's KEEP
list on sync/presence.cljs and sync/transport.cljs: neither has any
caller left once the rest of the cluster is gone (confirmed by grep),
because Fase 4's WS resume-by-revision work already built its own
self-contained module (db_worker_node_ws.cljs) instead of adopting
these as the plan assumed it would.

This touched far more than a mechanical deletion:

- frontend.worker.sync.client-op wasn't sync-only. Its client_ops
  SQLite table is also the idempotency marker for db_core.cljs's
  built-in-property repair migration and the forward/inverse-outliner-
  ops relay undo/redo reads back through history-action-ops-by-tx-id.
  Trimmed (not deleted) into frontend.worker.local-tx-log, keeping only
  the tx-entry/local-tx functions those two callers need and dropping
  the genuinely sync-only surface (pending-upload counts, checksums,
  graph-uuid, sync-conflict tracking, asset-op queues).

- frontend.worker.sync.apply-txs's apply-history-action! looked
  sync-specific (it lived next to the remote-tx-apply/rebase machinery)
  but turned out to be the *only* implementation ever registered for
  worker-undo-redo/*apply-history-action! -- undo_redo.cljs calls it on
  every db-transact undo/redo, local edits included, not just
  remote-origin ones. Deleting it outright would have silently broken
  undo/redo app-wide the next time anyone hit Cmd-Z. Ported the
  general-purpose slice (replaying a recorded entry's forward/inverse
  outliner ops against the current db state, plus the raw-tx-data
  fallback for edits that bypass the structured outliner-op layer) into
  a new frontend.worker.undo-redo.replay, along with persist-local-tx!
  (derive history ops on every local commit, stash them in
  local-tx-log, hand off to gen-undo-ops!) -- dropping only the
  remote-rebase-specific rest (CRDT conflict resolution, upload-queue
  bookkeeping). Wired into db_listener.cljs as a renamed
  :persist-local-tx handler (was :db-sync, a name that stopped meaning
  anything once sync was gone).

  Caught by running the full existing undo_redo_test.cljs suite (30
  deftests written for the old mechanism) against the new module after
  fixing its test harness to use the real db_listener.cljs dispatch
  instead of a hand-rolled listener -- all 29 tests port over passing
  with no behavior change, which is the actual evidence this preserves
  undo/redo correctness rather than just compiling.

- db_core.cljs's own 8 :thread-api/db-sync-* handlers (remote-graph-
  download import, close-db/invalidate-search-db/recreate-lock
  sync-specific variants) are deleted along with their sync-crypt/
  sync-download calls; :thread-api/unsafe-unlink-db, :close-db, and
  :release-access-handles keep their real (non-sync) bodies.
  db_worker_node.cljs's non-repo-methods set drops from 22 to 7
  entries (the 15 removed were all e2ee/db-sync/ui-request keywords
  backed by deleted files).

- worker/handler/export.cljs's :thread-api/recompute-checksum-diagnostics
  (and worker/db/validate.cljs's backing recompute-checksum-diagnostics,
  which pulled in logseq-db-sync.checksum) compared a local checksum
  against a remote one that no longer exists -- deleted outright, not
  ported, since there's nothing left to diff against.

- worker/handler/block.cljs's :block.temp/sync-conflicts metadata
  (backed by the deleted client-op/get-all-sync-conflicts) is gone;
  its client-side consumer (a retry-upload/clear-conflicts UI in
  components/block.cljs) is dead code now and still needs a follow-up
  pass -- flagging here since worker-side cleanup alone can't remove
  it without breaking that UI's thread-api calls at runtime.

Verified: app/electron/db-worker-node/mobile all compile clean. Full
worker test suite green (db_core_test 131, db_listener_test 11,
db_worker_test 8, db_worker_node_test 59, db_worker_node_lock_test 10,
local_tx_log_test 4, undo_redo_test 29, subs_test 50, subs_loader_test
2, config_test 6 -- 0 failures across all of them). The entire
src/test/frontend/worker/sync/ test directory (7 files) and
db_sync_test.cljs/db_sync_sim_test.cljs/node_sync_test.cljs (all
testing only-deleted namespaces) are deleted wholesale rather than
ported.

Known follow-up, not yet done: the client-side (frontend.handler.*,
frontend.components.*) RTC/E2EE/db-sync UI and its dead thread-api call
sites (components/block.cljs, components/repo.cljs,
components/settings.cljs, handler/user.cljs, handler/assets.cljs,
db/restore.cljs, handler/common/developer.cljs, and the
handler/db-based/{sync,rtc-flows,rtc-background-tasks}.cljs +
handler/events/rtc.cljs + components/rtc/* + db/rtc/* files themselves)
-- a separate commit.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Darío Sevilla
2026-08-26 14:59:17 +02:00
parent c96b8a7a68
commit 565c2114be
43 changed files with 1011 additions and 21460 deletions
Download Patch File Download Diff File Expand all files Collapse all files

347
src/main/frontend/worker/local_tx_log.cljs Normal file
View File

@@ -0,0 +1,347 @@
(ns frontend.worker.local-tx-log
"SQLite-backed local tx ledger, keyed by tx-id. Originally part of the
now-deleted remote sync client (frontend.worker.sync.client-op, see Fase
8 in docs/agent-guide); the sync-specific surface (pending-upload
counts/checksums/graph-uuid/conflict tracking/asset-op queues) is gone,
but two callers still need this table: db_core.cljs's built-in-property
repair uses it as an idempotency marker (has this repo already gotten the
repair tx?) via get-local-tx-entry/upsert-local-tx-entry!, and
undo_redo.cljs reads a tx's recorded forward/inverse outliner ops back out
via history-action-ops-by-tx-id so a repair tx can be undone/redone like
any other edit."
(:require [frontend.worker.state :as worker-state]
[goog.object :as gobj]
[logseq.db.sqlite.util :as sqlite-util]))
(defonce *repo->pending-local-tx-count (atom {}))
(def ^:private sqlite-schema-ready-key "__logseq_client_ops_schema_ready_v2")
(def ^:private sqlite-mode-key "__logseq_client_ops_sqlite_mode")
(def ^:private sync-meta-table-sql
"create table if not exists sync_meta (key text primary key, value text)")
(def ^:private client-ops-table-sql
(str "create table if not exists client_ops ("
"id integer primary key autoincrement,"
"kind text not null,"
"created_at integer not null,"
"tx_id text unique,"
"pending integer not null default 0,"
"failed integer not null default 0,"
"outliner_op text,"
"undo_redo text,"
"forward_outliner_ops text,"
"inverse_outliner_ops text,"
"inferred_outliner_ops integer,"
"normalized_tx_data text,"
"reversed_tx_data text,"
"asset_uuid text,"
"asset_op text,"
"asset_t integer,"
"asset_value text"
")"))
(def ^:private pending-index-sql
"create index if not exists idx_client_ops_pending_created on client_ops(kind, pending, created_at, id)")
(defn- client-ops-store
[repo]
(worker-state/get-client-ops-conn repo))
(declare ensure-sqlite-schema!)
(defn- detect-sqlite-mode
[^js db]
(or (gobj/get db sqlite-mode-key)
(let [mode
(cond
(not db) nil
(fn? (gobj/get db "prepare"))
(try
(let [^js stmt (.prepare db "select 1")]
(try
(if (fn? (gobj/get stmt "run"))
:better-sqlite
:sqlite-wasm)
(finally
(when (fn? (gobj/get stmt "finalize"))
(.finalize stmt)))))
(catch :default _
:sqlite-wasm))
(fn? (gobj/get db "exec")) :sqlite-wasm
:else nil)]
(when mode
(try
(gobj/set db sqlite-mode-key mode)
(catch :default _
nil)))
mode)))
(defn- sqlite-db?
[conn]
(some? (detect-sqlite-mode conn)))
(defn- sqlite-store-or-throw
[repo]
(when-let [store (client-ops-store repo)]
(if (sqlite-db? store)
(do
(ensure-sqlite-schema! store)
store)
(throw (ex-info "Legacy DataScript client-op storage is unsupported. Please back up the graph and re-download it."
{:type :db-sync/legacy-client-ops-storage
:repo repo})))))
(defn- parse-uuid-str
[v]
(when (string? v)
(try
(uuid v)
(catch :default _
nil))))
(defn- kw->str
[v]
(cond
(keyword? v) (name v)
(string? v) v
:else nil))
(defn- str->kw
[v]
(when (string? v)
(keyword v)))
(defn- bool->int [v] (if v 1 0))
(defn- int->bool [v] (not (or (nil? v) (= 0 v) (= false v))))
(defn- normalize-op-entries
[ops]
(let [ops' (some-> ops seq vec)]
(cond
(nil? ops')
nil
(and (keyword? (first ops'))
(vector? (second ops')))
[ops']
:else
ops')))
(defn- sqlite-run!
[^js db sql params]
(case (detect-sqlite-mode db)
:better-sqlite
(let [^js stmt (.prepare db sql)
run-fn (gobj/get stmt "run")]
(if (seq params)
(.apply run-fn stmt (to-array params))
(.call run-fn stmt)))
:sqlite-wasm
(.exec db #js {:sql sql
:bind (into-array params)})
nil))
(defn- sqlite-rows
[^js db sql params]
(case (detect-sqlite-mode db)
:better-sqlite
(let [^js stmt (.prepare db sql)
all-fn (gobj/get stmt "all")]
(vec (if (seq params)
(.apply all-fn stmt (to-array params))
(.call all-fn stmt))))
:sqlite-wasm
(let [^js result (.exec db #js {:sql sql
:bind (into-array params)
:rowMode "object"
:returnValue "resultRows"})]
(cond
(nil? result) []
(array? result) (vec result)
(fn? (gobj/get result "toArray")) (vec (.toArray result))
:else []))
[]))
(defn- sqlite-row
[db sql params]
(first (sqlite-rows db sql params)))
(defn- sqlite-with-tx!
[^js db f]
(case (detect-sqlite-mode db)
:better-sqlite
(let [tx-fn (.transaction db (fn [] (f db)))]
(tx-fn))
:sqlite-wasm
(if (fn? (gobj/get db "transaction"))
(.transaction db (fn [tx] (f tx)))
(f db))
(f db)))
(defn ensure-sqlite-schema!
[db]
(when (sqlite-db? db)
(when-not (true? (gobj/get db sqlite-schema-ready-key))
(sqlite-with-tx!
db
(fn [tx]
(sqlite-run! tx sync-meta-table-sql [])
(sqlite-run! tx client-ops-table-sql [])
(sqlite-run! tx pending-index-sql [])))
(try
(gobj/set db sqlite-schema-ready-key true)
(catch :default _
nil)))))
(defn- sqlite-get-meta
[db k]
(some-> (sqlite-row db "select value from sync_meta where key = ?" [(name k)])
(aget "value")))
(defn- sqlite-set-meta!
[db k v]
(sqlite-run! db
(str "insert into sync_meta (key, value) values (?, ?)"
" on conflict(key) do update set value = excluded.value")
[(name k) (str v)]))
(defn get-local-tx
[repo]
(when-let [store (sqlite-store-or-throw repo)]
(when-let [result (sqlite-get-meta store :local-tx)]
(js/parseInt result 10))))
(defn update-local-tx
[repo t]
{:pre [(and (integer? t) (>= t 0))]}
(let [store (sqlite-store-or-throw repo)
prev-t (get-local-tx repo)]
(assert (some? store) repo)
(when (and prev-t (< t prev-t))
(throw (ex-info "local-tx should be monotonically increasing"
{:repo repo
:prev-t prev-t
:new-t t})))
(sqlite-set-meta! store :local-tx t)))
(defn- row->pending-local-tx
[row]
(let [tx-id (parse-uuid-str (aget row "tx_id"))]
(when tx-id
{:tx-id tx-id
:outliner-op (str->kw (aget row "outliner_op"))
:forward-outliner-ops (or (normalize-op-entries
(sqlite-util/read-transit-str (aget row "forward_outliner_ops")))
[])
:inverse-outliner-ops (or (normalize-op-entries
(sqlite-util/read-transit-str (aget row "inverse_outliner_ops")))
[])
:inferred-outliner-ops? (int->bool (aget row "inferred_outliner_ops"))
:db-sync/undo-redo (str->kw (aget row "undo_redo"))
:tx (sqlite-util/read-transit-str (aget row "normalized_tx_data"))
:reversed-tx (sqlite-util/read-transit-str (aget row "reversed_tx_data"))})))
(defn upsert-local-tx-entry!
[repo {:keys [tx-id created-at pending? failed? outliner-op undo-redo
forward-outliner-ops inverse-outliner-ops inferred-outliner-ops?
normalized-tx-data reversed-tx-data]
:or {pending? true failed? false}}]
{:pre [(some? tx-id)]}
(let [store (sqlite-store-or-throw repo)]
(assert (some? store) repo)
(let [tx-id-str (str tx-id)
existing (sqlite-row store
"select pending, created_at from client_ops where kind = 'tx' and tx_id = ?"
[tx-id-str])
should-inc-pending? (not= 1 (some-> existing (aget "pending")))
forward-outliner-ops' (or (normalize-op-entries forward-outliner-ops) [])
inverse-outliner-ops' (or (normalize-op-entries inverse-outliner-ops) [])
created-at' (or (some-> existing (aget "created_at"))
created-at
(.now js/Date))]
(sqlite-run! store
(str "insert into client_ops ("
"kind, created_at, tx_id, pending, failed, outliner_op, undo_redo, "
"forward_outliner_ops, inverse_outliner_ops, inferred_outliner_ops, "
"normalized_tx_data, reversed_tx_data"
") values ('tx', ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
" on conflict(tx_id) do update set "
"created_at = excluded.created_at,"
"pending = excluded.pending,"
"failed = excluded.failed,"
"outliner_op = excluded.outliner_op,"
"undo_redo = excluded.undo_redo,"
"forward_outliner_ops = excluded.forward_outliner_ops,"
"inverse_outliner_ops = excluded.inverse_outliner_ops,"
"inferred_outliner_ops = excluded.inferred_outliner_ops,"
"normalized_tx_data = excluded.normalized_tx_data,"
"reversed_tx_data = excluded.reversed_tx_data")
[created-at'
tx-id-str
(bool->int pending?)
(bool->int failed?)
(kw->str outliner-op)
(kw->str undo-redo)
(sqlite-util/write-transit-str forward-outliner-ops')
(sqlite-util/write-transit-str inverse-outliner-ops')
(bool->int inferred-outliner-ops?)
(sqlite-util/write-transit-str (or normalized-tx-data []))
(sqlite-util/write-transit-str (or reversed-tx-data []))])
{:tx-id tx-id
:created-at created-at'
:should-inc-pending? should-inc-pending?})))
(defn get-local-tx-entry
[repo tx-id]
(when (uuid? tx-id)
(when-let [store (sqlite-store-or-throw repo)]
(some-> (sqlite-row store
(str "select tx_id, outliner_op, undo_redo, "
"forward_outliner_ops, inverse_outliner_ops, inferred_outliner_ops, "
"normalized_tx_data, reversed_tx_data "
"from client_ops where kind = 'tx' and tx_id = ? limit 1")
[(str tx-id)])
(row->pending-local-tx)))))
(defn adjust-pending-local-tx-count!
[repo delta]
(swap! *repo->pending-local-tx-count
(fn [m]
(let [base (or (get m repo) 0)
next (max 0 (+ base delta))]
(assoc m repo next)))))
(defn history-action-ops-by-tx-id
[repo tx-id]
(when-let [entry (get-local-tx-entry repo tx-id)]
{:db-sync/forward-outliner-ops (some-> (:forward-outliner-ops entry) seq vec)
:db-sync/inverse-outliner-ops (some-> (:inverse-outliner-ops entry) seq vec)}))
(defn cleanup-finished-history-ops!
[repo protected-tx-ids]
(if-let [store (sqlite-store-or-throw repo)]
(let [protected-tx-ids (set protected-tx-ids)
tx-id-rows (sqlite-rows store
(str "select tx_id from client_ops "
"where kind = 'tx' and pending = 0 and tx_id is not null")
[])
removable-tx-ids (->> tx-id-rows
(keep (fn [row]
(let [tx-id (parse-uuid-str (aget row "tx_id"))]
(when (and (uuid? tx-id)
(not (contains? protected-tx-ids tx-id)))
tx-id))))
vec)]
(when (seq removable-tx-ids)
(doseq [tx-id removable-tx-ids]
(sqlite-run! store
"delete from client_ops where kind = 'tx' and tx_id = ?"
[(str tx-id)])))
(count removable-tx-ids))
0))