dasemu/Synapse
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial import of LogDarioSeq baseline

Browse Source 
Establishes version control for the personal Logseq DB fork before
starting the server-client architecture migration and domain features
described in docs/agent-guide and the approved implementation plan.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Darío Sevilla
2026-08-25 15:32:15 +02:00
commit 40631362b3
2016 changed files with 543898 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

43
docs/adr/0001-nodejs-db-sync-server-adapter.md Normal file
View File

@@ -0,0 +1,43 @@
# ADR 0001: Node.js DB Sync Server Adapter for Self-Hosting
Date: 2026-01-30
Status: Proposed
## Context
Logseq's DB sync currently assumes a hosted environment for the sync server. This limits users who want to self-host the server on their own infrastructure and avoid reliance on Cloudflare-specific services.
We need a portable server adapter that can run on standard Node.js runtimes (Docker, bare metal, VPS, or on-prem) while preserving the existing DB sync protocol and client behavior.
## Decision
Create a Node.js server adapter for DB sync that:
- Runs on a standard Node.js runtime (LTS) without Cloudflare dependencies.
- Implements the existing DB sync protocol and endpoints without breaking client compatibility.
- Supports self-hosting via a documented configuration (env vars, ports, storage backends).
- Allows pluggable storage and auth providers to match the current hosted behavior.
The adapter will be a peer implementation of the existing server entrypoint, sharing protocol definitions and core logic where possible.
## Options Considered
1) Continue Cloudflare-only hosting
- Pros: no new maintenance
- Cons: blocks self-hosting; vendor lock-in
2) Rewrite the sync server in a new standalone service
- Pros: full control, clean slate
- Cons: high risk, larger scope, protocol divergence risk
3) Add a Node.js adapter around the existing sync server logic (chosen)
- Pros: minimizes protocol drift; faster to ship; leverages existing logic
- Cons: requires adapter layer and some refactoring for portability
## Consequences
- Additional maintenance surface for Node.js runtime compatibility.
- Requires careful separation of platform-specific code from shared sync logic.
- Enables self-hosting and reduces dependence on Cloudflare.
- May require new CI coverage for Node.js adapter builds and basic integration tests.
## Follow-up Work
- Identify Cloudflare-specific APIs and replace with portable interfaces.
- Define storage and auth provider interfaces for the adapter.
- Add documentation for self-hosting setup and deployment examples.
- Add minimal integration tests to validate protocol compatibility.

113
docs/adr/0002-undo-redo-worker-sync.md Normal file
View File

@@ -0,0 +1,113 @@
# ADR 0002: Undo/Redo scoped to local changes in worker sync
Date: 2026-01-28
Status: Accepted
## Context
The worker-based db-sync flow applies remote updates directly to the local DB.
Undo/redo must remain usable for the current user without replaying or reverting
server changes. The system already supports tx-meta flags (e.g., :local-tx?,
:gen-undo-ops?, :undo?, :redo?) and has conflict checks in undo logic.
We must ensure undo/redo never attempts a transaction that would result in
invalid client or server data. We also want a simple, sane model that avoids
tracking per-remote update history.
Constraints and goals:
- No server echo of the user's own txs back to the same client.
- Best effort undo: if conflicts make an undo unsafe, skip it and keep data
valid (never force invalid data).
- Remote updates must not create undo history.
- Local updates must create undo history as before.
## Decision
1) Only local txs may generate undo/redo ops. A tx is local if and only if
tx-meta contains :local-tx? true.
2) All remote/apply/rebase/import paths must set :local-tx? false and
:gen-undo-ops? false.
3) Undo/redo will be "best effort": if reversing a tx would violate invariants
(e.g., moved block, deleted parent, remaining children), the undo op is
dropped and history is cleared for that repo to prevent invalid data.
This keeps undo/redo stable and ensures remote updates never appear in the
user's undo stack.
## Architecture
### Source of truth for origin
- :local-tx? is the sole origin flag.
- Local changes (UI/outliner ops) attach :local-tx? true at the time of
submission to the worker.
- Remote changes (db-sync apply-remote, rtc remote update, snapshot import) set
:local-tx? false and :gen-undo-ops? false.
### Undo/redo recording
- Undo history is recorded only when tx-meta has :local-tx? true.
- Additional existing gates remain: :outliner-op is present, :gen-undo-ops?
is not false, and :create-today-journal? is not true.
- Redo stack is cleared on any new local undo-recorded op (existing behavior).
### Undo/redo execution safety
- Reverse datoms are computed from original tx-data.
- Conflict detection remains in place (moved blocks, deleted parents, children
still exist). These are treated as safe failures for best-effort undo.
- If a conflict is detected or reverse-tx is empty, the undo op is dropped and
history cleared for that repo to avoid invalid transacts.
- Undo/redo execution uses tx-meta flags :undo?/:redo? and :gen-undo-ops? false
to avoid recursive undo generation.
## Consequences
- Undo/redo only affects user-originated changes, never server updates.
- In rare cases, undo may be skipped after a remote conflict, but the database
remains valid.
- The model stays simple: no remote history tracking or per-entity versioning.
## Plan
1) Confirm the origin flag path
- Verify all local transact entry points set :local-tx? true
(e.g., frontend.db.transact/transact, apply-outliner-ops).
- Verify all remote/apply/rebase/import paths set :local-tx? false and
:gen-undo-ops? false (db-sync apply-remote, rtc remote updates,
snapshot import, db restore).
2) Gate undo generation on :local-tx?
- Update frontend.undo-redo/gen-undo-ops! to require :local-tx? true in
tx-meta in addition to existing checks.
3) Enforce best-effort safety
- On conflict (moved block, deleted parent, remaining children) or on
missing reverse tx data, drop the undo op and clear history for repo.
4) Documentation
- Update relevant internal docs or comments near undo/redo about the origin
flag and best-effort semantics.
## Test Scenarios (Manual)
1) Local change only
- Create a block, undo, redo.
- Expected: undo/redo works and affects only local changes.
2) Remote change only
- Receive a server update that modifies a block.
- Expected: undo stack is unchanged; undo does not revert remote update.
3) Local change after remote update
- Remote updates a block; user edits same block; undo.
- Expected: undo reverts only the user's edit; remote update remains.
4) Remote delete of parent before undo
- User creates child; remote deletes parent; user tries undo.
- Expected: undo is skipped safely; history cleared; no invalid data.
5) Remote move before undo
- User moves block; remote moves or deletes target; user tries undo.
- Expected: undo is skipped safely; history cleared; no invalid data.
6) Mixed local ops and remote tx batch
- Interleave local edits and remote sync.
- Expected: only local edits appear in undo stack; undo never produces
invalid data.
## Open Questions
- Do we need a user-visible notification when undo is skipped due to conflicts,
or is silent failure acceptable?
Silent failure acceptable.

87
docs/adr/0003-optional-sync-graph-encryption.md Normal file
View File

@@ -0,0 +1,87 @@
# ADR 0003: Optional Encryption When Creating a Sync Graph
Date: 2026-02-11
Status: Proposed
## Context
Sync graph creation currently assumes encrypted graph data. This is a good default
for privacy, but it is not always the right choice:
- Self-hosted users may run trusted infrastructure and prefer plaintext storage.
- Some users need easier integration with external tools that read graph data
directly from storage or APIs.
- Encryption/decryption adds CPU and complexity during sync operations.
We need to support both privacy-first and integration-first workflows without
breaking compatibility for existing encrypted graphs.
## Decision
Add an explicit graph-level option when creating a sync graph:
- `Encrypt graph data` (boolean)
Behavior:
1) Default remains encrypted for safety.
2) Users can disable encryption at graph creation time.
3) The selected mode is stored as graph metadata and treated as immutable for
that graph after creation.
There's already `:logseq.kv/graph-rtc-e2ee?`.
4) Existing encrypted graphs remain unchanged.
When encryption is disabled:
- Client does not generate/import graph encryption keys for sync payloads.
- Sync payloads and assets are sent as plaintext over TLS transport.
- Client skips encrypt/decrypt steps in DB and asset sync pipelines.
When encryption is enabled:
- Current E2EE behavior remains unchanged.
## Options Considered
1) Keep encryption mandatory
- Pros: strongest default privacy, simpler policy
- Cons: blocks valid self-hosted and interoperability use cases
2) Disable encryption by default
- Pros: easier integrations out of the box
- Cons: unsafe default for most users
3) Make encryption selectable at graph creation (chosen)
- Pros: keeps secure default while enabling self-hosted/integration scenarios
- Cons: introduces dual execution paths and UI/UX complexity
## Consequences
- Positive:
- Better support for self-hosted deployments.
- Easier interoperability with external tooling.
- Lower client CPU overhead for non-encrypted graphs.
- Negative:
- More branching in sync and asset code paths.
- Higher test matrix (encrypted vs non-encrypted).
- Users must understand security tradeoffs.
## Follow-up Work
1) API and protocol
- Add graph creation flag for encryption mode.
- Persist encryption mode in graph metadata and return it in graph info.
2) Client graph creation UI
- Add `Encrypt graph data` option with warning/help text.
- Default ON, with clear explanation of tradeoff.
3) DB sync and asset sync behavior
- Gate encryption and key-management logic on graph mode.
- Ensure non-encrypted graphs do not call key generation/import paths.
4) Migration and compatibility
- Keep existing graph behavior unchanged.
- Do not auto-migrate graph encryption mode.
5) Testing
- Add/create-path tests for both modes.
- Add sync + asset upload/download tests for encrypted and non-encrypted graphs.
- Add regression tests for mixed environments (desktop/web/self-hosted).
## Open Questions
- Should self-hosted servers be able to enforce a default or only allow one mode?
Defaults to encryption mode.
- Do we need a future migration flow to move a graph between encrypted and
non-encrypted modes?
Probably, let's leave it for future.

66
docs/adr/0004-sync-tx-entry-outliner-op.md Normal file
View File

@@ -0,0 +1,66 @@
# ADR 0004: Sync batches are ordered tx entries with persisted `:outliner-op`
Date: 2026-03-16
Status: Accepted
## Context
DB sync previously treated a client upload batch as one transit-encoded tx blob
and the client pull path could flatten remote tx entries into one local apply.
That made sync failures hard to debug because:
- a failed sync apply did not clearly identify which tx entry failed
- the server did not persist the original `:outliner-op`
- the client carried cross-batch tempid repair logic to make split txs work
We want sync errors to point to one specific tx entry and one specific
operation intent.
We also want the server and client to follow the same mental model:
- a batch is an ordered list of tx entries
- each tx entry is applied one by one
- each tx entry may carry its own `:outliner-op`
## Decision
1) `tx/batch` now sends a list of tx entries instead of one encoded tx blob.
2) Each tx entry contains:
- `:tx`
- optional `:outliner-op`
3) The server applies tx entries one by one, in order.
4) The server persists `:outliner-op` in `tx_log` and returns it in `pull/ok`.
5) The client applies pulled tx entries one by one and includes the returned
`:outliner-op` in tx-meta for remote apply.
6) Cross-batch tempid reuse is unsupported by contract. A tempid must be fully
resolved within a single tx entry.
## Storage and Protocol Notes
- The persisted sync tx log is the per-graph SQLite store initialized in
`logseq.db-sync.storage`, not the Cloudflare D1 graph index schema.
- Because of that, `outliner_op` is added via runtime SQLite schema setup for
the per-graph store, not a D1 migration under `worker/migrations/`.
- `pull/ok` entries now have shape:
- `{:t ... :tx ... :outliner-op ...}`
- `tx/batch` entries now have shape:
- `{:tx ... :outliner-op ...}`
## Consequences
- Positive:
- Sync failures can be traced to one tx entry and its `:outliner-op`.
- Client and server now share the same ordered per-entry apply model.
- The protocol is simpler than carrying cross-batch tempid repair logic.
- Negative:
- Older assumptions that tempids can be reused across tx entries are no
longer valid.
- The wire format changed for `tx/batch` and `pull/ok`.
## Follow-up Constraints
- New sync producers must keep each tx entry self-contained.
- If a future feature needs cross-entry references again, it must be introduced
explicitly as protocol behavior rather than repaired implicitly by the
client.
## Verification
- Updated frontend worker db-sync tests.
- Updated `deps/db-sync` storage, handler, and node-adapter tests.
- Passed:
- `bb dev:test -v frontend.worker.db-sync-test`
- `pnpm test` in `deps/db-sync`
- `bb dev:lint-and-test`

49
docs/adr/0005-atomic-outliner-batches-via-temp-conn.md Normal file
View File

@@ -0,0 +1,49 @@
# ADR 0005: Atomic Outliner Batches via Temp Conn
Date: 2026-03-16
Status: Accepted
## Context
Outliner batch operations previously used a "batch-tx mode" that wrote each
inner transaction to the real conn immediately, then reconstructed one logical
batch in the db listener by collecting tx reports and emitting a synthetic
batch exit event.
That approach had two problems:
- it was not truly atomic because intermediate writes were visible on the real
conn before the batch finished
- batch flattening in `transact-with-temp-conn!` did not normalize logical
entity identity the same way sync already does, which made some multi-step
batches generate invalid final tx data
The failing db-sync sim showed this clearly: multi-step outliner batches could
leave the final merged tx with conflicting entity ids for the same block uuid.
## Decision
1. Outliner batch execution now uses `logseq.db/transact-with-temp-conn!`.
2. A batch writes to a temp conn for the full duration of the batch and commits
once to the real conn at the end.
3. Nested outliner batches reuse the current temp conn instead of opening a
second batch.
4. The final tx emitted from `transact-with-temp-conn!` is normalized with the
existing `logseq.db.common.normalize/normalize-tx-data` helper before it is
committed to the real conn.
5. Worker db listeners no longer reconstruct batch tx reports from
`:batch-tx/*` metadata, because the real conn now receives one final tx.
## Consequences
- Positive:
- Outliner batches are actually atomic from the real conn's perspective.
- Listener code is simpler because it no longer has to stitch together a
logical batch from multiple intermediate tx reports.
- Temp-id and lookup-id conflicts are handled by the existing tx
normalization path instead of a batch-specific merge.
- Negative:
- Batch execution now depends on temp-conn context instead of tx-meta flags,
so any future code that expects `:batch-tx/*` metadata must not be readded.
## Verification
- Passed targeted regression:
- `bb dev:test -v frontend.worker.db-sync-sim-test/three-clients-single-repo-sim-test`
- Passed focused batch helper coverage:
- `bb dev:test -v logseq.db-test/test-transact-with-temp-conn!`

73
docs/adr/0006-sync-entity-checksum.md Normal file
View File

@@ -0,0 +1,73 @@
# ADR 0006: Sync Entity Checksum for Server/Client Parity
Date: 2026-03-16
Status: Accepted
## Context
DB sync currently treats matching `t` values as the main convergence signal.
That is necessary, but it is not sufficient to guarantee that the client and
server represent the same block graph.
We have seen cases where temp-id handling and rebase bugs produced logically
divergent entities while `t` still advanced normally. In those cases, the
server and client could believe they were in sync even though some entities had
different content or references.
We need a cheap parity check that works on both the frontend worker and the
Cloudflare Worker, and the server must stay within a 128 MB memory limit.
## Decision
1. Add a shared DB sync checksum over block entities only.
2. The compared attrs depend on whether the graph is E2EE:
- Unencrypted graphs compare:
- `:block/uuid`
- `:block/title`
- `:block/name`
- `:block/parent`
- `:block/page`
- E2EE graphs compare:
- `:block/uuid`
- `:block/parent`
- `:block/page`
3. Reference attrs are normalized to referenced block UUID strings before
hashing, so client/server entity ids do not affect the result.
4. Runtime checksum is maintained incrementally from changed entities in tx
reports and persisted locally:
- client stores it in client-op metadata
- server stores it in `sync_meta`
5. A full recompute checksum path is kept for simulation and test verification.
6. The server includes the checksum in state-bearing sync responses:
- `hello`
- `pull/ok`
- `tx/batch/ok`
7. The client compares the remote checksum against its local checksum only when
sync is expected to have converged:
- local `t` equals remote `t`
- no pending local txs
- no inflight upload txs
8. On mismatch, the client fails fast with `:db-sync/checksum-mismatch` instead
of silently continuing with a divergent graph.
## Consequences
- Positive:
- Matching `t` is now backed by entity-level parity over the core sync attrs.
- Runtime checksum is fast and memory-light on both client and server because
it does not rescan the whole graph on each compare.
- Divergence is detected immediately at the sync boundary instead of surfacing
later as harder-to-debug invalid entities.
- Negative:
- The checksum is a fast additive hash, not a cryptographic digest.
- E2EE graphs intentionally do not compare encrypted title/name fields.
- It intentionally ignores attrs outside the sync-invariant set above.
- Mismatch handling is fail-fast in this change; it does not automatically
repair a divergent graph.
## Verification
- Frontend worker coverage:
- `bb dev:test -v frontend.worker.db-sync-test/hello-checksum-mismatch-fails-fast-test`
- `bb dev:test -v frontend.worker.db-sync-test/pull-ok-out-of-order-stale-response-is-ignored-test`
- Sim parity coverage:
- `bb dev:test -v frontend.worker.db-sync-sim-test/two-clients-cut-paste-random-sim-test`
- Sync compatibility coverage:
- `bb dev:test -v frontend.worker.db-sync-test/pending-txs-rewrite-old-string-tempids-test`
- `bb dev:test -v frontend.worker.db-sync-test/rebase-order-fix-for-new-blocks-does-not-keep-string-tempids-test`

58
docs/adr/0007-db-sync-recycle-simplification.md Normal file
View File

@@ -0,0 +1,58 @@
# ADR 0007: Simplify Client DB-Sync for Recycle Semantics
Date: 2026-03-17
Status: Accepted
## Context
Client DB sync still contains logic shaped around hard delete.
That logic tracks deleted UUIDs, computes deleted subtrees, and runs a second
cleanup pass in temp connections to remove nodes after remote apply and local
rebase.
The recycle refactor changed deletion semantics.
Deleted pages and blocks now remain in the graph with stable UUIDs and are
hidden by placement and metadata instead of disappearing.
That means the deleted-context and delete-nodes cleanup flow is no longer the
right model for client sync.
It adds complexity, creates stale test assumptions around `db/retractEntity`,
and makes the rebase path harder to reason about.
## Decision
1. Client DB sync will stop treating deletion as entity disappearance.
2. The rebase/apply pipeline will operate on ordinary tx streams plus current DB
state.
3. `sanitize-tx-data` will no longer accept or derive deleted subtree state.
4. The extra temp-conn cleanup pass that deletes locally or remotely deleted
nodes will be removed.
5. Sync sanitization will keep only the behaviors that still matter:
- broken temp-id cleanup
- missing/ref-invalid datom cleanup
- remote/local conflict dropping
- page consistency fixes
- duplicate order fixes
6. Recycled targets are treated as invalid active refs by current DB state,
rather than through a separate deleted-context channel.
## Consequences
- Positive:
- The client sync implementation becomes shorter and easier to reason about.
- Delete and restore semantics stay consistent across local UI, undo/redo,
sync, and rebase.
- Tests can assert recycle behavior directly instead of hard-delete
side-effects.
- Negative:
- Old tests and assumptions built on `db/retractEntity` delete behavior must
be updated.
- Minimal stripped-down test DBs need recycle-aware helpers so they do not
count recycled nodes as active graph nodes.
## Verification
- DB sync unit coverage:
- `bb dev:test -v frontend.worker.db-sync-test`
- DB sync simulation coverage:
- `bb dev:test -v frontend.worker.db-sync-sim-test`
- Undo/redo compatibility coverage:
- `bb dev:test -v frontend.undo-redo-test`
- Recycle-aware long outliner helper coverage:
- `bb dev:test -e long -v frontend.modules.outliner.core-test/test-batch-transact`

73
docs/adr/0008-db-sync-simulation-and-replay-coverage.md Normal file
View File

@@ -0,0 +1,73 @@
# ADR 0008: Strengthen DB-Sync Simulation and Replay Coverage
Date: 2026-03-17
Status: Accepted
## Context
Recent DB-sync bugs have repeatedly escaped existing unit and simulation tests.
The failures were only exposed by real user sessions and browser logs, usually
after long offline editing, undo/redo, delete/recreate, recycle, and
sequential reconnect flows.
The current test suite has good focused coverage for many individual sync
helpers, but it still misses several classes of real-world behavior:
- editor-style multi-step writes such as `insert empty block -> save title`
- long-lived pending local tx queues
- undo/redo stacks that survive many sync rounds
- sequential reconnect flows where one client uploads first and another
reconnects later
- replay of real bad `local-txs` and `remote-txs` captured from logs
That gap makes DB-sync regressions too dependent on manual reproduction and
post-hoc log analysis.
## Decision
1. Treat DB-sync simulation as a first-class regression surface, not just a
long-running smoke test.
2. Keep adding deterministic repro tests for every real sync bug that reaches
logs or manual QA.
3. Expand offline multi-client sims to cover:
- both reconnect orders
- repeated offline/online cycles
- long undo/redo stacks
- heavier weighting for `delete-blocks`, `move-blocks`,
`cut-paste-block-with-child`, `undo`, and `redo`
4. Add invariant checks during simulation, not only after final convergence:
- no malformed pending tx items
- no malformed reversed tx items
- no invalid lookup-only reverse txs
- no DB structural issues such as missing parent/page or page mismatch
5. Introduce replay-oriented tests that can execute compact fixtures extracted
from real logs:
- persisted `local-txs`
- remote tx batches
- reconnect order
6. Prefer editor-faithful simulation helpers over simplified synthetic ops for
block creation and editing.
7. Keep a small corpus of fixed seeds and exact repro flows in the repository,
so future regressions are rerun automatically in CI and local test runs.
## Consequences
- Positive:
- Real sync bugs can be captured once and kept as permanent regressions.
- Multi-client offline behavior gets closer to real user workflows.
- Broken pending/reverse tx shapes fail earlier and closer to the source.
- Investigation time drops because logs can be turned into executable tests.
- Negative:
- Long-running DB-sync tests become more expensive.
- Some simulations may still approximate, rather than exactly match, UI
timing and browser worker behavior.
- The test suite will need ongoing curation so fixed seeds and replay
fixtures stay useful instead of becoming noisy.
## Verification
- Exact repro simulation coverage:
- `bb dev:test -v frontend.worker.db-sync-sim-test/two-clients-offline-delete-vs-children-undo-redo-repro-test`
- Aggressive offline random coverage:
- `bb dev:test -v frontend.worker.db-sync-sim-test/two-clients-both-offline-random-undo-redo-sequential-reconnect-sim-test`
- Reverse/pending tx shape coverage:
- `bb dev:test -v frontend.worker.db-sync-test/reverse-local-txs-skips-invalid-reverse-step-test`
- `bb dev:test -v frontend.worker.db-sync-test/reverse-local-txs-skips-missing-lookup-entity-step-test`
- `bb dev:test -v frontend.worker.db-sync-test/replace-string-block-tempids-rewrites-retract-entity-string-uuid-test`
- Undo/redo validity coverage:
- `bb dev:test -v frontend.undo-redo-test/undo-validation-rejects-invalid-recycle-restore-tx-test`

69
docs/adr/0009-hybrid-schema-deletion.md Normal file
View File

@@ -0,0 +1,69 @@
# ADR 0009: Hybrid Delete Semantics for Schema Entities
Date: 2026-03-18
Status: Accepted
## Context
`delete-blocks` and page deletion now recycle entities instead of hard deleting
them.
That works well for ordinary content because users can restore deleted blocks
and pages from Recycle, and active UI/search can treat recycled content as
hidden state.
It does not work as well for DB-schema entities.
Tag pages, property pages, and closed-value blocks continue to exist after
deletion and can leak into active semantics such as:
- class/property enumeration
- closed-value lookup and scoped choices
- bidirectional/property-derived views
- sync repair logic when schema roots disappear but dependents still reference
them
At the same time, sync still needs to remain robust when `:db/retractEntity`
appears again for this subset of entities and for their descendants.
## Decision
1. Adopt hybrid delete semantics:
- recycle ordinary blocks and ordinary content pages
- hard retract tag/class pages, property pages, and closed-value blocks
2. When deleting a schema root page, retract the schema page tree rather than
moving it to Recycle.
3. When deleting a closed-value block, retract the choice entity directly.
4. Keep raw entity access unchanged; deleted schema entities should disappear by
DB state, not by UI-only filtering.
5. Keep recycle-aware behavior for content entities:
- deleted-state page route for recycled content
- deleted styling for recycled references/embeds
- search/index filtering for recycled content and descendants
6. Make sync robust for missing content locations:
- if a content node loses `:block/parent` or `:block/page` during sync
application, move it to Recycle rather than leaving it structurally
invalid
7. Preserve valid `:db/retractEntity` handling in sync sanitize/rebase/apply for
schema deletions.
## Consequences
- Positive:
- Schema entities return to true disappearance semantics, which matches how
most schema helpers already reason about deletion.
- Content keeps recycle/restore UX.
- Sync has a deterministic repair rule for orphaned content blocks.
- Negative:
- Delete semantics are now mixed by entity kind, so the rule must remain
centralized in the outliner/domain layer.
- Some old recycled schema entities may still need active-use filtering until
users delete them under the new semantics.
- Sync logic is slightly more complex because it must accept both recycle and
hard-retract flows.
## Verification
- Model/helper coverage:
- `bb dev:test -v frontend.db.db-based-model-test/get-all-classes-filters-recycled-test`
- `bb dev:test -v frontend.db.db-based-model-test/get-all-properties-filters-recycled-test`
- Sync repair coverage:
- `bb dev:test -v frontend.worker.db-sync-test/missing-parent-after-remote-retract-moves-child-to-recycle-test`
- Additional targeted coverage to keep or extend:
- page/property/closed-value delete semantics
- bidirectional helper behavior with deleted or recycled entities
- search index invalidation for recycled content

147
docs/adr/0010-op-driven-client-rebase.md Normal file
View File

@@ -0,0 +1,147 @@
# ADR 0010: Canonical Op-Driven Client Rebase With Legacy Tx Surgery Isolation
Date: 2026-03-19
Status: Accepted
## Context
Client sync rebase currently relies on custom tx-data surgery to keep pending
local changes alive after remote txs are applied.
That approach has several problems:
- the logic is hard to reason about because it edits datoms instead of replaying
user intent
- it does not preserve intent well when a local change originated from a higher
level outliner action
- the helper set keeps growing with special cases for missing refs, structural
conflicts, and deleted entities
- the outliner-op surface is too large to replay directly without first
reducing it
At the same time, not every outliner op needs to remain a first-class replay
operation.
Many ops can be safely represented by `:transact` when their tx-data is already
self-contained and does not depend on rerunning outliner logic.
We also have existing persisted pending rows that only store tx-data and reverse
tx-data.
Those rows still need a compatibility path, but that path should not define the
new rebase architecture.
## Decision
1. New pending local tx rows will persist semantic `:outliner-ops` in addition
to their existing tx-data payload.
2. Client rebase will become op-driven for all new pending rows:
- reverse local txs
- apply remote txs
- transform or drop stored local ops against the post-remote temp db
- replay the surviving ops to regenerate rebased tx-data
3. Reduce the replay-visible outliner-op surface to a small canonical set:
- `:insert-blocks`
- `:save-block`
- `:move-blocks`
- `:delete-blocks`
- `:transact`
4. Normalize higher-level ops into that canonical set before persistence.
Examples:
- `:indent-outdent-blocks` becomes `:move-blocks`
- `:move-blocks-up-down` becomes `:move-blocks`
- `:rename-page` becomes `:save-block`
5. Introduce an explicit safe-`:transact` classifier:
- if replaying tx-data directly is sufficient and does not require rerunning
outliner logic, persist the op as canonical `:transact`
- otherwise keep it as one of the canonical outliner replay ops
- expected canonical `:transact` cases include direct tx replay actions such
as undo, redo, import replay, reaction toggles, and property-value updates
6. Treat undo/redo as canonical `:transact` actions.
- persist the exact tx-data that the successful undo or redo applied
- keep the action atomic as one pending tx row
- do not reconstruct the original higher-level outliner ops that were
undone or redone
7. Treat `:batch-import-edn` as canonical `:transact` after successful local
execution.
- persist the exact tx-data that the import applied
- do not rerun import expansion during rebase
8. Keep the following op kinds as replay-visible semantic ops because they must
be reevaluated against current DB state:
- `:save-block`
- `:insert-blocks`
- `:move-blocks`
- `:delete-blocks`
- `:create-page`
- `:delete-page`
- `:upsert-property`
9. Stop using raw tx-data surgery in the normal rebase path for new rows.
10. Move the current tx-data surgery helpers into a dedicated legacy namespace.
That legacy namespace is only for compatibility handling of old persisted
pending rows that do not have stored `:outliner-ops`.
11. Add explicit owned-block filtering in the new op-driven rebase path.
Initially cover:
- reaction blocks owned by `:logseq.property.reaction/target`
- property history blocks owned by `:logseq.property.history/block`
- property history blocks whose effective owner disappears through deleted
`:logseq.property.history/ref-value`
12. If a local op creates or updates one of those owned blocks and the owning
block was deleted remotely, drop that op.
13. Treat each pending tx row as one user action and keep it atomic during
rebase.
14. If any op in a pending tx becomes invalid during rebase, drop the whole
pending tx rather than keeping a partial replay of that user action.
15. Keep this refactor client-only for now.
The sync wire format and server tx log remain unchanged.
## Consequences
- Positive:
- Rebase reasons about user intent instead of datom accidents.
- The number of op kinds that rebase must understand becomes much smaller.
- Safe direct tx replay remains available through canonical `:transact`
without forcing every operation through outliner code.
- Undo/redo stays aligned with its real intent: replay the exact applied tx,
not rerun a reconstructed higher-level command.
- Import replay stays aligned with its real intent: preserve the exact local
import result rather than recomputing import expansion later.
- Legacy compatibility is isolated instead of contaminating the new design.
- Owned-block cleanup becomes an explicit semantic rule rather than another
tx-data patch.
- Rebase behavior stays aligned with the mental model that one pending tx is
one user action that either survives or is discarded as a whole.
- Negative:
- We must maintain a canonicalization layer from original outliner ops to the
reduced replay set.
- Some existing ops need careful classification to decide whether they are
safe `:transact` or must stay true outliner replays.
- For a transition period, the codebase will contain both the new rebase path
and the isolated legacy compatibility path.
- If one invalid op is grouped together with otherwise valid work in the same
pending tx, the whole user action will be lost during rebase.
## Follow-up Constraints
- New pending tx producers must persist canonical `:outliner-ops`.
- New pending tx producers must preserve user-action boundaries because rebase
will treat each persisted tx row atomically.
- Canonicalization should happen when persisting local pending txs, not lazily
during rebase.
- Undo/redo producers should persist canonical `:transact` actions using the
exact tx-data they applied.
- Import producers should persist canonical `:transact` actions using the exact
tx-data they applied.
- The main sync apply namespace should not call legacy tx-surgery helpers for
new pending rows.
- The legacy namespace should be clearly named and easy to delete once old
pending-row compatibility is no longer needed.
## Verification
- Add or update frontend worker db-sync coverage for:
- persistence of canonical `:outliner-ops`
- canonical reduction of `:indent-outdent-blocks` and
`:move-blocks-up-down` into `:move-blocks`
- safe `:transact` replay versus true outliner replay classification
- undo/redo persistence as atomic canonical `:transact` actions
- `:batch-import-edn` persistence as atomic canonical `:transact`
- `:rename-page` canonicalization to `:save-block`
- op-driven rebase preserving pending tx boundaries
- dropping owned reaction/history ops when their owner was deleted remotely
- dropping the whole pending tx when any op in that user action becomes
invalid
- routing legacy pending rows without stored ops through the legacy namespace
- Expected targeted command:
- `bb dev:test -v frontend.worker.db-sync-test`

378
docs/adr/0011-undo-redo-semantic-inverse-op-persistence.md Normal file
View File

@@ -0,0 +1,378 @@
# ADR 0011: Undo/Redo Semantic Inverse-Op Persistence and Local-Op Referenced UI History
Date: 2026-03-21
Status: Proposed
## Context
ADR 0010 deliberately classified undo and redo as canonical `:transact` actions.
That choice kept pending-row persistence simple, but it also preserved undo and redo as raw datom replay rather than preserving their intent.
That tradeoff is now the main weakness in the sync rebase model.
During client rebase, pending local actions are reversed, remote txs are applied, and then pending local actions are replayed.
That architecture works well when the persisted action is a semantic op such as `:save-block`, `:insert-blocks`, or `:move-blocks`.
It works poorly when the persisted action is a raw undo or redo tx payload because the replay logic can no longer distinguish commutative remote updates from true semantic conflicts.
The current undo/redo design also duplicates DB history across two places.
The main thread owns the undo and redo stacks because it stores UI state such as route state, editor cursor, and focus context.
At the same time, the worker already persists local DB actions in client-ops storage with a stable `tx-id`.
Those two histories are related but not unified.
As a result, sync rebase, undo/redo, and pending local action persistence can drift apart.
The user action identity is already represented in the worker by `:db-sync/tx-id`, but the UI undo stack still stores the DB tx payload itself instead of referencing that persisted action.
This ADR defines a plan to make the client-op row the source of truth for DB history while keeping the main thread as the source of truth for UI state.
Constraints:
- The main thread must continue to own UI-only undo state.
- The sync wire protocol should remain unchanged.
- Existing persisted pending rows without semantic metadata are intentionally out of scope for this change.
- Rebase should continue to preserve one logical pending row as one user action.
- The design must preserve redo, not just undo.
Related ADRs:
- Builds on ADR 0002.
- Builds on ADR 0010.
- Supersedes ADR 0010 decision 6, which treated undo/redo as canonical `:transact`.
## Problem Summary
The current model has three tightly coupled problems.
First, undo and redo are persisted as raw tx replay.
That makes rebase unable to reason about their semantics.
Second, the undo stack on the main thread stores DB payloads directly instead of referencing the persisted local action row in worker storage.
That means the UI history and worker history are not guaranteed to stay aligned.
Third, the current worker pending-row schema persists forward tx data and reverse tx data, but it does not persist both semantic forward ops and semantic inverse ops as first-class action metadata.
That prevents robust rebase and robust redo from using the same action identity.
The result is that a benign remote update such as a title edit can still force undo/redo to fall back to raw replay, which is unnecessarily lossy and much harder to classify correctly.
## Decision
1. The worker client-op row identified by `:db-sync/tx-id` becomes the source of truth for DB undo/redo history.
2. The main-thread undo/redo stacks will store UI state plus a reference to that worker action row, not the DB tx payload itself.
3. New client-op rows will persist both forward semantic outliner ops and inverse semantic outliner ops.
4. Redo will replay the persisted forward semantic ops for the referenced action row.
5. Undo will replay the persisted inverse semantic ops for the referenced action row.
6. New rebase logic will use semantic forward or inverse ops for new undo/redo-backed action rows instead of raw datom replay.
7. Existing `normalized-tx-data` and `reversed-tx-data` fields remain temporarily for debugging, validation, and controlled cleanup, but they will not define replay behavior for the new undo/redo architecture.
8. The logical `tx-id` remains stable across rebase rewrites so that main-thread stack entries do not break when the worker rewrites the row contents.
9. Undo/redo rows that cannot be represented with semantic inverse ops are unsupported until a semantic representation is added explicitly.
## Target Architecture
The architecture separates UI history from DB history while keeping both linked through one stable action identifier.
```text
+-------------------+ tx-id +----------------------------+
| Main thread | -----------------------> | Worker client-ops storage |
| undo/redo stack | | one row per logical action |
| | <----------------------- | stable tx-id |
| UI state | action result | forward semantic ops |
| editor cursor | | inverse semantic ops |
| route/sidebar | | normalized tx data |
+-------------------+ | reverse tx data (legacy) |
+----------------------------+
|
v
+----------------------------+
| Sync rebase + upload |
| replay by semantic ops |
+----------------------------+
```
The main thread continues to own UI-only history because the worker should not become responsible for route or editor selection state.
The worker becomes responsible for DB action identity and replay semantics because that is the same place that already owns pending local action persistence, sync upload, and rebase.
## Data Model Changes
The client-op row needs to distinguish between the identity of a logical action and the concrete tx-data that happened to be generated on the current device revision.
The row should keep the existing logical `tx-id`.
The row should add explicit semantic operation fields for both directions.
Recommended schema additions:
| Field | Purpose |
| --- | --- |
| `:db-sync/forward-outliner-ops` | Canonical semantic ops that reapply the action. |
| `:db-sync/inverse-outliner-ops` | Canonical semantic ops that undo the action. |
| `:db-sync/history-kind` | Distinguishes regular action rows from specialized worker history rows if needed. |
| `:db-sync/source-tx-id` | Optional link from an undo/redo execution row back to the original action row during migration. |
| `:db-sync/semantic-persistence-version` | Marks rows that are safe for semantic replay. |
The existing fields should remain during migration:
| Existing field | Migration role |
| --- | --- |
| `:db-sync/normalized-tx-data` | Validation reference and emergency cleanup aid. |
| `:db-sync/reversed-tx-data` | Local safety checks and controlled cleanup aid. |
| `:db-sync/outliner-op` | Existing summary/debug field. |
| `:db-sync/outliner-ops` | Can be folded into `forward-outliner-ops` once migration is complete. |
The main-thread undo stack entry shape should change accordingly.
The DB portion of a stack entry should become a small reference object instead of an embedded tx payload.
Recommended main-thread DB history payload:
| Field | Purpose |
| --- | --- |
| `:local-op-tx-id` | Stable reference to worker client-op row. |
| `:history-direction` | `:forward` in undo stack and `:inverse` in redo stack if needed for UI bookkeeping. |
| `:history-version` | Allows invalidating stale stack entries across releases. |
The main-thread stack entry should continue to store editor cursor and UI route state as it does today.
## Semantic Persistence Rules
The plan depends on persisting semantic inverse ops instead of reconstructing them later from raw datoms.
That means the inverse ops must be created at the time a local action is first recorded into undo history.
The worker should never have to guess the inverse of an already-lost user intent from only tx-data if the action was produced by a known canonical outliner op.
The persistence rules are:
1. A regular local action row persists canonical forward ops.
2. The undo stack entry points at that action row by `tx-id`.
3. Undo execution uses the inverse semantic ops stored on the referenced row.
4. Redo execution uses the forward semantic ops stored on the referenced row.
5. If a row does not have semantic inverse ops, it must not enter the semantic rebase path.
## Canonical Op Surface
This ADR does not require every outliner op to become replay-visible immediately.
It does require the canonical replay-visible surface for undo/redo to be explicit and versioned.
Recommended canonical surface for forward and inverse persistence:
- `:save-block`
- `:insert-blocks`
- `:move-blocks`
- `:delete-blocks`
- `:set-block-property`
- `:remove-block-property`
- `:batch-set-property`
- `:batch-remove-property`
- `:delete-property-value`
- `:batch-delete-property-value`
- `:create-property-text-block`
- `:upsert-closed-value`
- `:delete-closed-value`
- `:add-existing-values-to-closed-values`
- `:create-page`
- `:delete-page`
- `:rename-page`
If an action cannot be expressed in that surface with a safe inverse, it should remain unsupported until a semantic representation is added deliberately.
That is preferable to reclassifying it as safe raw replay by accident.
## Inverse Op Generation Strategy
Inverse semantic ops should be created from the original action metadata and the pre-action DB state, not by reverse-engineering datoms after the fact.
The current main-thread undo history already retains the original tx meta, which includes `:outliner-ops` for many actions.
That existing signal should be preserved and extended rather than discarded.
The generation strategy should be:
1. Start from the original canonical forward ops attached to the local action.
2. Resolve all entity references to stable ids at persistence time.
3. Build inverse canonical ops while the pre-action DB state is still available.
4. Persist both directions on the action row under the same `tx-id`.
Representative inverse mappings:
| Forward op | Inverse op strategy |
| --- | --- |
| `:save-block` | Persist a `:save-block` payload built from pre-action block content and relevant pre-action refs. |
| `:insert-blocks` | Persist `:delete-blocks` for the created roots, or a more specific inverse if a safer canonical form exists. |
| `:move-blocks` | Persist `:move-blocks` back to the pre-action target with stable target id and structural opts. |
| `:delete-blocks` | Persist a hard-delete inverse only if block recreation is represented explicitly. Recycle-based restoration is no longer part of block deletion semantics. |
| `:set-block-property` | Persist `:set-block-property` or `:remove-block-property` depending on whether the property existed before the action. |
| `:batch-set-property` | Persist a batch inverse that restores prior values per block rather than a single blind batch overwrite. |
| `:create-page` | Persist `:delete-page` or a page retract inverse depending on page type. |
| `:delete-page` | Persist `:create-page` plus restoration of prior page content and relationships, or represent page restoration as a dedicated canonical op. |
The plan does not require every mapping to be implemented in one patch.
It does require the migration plan to make unsupported actions explicit so that they cannot silently flow into the new architecture.
## Why `tx-id` Must Be the Main-Thread DB History Reference
The UI stack needs a stable identifier for the DB action because that action may be:
- pending locally and not yet synced
- rebased by the worker
- acknowledged by the server
- rewritten during local compaction
- invalidated because the worker dropped it as unreplayable
If the main thread stores raw tx-data instead of `tx-id`, the UI stack and worker pending history can diverge.
If the main thread stores `tx-id`, the worker can rewrite the row contents while preserving logical action identity.
That makes the worker free to update the concrete replay payload during rebase without breaking the main-thread history pointer.
The worker must therefore treat `tx-id` as logical action identity rather than as a disposable row key.
## Cutover Plan
This change intentionally uses a strict cutover instead of a backward-compatible migration for old pending rows.
Cutover rules:
1. Existing pending rows without semantic persistence fields are cleared or dropped at startup.
2. Existing main-thread stack entries without `tx-id` are invalid after the cutover.
3. Old-format in-memory history is not preserved across the change.
4. New rows must always be written with the new semantic fields once the feature flag is enabled.
## Implementation Plan
### Phase 1. Add action-identity and semantic fields to client-op storage.
- Update `/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/worker/sync/client_op.cljs` schema for forward and inverse semantic ops plus version metadata.
- Keep existing fields intact.
- Ensure `tx-id` remains stable when a pending row is rewritten after rebase.
### Phase 2. Make the worker action row the source of truth for DB history.
- Update `/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/worker/sync/apply_txs.cljs` so local action persistence writes both directions when available.
- Split the meaning of existing `:db-sync/outliner-ops` from the new explicit forward and inverse fields rather than overloading one field for two roles.
- Preserve current `normalized-tx-data` and `reversed-tx-data` during migration.
### Phase 3. Change main-thread undo stack payloads from embedded tx data to `tx-id` references.
- Update `/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/undo_redo.cljs` so stack entries keep UI state and worker action references instead of DB tx payloads.
- Keep cursor and route state on the main thread.
- Remove the assumption that main-thread history owns the DB replay payload.
### Phase 4. Add a worker API for undo and redo execution by `tx-id`.
- Introduce a worker-facing command that resolves the client-op row by `tx-id`.
- Undo execution should replay `inverse-outliner-ops`.
- Redo execution should replay `forward-outliner-ops`.
- The command should return enough result metadata for the main thread to restore cursor and UI state after the worker confirms DB success.
### Phase 5. Replace raw undo/redo sync persistence with semantic persistence.
- Stop classifying new undo/redo executions as canonical raw `:transact` in `/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/worker/sync/apply_txs.cljs`.
- Instead, persist their semantic forward and inverse operations under the same logical action identity.
- Remove the old undo/redo raw replay path for pending-row persistence instead of keeping a compatibility branch.
### Phase 6. Extend rebase to use semantic inverse and forward ops for undo/redo rows.
- In `/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/worker/sync/apply_txs.cljs`, route undo/redo rows through semantic replay only.
- Preserve one-row-per-user-action atomicity.
- If any op in a row becomes invalid, drop or quarantine the whole row rather than partially replaying it.
### Phase 7. Add explicit invalidation rules for main-thread stack entries.
- If the worker drops or quarantines a referenced action row, the main-thread stack entry that points at that `tx-id` must be invalidated.
- Add a worker-to-main-thread signal so the UI can clear stale history references without guessing.
- Avoid silent dangling `tx-id` references in the UI stack.
### Phase 8. Add compaction and cleanup rules.
- Decide when an acknowledged action row can be compacted while still remaining undoable.
- If compaction rewrites stored payloads, preserve `tx-id`.
- If compaction removes a row, invalidate any stack references to it first.
## Required Code Areas
The following files are expected to change.
| File | Responsibility |
| --- | --- |
| `/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/undo_redo.cljs` | Change stack payloads to `tx-id` references and route DB replay to worker by action id. |
| `/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/handler/history.cljs` | Keep UI restore flow but consume worker-driven undo/redo result metadata. |
| `/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/worker/sync/client_op.cljs` | Extend client-op schema with semantic forward and inverse fields plus version metadata. |
| `/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/worker/sync/apply_txs.cljs` | Persist semantic undo/redo ops, keep stable `tx-id`, and use semantic replay in rebase. |
| `/Users/tiensonqin/Codes/projects/logseq/web/deps/outliner/src/logseq/outliner/op.cljs` | Reuse canonical op shapes and extend replay-visible surface if new canonical ops are introduced. |
| `/Users/tiensonqin/Codes/projects/logseq/web/deps/outliner/src/logseq/outliner/recycle.cljs` | Provide a replay-safe restoration entrypoint if `:restore-recycled` becomes canonical. |
| `/Users/tiensonqin/Codes/projects/logseq/web/src/test/frontend/undo_redo_test.cljs` | Cover `tx-id`-referenced history behavior and main-thread invalidation. |
| `/Users/tiensonqin/Codes/projects/logseq/web/src/test/frontend/worker/db_sync_test.cljs` | Cover semantic persistence and rebase of undo/redo action rows. |
| `/Users/tiensonqin/Codes/projects/logseq/web/src/test/frontend/worker/db_sync_sim_test.cljs` | Cover multi-client rebase scenarios where remote txs commute or conflict with undo/redo. |
## Edge Cases That Must Be Designed Explicitly
### Remote non-conflicting updates.
A remote title update on an unrelated block should not invalidate a local undo stack entry.
### Remote updates on the same entity but different attrs.
The design should allow semantic replay when the attrs commute and reject it when they do not.
### Structural conflicts.
Parent, page, order, delete, recycle, and missing-ref conflicts must invalidate the whole action row, not a subset of its ops.
### Undo of a batch action.
One logical batch action should remain one `tx-id` and should not split into multiple undoable rows during persistence or rebase.
### Redo after rebase.
Redo must continue to point at the same action identity even if the worker rewrote the row payloads during remote rebase.
### Session boundary.
Main-thread stacks are in-memory only today.
That remains acceptable, but the new `tx-id` reference model must not assume the stack survives app restart unless a later ADR decides to persist it.
### Unsupported inverse mappings.
An unsupported op must be explicit and traceable.
It must not silently fall back to safe-looking semantic replay if it is actually raw replay.
### Old pending rows.
Existing rows without semantic fields are intentionally not supported after the cutover.
They should be cleared rather than replayed.
## Risks
The largest risk is trying to migrate everything at once.
Undo/redo touches the main thread, the worker, client-op persistence, and rebase logic.
The plan therefore intentionally stages the work so that logical action identity and schema changes land before the main-thread stack rewrite and before the semantic rebase cutover.
Another risk is under-specifying inverse mappings for high-level actions such as delete-page.
Those mappings should be treated as explicit product decisions, not left to raw tx fallback by accident.
Another risk is user-visible loss of old pending rows during the cutover.
That tradeoff is accepted because preserving old raw replay rows would preserve the exact failure mode this ADR is intended to remove.
## Verification Plan
Verification must cover both behavior and migration safety.
Required coverage:
- main-thread undo stack entry stores `tx-id` plus UI state, not raw DB tx payload
- worker persists semantic forward and inverse ops on new local action rows
- undo executes by `tx-id` and replays inverse semantic ops
- redo executes by `tx-id` and replays forward semantic ops
- new undo/redo rows no longer persist as canonical raw `:transact`
- rebase preserves logical `tx-id` while rewriting row payloads
- benign remote updates continue to allow undo/redo replay
- structural conflicts invalidate the whole referenced action row
- dropped or quarantined worker rows invalidate corresponding UI history entries
- old rows without semantic fields are cleared and do not replay
Targeted commands once implementation begins:
- `bb dev:test -v frontend.undo-redo-test`
- `bb dev:test -v frontend.worker.db-sync-test`
- `bb dev:test -v frontend.worker.db-sync-sim-test`
## Consequences
Positive consequences:
- Undo/redo DB history and sync pending history converge on one logical action model.
- Rebase can reason about intent instead of replaying raw datoms for new undo/redo rows.
- The main thread keeps ownership of UI-only state without also becoming the long-term store of DB replay payloads.
- `tx-id` becomes a durable action identity that can survive rebase rewrites.
Negative consequences:
- The worker schema and the main-thread undo model both change at the same time.
- Several inverse-op mappings require explicit product and implementation decisions.
- The cutover is intentionally strict and may discard old pending rows that cannot satisfy the new semantic contract.
## Follow-up Notes
This ADR intentionally does not specify the final retention policy for acknowledged action rows.
It only requires that action-row lifetime be compatible with undo/redo stack references.
If future work decides to persist the UI stack across restarts, that should be handled in a separate ADR after this action-identity model is in place.

308
docs/adr/0012-worker-owned-undo-redo.md Normal file
View File

@@ -0,0 +1,308 @@
# ADR 0012: Move Undo/Redo Recording and Replay to the DB Worker
Date: 2026-03-21
Status: Proposed
## Context
`frontend.undo-redo` currently runs on the main thread.
That means undo and redo recording depends on main-thread listeners observing DB
tx reports after they have already crossed the worker boundary.
This split has become a recurring source of drift:
- the worker is the source of truth for the browser Datascript DB
- the worker already persists local actions in client-op storage
- the worker already owns rebase and semantic replay
- the main thread still owns undo/redo stack mutation for DB history
That architecture forces the main thread to reconstruct DB history from a
worker-synchronized tx report instead of observing the DB change at the place
where it actually happens.
The result is fragile metadata flow.
We have already seen bugs caused by:
- `:outliner-ops` being stripped or reshaped during worker-to-main-thread sync
- undo/redo-generated tx rows overwriting the original client-op row
- semantic forward and inverse ops diverging between worker persistence and
main-thread undo stack payloads
- special cases such as `:replace-empty-target?`, block concat, and
`:set-block-property` depending on worker-local replay behavior anyway
The one main-thread-only input that `frontend.undo-redo` still needs is
`@state/*editor-info`.
Today that atom is read and reset on the main thread inside
`frontend.undo-redo/gen-undo-ops!`.
If undo/redo recording moves to the worker, the worker can no longer deref the
main-thread atom directly.
## Decision
1. DB undo/redo recording and replay move to the DB worker.
2. The worker becomes the only place that listens to DB tx reports for DB
history generation.
3. The main thread remains responsible for UI-derived history inputs only:
editor cursor/focus metadata and UI-state snapshots.
4. `@state/*editor-info` will not be read from the worker directly.
It will be replaced by an explicit main-thread-to-worker handoff protocol.
5. The worker owns the undo stack and redo stack for DB actions and UI-adjacent
metadata attached to those actions.
6. The main thread will invoke worker APIs for:
- recording pending editor info
- recording UI-only state history entries
- undo
- redo
- clear history
7. The main thread will stop generating DB undo history from `:db/sync-changes`
events.
8. The worker-owned history row should not keep a separate persisted
`:db-sync/outliner-ops` field.
`:db-sync/forward-outliner-ops` is the only canonical persisted forward
semantic field.
## Rationale
The worker is already the place where all browser DB facts become real:
- local outliner ops are applied there
- remote sync txs are applied there
- pending local rows are persisted there
- semantic forward and inverse ops are canonicalized there
- rebase happens there
Undo/redo recording should therefore observe worker DB tx reports directly
instead of reconstructing them after the worker has serialized, sanitized, and
rebroadcast them.
That removes an entire class of metadata transport bugs.
It also matches ADR 0011 more closely: the worker action row is supposed to be
the source of truth for DB history. Recording DB history on the main thread is
in tension with that decision.
## Target Architecture
```text
+------------------------------+ thread-api +---------------------------+
| Main thread | ----------------------------> | DB worker |
| | | |
| editor lifecycle | push pending editor-info | pending editor-info store |
| route/sidebar state | push ui-state entries | undo stack |
| history handler | undo / redo / clear | redo stack |
| restore cursor + route | <---------------------------- | DB replay + result meta |
+------------------------------+ +---------------------------+
```
The worker stack entry becomes the single logical history item for both:
- DB replay metadata
- UI-adjacent metadata needed after replay
Representative worker stack item:
```clojure
{:tx-id #uuid "..."
:kind :db-action ; or :ui-state-only
:editor-info {:block-uuid ...
:container-id ...
:start-pos ...
:end-pos ...}
:ui-state-str "...optional transit..."
:forward-outliner-ops [...]
:inverse-outliner-ops [...]
:outliner-op :save-block}
```
The target row schema is therefore:
- `:db-sync/tx-id`
- `:db-sync/outliner-op`
- `:db-sync/forward-outliner-ops`
- `:db-sync/inverse-outliner-ops`
- worker-owned cursor/UI metadata as needed
It intentionally does not include a separate persisted
`:db-sync/outliner-ops`.
## `*editor-info` Handoff
The worker must not read `@state/*editor-info` directly.
That atom lives on the main thread and represents ephemeral UI state.
Instead, we will replace the implicit shared-state read with an explicit
handoff.
### Rule
The main thread owns editor-info production.
The worker owns editor-info consumption.
### Mechanism
Add a worker-side pending editor-info slot keyed by repo.
Suggested API:
- `:thread-api/undo-redo-set-pending-editor-info`
- args: `repo`, `editor-info-or-nil`
- `:thread-api/undo-redo-record-ui-state`
- args: `repo`, `ui-state-str`
- `:thread-api/undo-redo-undo`
- args: `repo`
- `:thread-api/undo-redo-redo`
- args: `repo`
- `:thread-api/undo-redo-clear-history`
- args: `repo`
### Consumption and reset semantics
When the worker records a new local DB action into undo history:
1. read pending editor-info for the repo
2. attach it to the new stack item if present
3. clear the pending editor-info slot immediately after the stack item is
created
This preserves the current one-shot semantics of `@state/*editor-info` without
requiring the worker to deref or mutate main-thread state directly.
### Main-thread responsibilities
The main thread should:
- capture editor-info at the same points it does today
- send the current snapshot to the worker before the local DB action is
submitted or immediately when editor focus/cursor changes, whichever path is
simpler and consistent
- stop relying on worker `:db/sync-changes` to retroactively capture cursor
state
The main thread may still keep a local `*editor-info` atom for editor UI code,
but it is no longer the undo recorder’s source of truth.
## UI-State History
UI-state-only undo entries such as route/sidebar snapshots cannot be generated
by the worker from DB tx reports.
Those entries should be pushed explicitly from the main thread into the worker
history stack.
Two entry classes will therefore exist in the worker stack:
1. `:db-action`
2. `:ui-state-only`
Undo/redo execution will return enough metadata for the main thread to restore:
- route
- sidebar state
- editor cursor
The worker should not attempt to perform UI restoration itself.
## Consequences
### Positive
- DB undo/redo history is recorded at the actual DB source of truth.
- No more dependence on `:db/sync-changes` preserving semantic metadata exactly.
- Worker persistence, worker replay, and worker history all use the same action
identity.
- Main-thread history bugs caused by tx-meta sanitization disappear.
- Undo/redo debugging becomes simpler because the worker owns the full DB
history lifecycle.
### Negative
- The worker history stack now stores UI-adjacent metadata that originates on
the main thread.
- New thread APIs are required.
- Main-thread editor lifecycle code must actively synchronize pending
`editor-info`.
- The migration touches both undo/redo and worker message flow at once.
## Implementation Plan
### Phase 1. Introduce worker-owned undo/redo module
- Create a worker namespace, e.g.
`/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/worker/undo_redo.cljs`
- Move stack storage and DB history generation there.
- Register worker DB listener(s) against the worker Datascript conn.
- Remove persisted `:db-sync/outliner-ops` from the target worker history row
shape instead of carrying it forward as a parallel field.
### Phase 2. Replace main-thread DB history generation
- Remove DB-history recording from
`/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/undo_redo.cljs`
- Keep only main-thread coordination helpers if still needed.
- Route history handler calls through worker thread APIs.
### Phase 3. Add pending editor-info handoff
- Add worker API to set pending editor-info.
- Update
`/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/modules/outliner/ui.cljc`
and any direct local transact paths to send editor-info to the worker.
- Consume and clear the pending editor-info slot when a local history item is
recorded.
### Phase 4. Move UI-state history writes to worker
- Replace `record-ui-state!` main-thread stack mutation with worker API calls.
- Keep route/sidebar serialization on the main thread.
### Phase 5. Return worker-owned undo/redo result metadata
- Worker undo/redo APIs should return:
- `:undo?`
- `:editor-info`
- `:ui-state-str`
- optional block content or replay diagnostics
- Main-thread history handler restores cursor and route from that result.
## Files Expected to Change
| File | Change |
| --- | --- |
| `/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/undo_redo.cljs` | Remove main-thread DB listener ownership, keep only coordinator logic if still needed. |
| `/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/handler/history.cljs` | Call worker undo/redo APIs and restore UI from returned metadata. |
| `/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/modules/outliner/ui.cljc` | Send editor-info snapshots to the worker before local action submission. |
| `/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/handler/editor/lifecycle.cljs` | Stop recording editor-info directly into main-thread undo stack; feed worker pending editor-info instead. |
| `/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/worker/db_worker.cljs` | Expose worker thread APIs for pending editor-info, UI-state history, undo, redo, and clear-history. |
| `/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/worker/db_listener.cljs` | Attach worker undo/redo recording directly to worker DB tx reports. |
| `/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/worker/sync/client_op.cljs` | Remove `:db-sync/outliner-ops` from the target worker-owned undo/redo row model and use `:db-sync/forward-outliner-ops` instead. |
| `/Users/tiensonqin/Codes/projects/logseq/web/src/main/frontend/worker/sync/apply_txs.cljs` | Keep worker replay aligned with worker-owned history rows. |
| `/Users/tiensonqin/Codes/projects/logseq/web/src/test/frontend/undo_redo_test.cljs` | Replace main-thread DB-history expectations with coordinator/result expectations. |
| `/Users/tiensonqin/Codes/projects/logseq/web/src/test/frontend/worker/db_sync_test.cljs` | Add worker-owned history recording and replay coverage. |
## Alternatives Considered
### 1. Keep `frontend.undo-redo` on the main thread and preserve more tx-meta
Rejected.
This keeps the wrong ownership boundary.
It reduces one transport bug at a time but does not remove the architectural
duplication between main-thread history and worker DB history.
### 2. Let the worker call back into the main thread to read `@state/*editor-info`
Rejected.
That would create an implicit cross-thread read dependency around ephemeral UI
state.
It is harder to reason about than explicit handoff, and reset semantics become
ambiguous.
### 3. Keep DB history in the worker and UI history in a separate main-thread stack
Possible, but inferior to a single worker-owned stack item keyed by `tx-id`.
It still splits one logical action across two structures and reintroduces
alignment problems.
## Open Questions
1. Should pending editor-info be pushed:
- only at transact boundaries
- or eagerly on every cursor change with last-write-wins semantics?
Recommendation:
push at transact boundaries first.
It matches current one-shot behavior and avoids unnecessary worker chatter.
2. Should `:ui-state-only` entries live in the same stack as DB actions?
Recommendation:
yes.
One logical undo/redo stream is simpler than coordinating two stacks.
3. Do we still need `@state/*editor-info` after the migration?
Recommendation:
keep it as a UI helper until the move is complete, but stop using it as undo
history source of truth.

187
docs/adr/0013-worker-owned-undo-redo-test-ownership.md Normal file
View File

@@ -0,0 +1,187 @@
# ADR 0013: Worker-Owned Undo/Redo Test Ownership
Date: 2026-03-21
Status: Proposed
## Context
ADR 0012 moves DB undo/redo ownership from the main thread to the db worker.
That architectural move is not complete if the test suite still treats
`src/test/frontend/undo_redo_test.cljs` as the primary place to assert DB
history behavior.
Today the old main-thread file still contains most undo/redo tests, including:
- local tx recording
- semantic forward and inverse metadata
- insert/save/delete replay sequences
- replay conflict and skip behavior
- validation behavior tied to DB replay
Those tests were correct for the old design, but they now encourage the wrong
ownership boundary.
The worker already owns:
- worker datascript DB mutation
- client-op persistence
- semantic replay
- pending action identity
- replay safety decisions
The main thread now mainly owns:
- route/sidebar UI restoration
- cursor restoration from worker result metadata
- thin browser-facing proxy calls
The test suite should reflect that split directly.
## Decision
1. `src/test/frontend/worker/undo_redo_test.cljs` becomes the primary home for
DB-history-focused undo/redo tests.
2. `src/test/frontend/undo_redo_test.cljs` is reduced to main-thread-only
coordination tests.
3. Replay, conflict, and rebase-heavy assertions that are really worker replay
behavior belong in `src/test/frontend/worker/db_sync_test.cljs`, not in
either undo-redo namespace test file.
4. No new DB-history behavior test should be added to
`src/test/frontend/undo_redo_test.cljs`.
## Test Ownership Rules
### Keep in `src/test/frontend/worker/undo_redo_test.cljs`
- local tx recording gates
- semantic metadata persistence
- worker-owned undo stack and redo stack mutation
- canonical action-id and semantic op persistence
- worker-owned DB-history entries that include pending editor-info and UI-state
metadata
### Keep in `src/test/frontend/worker/db_sync_test.cljs`
- `apply-history-action!` replay behavior
- semantic replay correctness
- conflict and skip behavior caused by worker replay safety
- rebase interactions
- client-op row persistence and rewrite behavior
### Keep in `src/test/frontend/undo_redo_test.cljs`
- route/sidebar UI-state restoration on the main thread
- cursor restoration from worker result metadata
- browser-facing proxy semantics in `frontend.undo-redo` and
`frontend.handler.history`
- any pure coordination helper that still lives only on the main thread
## Migration Buckets
### Bucket 1. Recording and metadata tests.
Move these first:
- `undo-records-only-local-txs-test`
- `undo-history-records-semantic-action-metadata-test`
- `undo-history-records-forward-ops-for-editor-save-block-test`
- `undo-history-canonicalizes-insert-block-uuids-test`
These tests map directly to worker history ownership and do not require route or
cursor restoration.
### Bucket 2. Basic local replay tests.
Move next, but only after the worker fixture persists matching client-op rows:
- `undo-works-for-local-graph-test`
- `repeated-save-block-content-undo-redo-test`
- `repeated-editor-save-block-content-undo-redo-test`
- `editor-save-two-blocks-undo-targets-latest-block-test`
- `new-local-save-clears-redo-stack-test`
If a test still depends on browser-only helpers, rewrite it to assert worker DB
behavior directly or leave it temporarily unmoved.
### Bucket 3. Delete/recycle sequence tests.
Reclassify before moving:
- `insert-save-delete-sequence-undo-redo-test`
- `undo-redo-works-for-recycle-delete-test`
If the real behavior under test is semantic replay, keep or move it to
`src/test/frontend/worker/db_sync_test.cljs`.
### Bucket 4. Conflict and skip tests.
Reclassify these into worker replay coverage unless they are truly worker stack
mutation tests:
- `undo-conflict-clears-history-test`
- `undo-works-with-remote-updates-test`
- `undo-skips-when-parent-missing-test`
- `undo-skips-when-block-deleted-remote-test`
- `undo-skips-when-undo-would-create-cycle-test`
- `undo-skips-conflicted-move-and-keeps-earlier-history-test`
- `redo-builds-reversed-tx-when-target-parent-is-recycled-test`
- `undo-skips-move-when-original-parent-is-recycled-test`
### Bucket 5. Main-thread leftovers.
After Buckets 1-4:
- keep only route restoration
- keep only cursor restoration
- keep only proxy/coordination tests
- add a namespace comment in `src/test/frontend/undo_redo_test.cljs` that it is
no longer the DB-history source-of-truth test file
## Implementation Rules
1. Every moved worker test must use a fixture that owns both:
- worker datascript conn
- worker client-ops conn
2. That fixture must route tx reports through both:
- `frontend.worker.sync/enqueue-local-tx!`
- `frontend.worker.undo-redo/gen-undo-ops!`
3. Tests that expect persisted action lookup must attach a stable
`:db-sync/tx-id`.
4. Do not duplicate replay tests between `worker/undo_redo_test.cljs` and
`worker/db_sync_test.cljs`.
5. Delete the original test only after the worker version passes.
## Consequences
### Positive
- The test suite matches the actual runtime ownership boundary.
- Worker replay regressions are caught where they actually happen.
- The old main-thread test file becomes much easier to reason about.
### Negative
- Some existing tests need adaptation rather than straight copy because they
implicitly relied on old main-thread stack storage.
- The migration requires explicit decisions about whether a test is about
worker history ownership or worker replay behavior.
## Verification
Representative focused commands during migration:
```bash
bb dev:test -v frontend.worker.undo-redo-test/undo-records-only-local-txs-test
bb dev:test -v frontend.worker.undo-redo-test/undo-history-records-semantic-action-metadata-test
bb dev:test -v frontend.worker.undo-redo-test/undo-history-canonicalizes-insert-block-uuids-test
bb dev:test -v frontend.worker.db-sync-test/apply-history-action-redo-replays-save-block-test
bb dev:test -v frontend.worker.db-sync-test/apply-history-action-redo-replays-block-concat-test
bb dev:test -v frontend.worker.db-sync-test/apply-history-action-redo-replays-paste-into-empty-target-test
```
## Exit Criteria
- `src/test/frontend/worker/undo_redo_test.cljs` owns the DB-history recording
tests.
- `src/test/frontend/undo_redo_test.cljs` contains only main-thread coordination
tests.
- No worker-owned DB-history scenario is tested only in the main-thread file.

58
docs/adr/0014-kv-row-r2-snapshot-download.md Normal file
View File

@@ -0,0 +1,58 @@
# ADR 0014: KV-Row R2 Snapshot Download With Worker-Owned Low-Memory Import
Date: 2026-04-01
Status: Proposed
## Context
Snapshot download previously exported Datascript datoms as gzip NDJSON from the
server and parsed/transacted datoms on the client main-thread handler path.
That design had two issues:
1. Server snapshot export walked full datoms and spent avoidable CPU/memory.
2. Client download logic lived in handler code and was not aligned with worker
ownership for large-graph import.
We already use framed Transit `kvs` rows for snapshot upload. Download should
converge on the same wire format.
## Decision
1. `GET /sync/:graph-id/snapshot/download` and `/snapshot/stream` export framed
Transit `kvs` rows (`[addr content addresses]`) instead of datom NDJSON.
2. Snapshot download payload content-type is `application/transit+json`
(gzip-compressed when available).
3. Server snapshot export reads directly from sqlite `kvs` rows in ascending
`addr` batches and streams framed payloads to response/R2.
4. Graph snapshot download orchestration is moved to
`frontend.worker.sync.download` and invoked from db-worker thread API.
5. Handler code delegates graph download to worker API instead of parsing
snapshot payloads directly.
6. Client import adds row-chunk API (`:thread-api/db-sync-import-rows-chunk`).
Row batches are staged in temp sqlite, then replayed into target conn in
schema-first order.
7. Replay order must transact schema-critical datoms before regular data:
- `:logseq.kv/schema-version` entity datoms
- attribute-definition datoms (`:db/ident` and `:db/*` metadata such as
`:db/valueType`, `:db/cardinality`, `:db/unique`, `:db/isComponent`)
- all remaining datoms
## Consequences
### Positive
- Lower server CPU/memory for snapshot export (no datom NDJSON generation).
- Download/upload snapshot format is unified around framed `kvs` rows.
- Download pipeline ownership moves to worker sync module.
- Schema-first replay protects index/schema correctness for large imports.
### Tradeoffs
- Client still performs datom replay during finalize to rebuild a consistent
target store, so import cost shifts to worker finalize phase.
- Adds temp sqlite staging and one additional import path (`rows` alongside
legacy datom chunk path).
## Verification
- Server tests assert snapshot download/stream return framed kv rows with
transit content-type and sorted addresses.
- Handler tests assert graph download delegates to worker API and maintains
download-state lifecycle.
- Worker tests assert rows-chunk API wiring and schema-first import ordering.

123
docs/adr/0015-client-ops-and-sync-meta-in-client-sqlite.md Normal file
View File

@@ -0,0 +1,123 @@
# ADR 0015: Store Client Ops in SQLite `client_ops` and Metadata in `sync_meta`
Date: 2026-04-10
Status: Accepted
## Context
Client sync state is currently persisted via a DataScript-backed store in the
dedicated `client-ops-*` sqlite database (`kvs` + DataScript entities).
This design has become a bottleneck for local sync hot paths:
1. DataScript entity/datoms read-write patterns add avoidable CPU overhead for
client op persistence and lookup.
2. Persisting client ops as DataScript entities consumes more space than
purpose-built relational rows for the same data.
3. Metadata (`local-tx`, checksum, graph uuid) and operation payloads are mixed
in DataScript entity shape, which complicates low-level sqlite debugging and
parity with server-side sync storage.
Server sync storage already uses explicit sqlite tables (`tx_log`,
`sync_meta`) over the same sqlite runtime. Client storage should follow the
same direction.
## Decision
1. Keep the existing dedicated `client-ops-*` sqlite database file.
Do not merge this storage into the main graph sqlite DB in this change.
2. Replace DataScript persistence for client ops/metadata with direct sqlite
tables:
- `sync_meta` for key/value metadata rows
- `client_ops` for operation rows
3. `sync_meta` schema:
- `key TEXT PRIMARY KEY`
- `value TEXT`
4. `sync_meta` keys for this ADR:
- `local-tx`
- `checksum`
- `graph-uuid`
5. `client_ops` stores both local tx ops and asset ops (single table,
discriminated by row kind).
6. `client_ops` minimum columns:
- identity and ordering:
- `id INTEGER PRIMARY KEY AUTOINCREMENT`
- `kind TEXT NOT NULL` (`'tx'` or `'asset'`)
- `created_at INTEGER NOT NULL`
- local tx fields:
- `tx_id TEXT` (uuid string)
- `pending INTEGER NOT NULL DEFAULT 0`
- `failed INTEGER NOT NULL DEFAULT 0`
- `outliner_op TEXT`
- `undo_redo TEXT` (`undo`/`redo`/`none`)
- `forward_outliner_ops TEXT` (Transit string)
- `inverse_outliner_ops TEXT` (Transit string)
- `inferred_outliner_ops INTEGER`
- `normalized_tx_data TEXT` (Transit string)
- `reversed_tx_data TEXT` (Transit string)
- asset fields:
- `asset_uuid TEXT`
- `asset_op TEXT` (`update-asset`/`remove-asset`)
- `asset_t INTEGER`
- `asset_value TEXT` (Transit or JSON string payload)
7. Required indexes/constraints:
- unique tx identity for undo/redo lookup:
- `UNIQUE(tx_id)` where `kind='tx'` and `tx_id IS NOT NULL`
- pending tx scan in stable order:
- index on `(kind, pending, created_at, id)`
- asset op lookup by uuid:
- index on `(kind, asset_uuid)`
8. `asset ops` must be stored in `client_ops` (not separate DataScript
entities).
9. Client code must access this data via SQL adapter functions (insert/select/
update/delete), not `d/datoms`/`d/entity` over client-op storage.
10. Keep payload encoding compatible with existing sync payload handling by
storing complex fields as Transit strings.
## Consequences
### Positive
- Hot path operations (enqueue local tx, pending scan, tx-id lookup, asset-op
mutation) become cheaper and more predictable.
- Storage is denser than DataScript entity persistence for client ops.
- Debugging and inspection are easier with explicit sqlite rows.
- Client layout aligns with server-style metadata table (`sync_meta`).
### Tradeoffs
- Introduces SQL adapter code replacing generic DataScript entity APIs.
- Requires new client-op table schema bootstrap and index management.
- Existing client-op DataScript rows are no longer read by new code.
## Safety and Integrity Rules
1. All mutating operations on `client_ops`/`sync_meta` use sqlite transactions.
2. Ordering guarantees for pending tx upload rely on `(created_at, id)` order.
3. Undo/redo lookup remains keyed by `tx_id` and must continue to read forward/
inverse semantic ops from persisted local tx rows.
4. `pending` and `failed` flags are explicit row state and must not be inferred
from absence.
5. This change does not alter checksum semantics; it only changes where client
checksum metadata is stored.
## Migration
Chosen policy: no migration from old DataScript client-op entities.
On first open with this schema:
- initialize `sync_meta` + `client_ops` tables and indexes.
- start client-op state empty.
Rationale:
- client-op state is derived/operational, not authoritative graph content.
- normal sync flows (pull/upload/asset requeue) reconstruct operational state.
- avoids fragile one-time conversion logic across old entity layouts.
## Verification
Focused verification after implementation:
```bash
bb dev:test -v frontend.worker.db-sync-test/handle-local-tx-persists-pending-local-tx-test
bb dev:test -v frontend.worker.db-sync-test/flush-pending-clears-pending-local-tx-count-test
bb dev:test -v frontend.worker.undo-redo-test/undo-history-records-semantic-action-metadata-test
bb dev:test -v frontend.worker.db-sync-test/asset-ops-queue-coalesces-update-and-remove-test
```
Additional checks:
- pending tx query returns rows in deterministic `(created_at, id)` order.
- undo/redo can still resolve tx history by `tx_id`.
- `local-tx`, `checksum`, and `graph-uuid` read/write through `sync_meta`.

291
docs/adr/0016-markdown-mirror.md Normal file
View File

@@ -0,0 +1,291 @@
# ADR 0016: Electron Markdown Mirror
Date: 2026-05-05
Status: Accepted
## Context
Logseq DB graphs do not expose one editable Markdown file per page in the graph
directory. Some desktop workflows still need a Markdown representation that can
be read by external tools, backed up, indexed, or inspected outside Logseq.
The mirror must not become another graph source of truth. Editing should remain
fast, and saving a block must not wait for Markdown rendering or filesystem
writes on the renderer main thread.
The first supported runtime is the Electron desktop app. Browser and mobile
builds do not have the same graph-directory filesystem guarantees.
## Decision
1. Add an Electron-only Settings toggle for Markdown Mirror.
2. Persist the toggle in Electron user settings as
`:feature/markdown-mirror?`.
3. When the setting is enabled for the Electron app, Logseq writes derived
Markdown files under the current graph directory:
- journals:
`mirror/markdown/journals/<journal-file-name>.md`
- other pages:
`mirror/markdown/pages/<page-file-name>.md`
4. For a graph at `~/logseq/graphs/graph-xxx`, mirror files are written under:
- `~/logseq/graphs/graph-xxx/mirror/markdown/journals/`
- `~/logseq/graphs/graph-xxx/mirror/markdown/pages/`
5. Markdown Mirror is derived output. The DB remains the source of truth.
6. Files under `mirror/markdown/**` must be ignored by graph import, file
watchers, and graph parsing so the mirror never feeds back into the graph.
7. The feature is not available in browser or mobile builds, even if a stale
setting value exists.
8. Settings exposes an explicit "Regenerate full mirror" action that asks the
DB worker to rewrite the complete mirror for the current graph.
9. Built-in pages and property pages, including user-created properties, are not
exported to the mirror. User Tag/Class pages are normal user content and are
exported.
## Runtime Ownership
1. The renderer owns only:
- the Settings row
- reading and updating `:feature/markdown-mirror?`
- pushing the enabled state to the DB worker when it changes
2. The DB worker owns:
- detecting affected page ids from successful local transactions
- rendering page Markdown from the worker DB snapshot
- scheduling and coalescing mirror jobs
- running explicit full-mirror regeneration jobs
- invoking platform filesystem writes
3. The Electron main process must not render Markdown. It may provide filesystem
primitives if needed, but content generation stays with the worker.
4. Editor save paths enqueue mirror work and return immediately. They must not
wait for rendering, directory creation, stat, write, rename, or delete.
## Reusable Core and CLI Path
1. Markdown Mirror path planning, filename normalization, page rendering, write
deduplication, atomic writes, rename cleanup, and delete cleanup live in a
worker/core namespace that does not depend on Electron UI state.
2. The Electron app only owns feature activation through Settings.
3. The CLI should be able to reuse the same core by passing an explicit graph,
DB snapshot, and node filesystem platform context.
4. Future CLI support should not introduce a second Markdown serializer or a
different filename normalization policy.
5. Future CLI support should reuse the same mirror-path allocation index so the
Electron app and CLI do not produce different file names for the same graph.
## Output Layout and Naming
1. Journal pages are written below `mirror/markdown/journals/`.
2. Non-journal pages are written below `mirror/markdown/pages/`.
3. Journal file names use the existing Logseq journal file-name rules for the
graph configuration.
4. Non-journal page file names use the normalized page title:
`<page-file-name>.md`.
5. Page file names must stay friendly to external Markdown tools such as Emacs,
VS Code, and Obsidian. Do not include page uuid in normal mirror file names.
6. Page title is not page identity. The page uuid is still the internal mirror
identity, but it is stored in the mirror index rather than exposed in the
file name.
7. Duplicate non-journal page titles are handled by stable title suffix
allocation:
- first allocated page: `pages/Foo.md`
- second allocated page: `pages/Foo (2).md`
- third allocated page: `pages/Foo (3).md`
8. Once a page uuid is assigned a mirror path, keep that path stable until the
page is renamed or deleted. Do not renumber existing duplicate-title mirror
paths when another duplicate is created or removed.
9. The implementation keeps a per-graph mirror index under
`mirror/markdown/.index.edn`.
10. The mirror index stores at least:
- page uuid -> relative mirror path
- relative mirror path -> page uuid
- page uuid -> last known normalized title stem
11. The mirror index is implementation metadata for path stability. It is not
graph content and must be ignored by graph import and watchers along with the
rest of `mirror/markdown/**`.
12. All mirror file names pass through a single cross-platform filename
normalizer before joining paths.
13. Duplicate journal-day entities indicate invalid graph state for the mirror.
The implementation must fail those journal mirror jobs and surface a
diagnostic instead of choosing a winner.
14. If two entities still map to the same mirror path, the implementation must
fail the mirror job for that path and surface a diagnostic instead of
overwriting an unrelated page.
15. Page rename moves the mirror by writing the new path, updating the mirror
index, and deleting the old path after the new file has been written.
16. Page deletion deletes the corresponding mirror file and removes the page uuid
from the mirror index.
17. The write guard must reject any computed path outside the graph's
`mirror/markdown/` directory.
18. Built-in pages and property pages are excluded from path allocation and
mirror writes. User Tag/Class pages are not excluded by this rule. If a
previously mirrored page becomes excluded, the old mirror file is removed.
## Duplicate Page Title Allocation
1. For non-journal pages, compute the normalized title stem first.
2. If the page uuid already exists in the mirror index and the normalized title
stem did not change, reuse the indexed path.
3. If the page uuid is new for that title, allocate the first unused path in this
sequence:
- `pages/<stem>.md`
- `pages/<stem> (2).md`
- `pages/<stem> (3).md`
4. A path is considered unavailable when the mirror index maps it to a different
live page uuid.
5. Deleted page paths become available for future allocation only after the
deleted page uuid is removed from the index.
6. Rename is treated as a new allocation for the new title stem. Existing pages
with the old title keep their already allocated paths.
7. If the mirror index is missing or unreadable, rebuild it from the current DB
in deterministic page order before writing. Deterministic order should use a
stable key such as page title plus page uuid.
8. The rebuilt index is allowed to choose paths for pages that had no previous
allocation. It must not overwrite a live existing path that is already mapped
to another page uuid.
## Rename and Delete
1. Page rename moves the mirror by writing the new path and deleting the old
path after the new file has been written.
2. Page deletion deletes the corresponding mirror file.
3. The implementation keeps a small per-graph mirror index keyed by page uuid so
rename and delete handling does not require scanning the mirror directory on
every transaction.
## Filename Normalization
1. Mirror file names must be portable across macOS, Windows, and Linux.
2. Use one shared normalizer for journal and page mirror file names.
3. The normalizer must:
- reject or replace path separators (`/`, `\`)
- reject or replace Windows-invalid characters (`<`, `>`, `:`, `"`, `|`,
`?`, `*`) and ASCII control characters
- reject or rewrite reserved Windows device names such as `CON`, `PRN`,
`AUX`, `NUL`, `COM1` through `COM9`, and `LPT1` through `LPT9`
- trim trailing spaces and dots because Windows does not preserve them
- reject empty names after normalization
- bound each file-name component to a safe byte length before appending
`.md`
4. Normalize Unicode to one canonical form before sanitizing so the same page
title produces the same mirror path across filesystems with different Unicode
normalization behavior.
5. The normalizer must be deterministic and must not depend on the current
operating system. A graph mirrored on macOS should choose the same logical
file name as the same graph mirrored on Windows.
6. If normalization changes the display title segment, the mirror index and
duplicate-title suffix allocation still preserve identity for non-journal
pages.
7. If a journal title normalizes to an unsafe or colliding file name, fail the
journal mirror job and surface diagnostics instead of inventing a fallback
name.
8. Path construction must join only validated path components. It must never
concatenate unchecked page titles into filesystem paths.
## Scheduling and Performance
1. Mirror rendering is incremental. A transaction schedules only pages affected
by that transaction.
2. Jobs are coalesced by page uuid. If a page is edited repeatedly before its
job runs, only the latest worker DB state is rendered.
3. Scheduling uses a short debounce window per graph to reduce write churn while
preserving near-real-time output.
4. Mirror writes are serialized per graph to avoid path races during rename and
delete.
5. Before writing, compare the generated Markdown with the current file content
or with the last written content hash. Skip the write when content is
unchanged.
6. Write files atomically:
- ensure the target directory exists
- write to a temporary file in the same directory
- rename the temporary file over the target
7. Heavy work is forbidden on the renderer main thread:
- no full-graph export
- no Markdown rendering
- no filesystem reads or writes
- no synchronous IPC waiting for mirror completion
8. Full regeneration is an explicit Settings action. The renderer only sends a
worker request; page selection, rendering, and filesystem writes stay in the
DB worker.
9. Enabling the setting starts incremental mirroring for subsequent page edits.
It does not implicitly run full regeneration.
## Markdown Content
1. Reuse the existing page-to-Markdown export pipeline used by worker export
APIs instead of introducing a separate renderer-side serializer.
2. The mirror output uses the Logseq Markdown syntax described in
`docs/logseq-markdown-syntax.md`.
3. Mirror files always include a page `id:: <uuid>` line so the file can be
associated with its DB page without encoding the uuid in the file name.
4. Blocks are Markdown list items that use `-`.
5. User-visible page and block properties are Markdown list items that use `*`
and keep the Logseq property marker, for example `* owner:: [[Alice]]`.
6. Open default property values are exported as nested value blocks below the
property key. Closed values remain inline with the property key.
7. Node property values are exported as Logseq page references, for example
`[[Node title]]`, rather than block reference syntax.
8. Task status is encoded on the block line, for example `- TODO ship`, and is
not exported as a separate property list item.
9. Assets are referenced as normal exported Markdown references. This ADR does
not copy assets into `mirror/markdown/`.
10. Page references remain in Logseq wiki-link form, for example `[[Foo]]`.
11. Ambiguous page references caused by duplicate page titles are an accepted
limitation of Markdown Mirror. Do not rewrite page references to uuid-based
links or relative Markdown links in this ADR.
## Failure Handling
1. Filesystem and path errors fail the mirror job for the affected page.
2. Failures are logged with graph, page uuid, target path, and error details.
3. Repeated failures should be visible from Settings or diagnostics; do not show
a toast on every keystroke.
4. The feature must not silently fall back to browser storage, OPFS, or another
output directory.
5. If the graph directory is not available, the worker rejects mirror jobs for
that graph until the graph is reopened with a valid directory.
## Non-goals
1. Markdown Mirror is not bidirectional sync.
2. Editing files in `mirror/markdown/` does not update the graph.
3. The mirror is not a backup format with guaranteed import fidelity.
4. The mirror does not replace existing graph export features.
5. The mirror does not support browser or mobile runtimes in this ADR.
## Consequences
### Positive
- Desktop users get a readable Markdown projection inside the graph directory.
- Editor latency is protected because rendering and disk I/O are worker-owned
and asynchronous.
- The output layout is predictable for tools that watch journals and pages
separately.
- Page file names remain readable and practical in external Markdown tools.
- Ignoring `mirror/markdown/**` prevents mirror-generated files from becoming
graph input.
### Tradeoffs
- The mirror can lag slightly behind the latest edit because writes are
debounced and serialized.
- A per-graph mirror index is needed for reliable rename and delete cleanup.
- Duplicate page references such as `[[Foo]]` remain ambiguous in mirror output.
- The first version does not backfill every existing page automatically when the
setting is enabled; users run full regeneration explicitly.
- External edits to mirror files are overwritten by later Logseq edits.
- Property pages are intentionally absent from the mirror, so the output is not
a complete DB export even though page and block property drawers are included.
## Verification
Implementation should add focused coverage for:
```bash
bb dev:test -v frontend.worker.markdown-mirror-test/enabled-electron-edit-writes-page-mirror-test
bb dev:test -v frontend.worker.markdown-mirror-test/enabled-electron-edit-writes-journal-mirror-test
bb dev:test -v frontend.worker.markdown-mirror-test/disabled-setting-does-not-write-mirror-test
bb dev:test -v frontend.worker.markdown-mirror-test/repeated-edits-coalesce-to-latest-content-test
bb dev:test -v frontend.worker.markdown-mirror-test/rename-removes-old-mirror-path-test
bb dev:test -v frontend.worker.markdown-mirror-test/delete-removes-mirror-file-test
bb dev:test -v frontend.worker.markdown-mirror-test/same-title-pages-write-distinct-stable-friendly-paths-test
bb dev:test -v frontend.worker.markdown-mirror-test/page-references-remain-wiki-links-test
bb dev:test -v frontend.worker.markdown-mirror-test/page-mirror-exports-property-values-test
bb dev:test -v frontend.worker.markdown-mirror-test/page-mirror-exports-page-property-values-test
bb dev:test -v frontend.worker.markdown-mirror-test/page-mirror-exports-node-property-values-as-page-refs-test
bb dev:test -v frontend.worker.markdown-mirror-test/full-regeneration-writes-existing-non-built-in-non-property-pages-test
bb dev:test -v frontend.worker.markdown-mirror-test/invalid-filename-characters-are-normalized-test
bb dev:test -v frontend.worker.markdown-mirror-test/windows-reserved-filename-fails-with-diagnostic-test
bb dev:test -v frontend.worker.markdown-mirror-test/mirror-path-collision-fails-without-overwrite-test
```
Additional checks:
- `mirror/markdown/**` is excluded from graph parsing and file watchers.
- Editor save does not await mirror completion.
- Browser and mobile builds do not expose the setting and do not schedule mirror
jobs.
- Atomic write failures do not leave partial target files.

195
docs/adr/0017-url-target-graph-resolution.md Normal file
View File

@@ -0,0 +1,195 @@
# ADR 0017: URL Target Graph Resolution
Date: 2026-05-20
Status: Accepted
## Context
Logseq URLs can point at a graph, page, block, or file. Today the graph target is
not carried through one coherent flow.
The custom protocol path is handled in Electron main:
- `logseq://graph/<graph>` and `logseq://new-window/<graph>` are parsed in
`electron.url/local-url-handler`.
- Electron resolves `<graph>` through `electron.handler/get-graph-name`.
- If no window is already associated with that graph, Electron asks the current
renderer to run `openNewWindowOfGraph`.
- The renderer then opens a new Electron window with `#/?graph=<repo>`.
This depends on renderer IPC timing and renderer-local storage. On first app
launch, the initial deeplink is handled soon after the main window is created,
while the renderer listener can still be unavailable. When the message is lost,
the newly opened app falls back to the stored `:git/current-repo`, so the URL
appears to always open the current graph.
The web app has a related weakness. `frontend.state` only reads `:graph` from
`frontend.util/parse-params` during initial state creation, and that parser only
understands hash fragments shaped like `#/?graph=...`. Page/block web URLs and
later hash changes can route inside the currently opened graph instead of first
switching to the graph named by the URL.
Graph names are not stable identifiers. Users can rename graphs, and two linked
graphs can share the same short basename. URLs that are meant to survive
renames, browser sessions, and device handoffs need to identify the graph by id.
## Decision
Make URL graph selection an explicit target object and process it before route
navigation in the renderer, with a small persisted graph registry used to turn
stable graph ids into local repos.
1. Introduce a shared URL target model for Electron protocol URLs, web URLs, and
mobile URLs:
- `:graph-id` - canonical graph id from the URL.
- `:graph-identifier` - compatibility alias, such as a short graph name or
full DB repo name.
- `:repo` - resolved full local repo name, for example `logseq_db_work`.
- `:route` - one of graph home, page, block, or file.
- `:open-mode` - current window or new window.
- `:source` - protocol URL, web URL, startup argv, or second instance.
2. Use `:graph-id` as the canonical URL identity. For remote graphs,
`:graph-id` is the remote graph UUID. For local-only graphs, `:graph-id` is
the local graph UUID. Generated URLs should include `graph-id` whenever the
graph id is known.
3. Resolve URL graph targets through one graph resolver that accepts, in order:
- graph ids from the local graph registry;
- full DB repo names, such as `logseq_db_work`;
- short local graph names, such as `work`;
- existing canonicalized DB repo names.
4. Treat an unresolved graph id as an unresolved URL target, not as a successful
match for the current graph. Mobile reports this to the user. Web startup
currently logs the unresolved target and continues with the tab-local or
stored graph so the app can still boot.
5. For Electron custom protocol URLs, the main process owns URL intake and
graph-name/graph-id resolution. This branch keeps the existing
renderer-mediated window creation path for custom protocol URLs and teaches
it to resolve graph ids through the registry.
6. For web and mobile URLs, the renderer owns URL intake and graph resolution.
It must switch to the target graph before applying the route target.
7. Once the target graph is loaded, apply the route target exactly once.
8. Do not store URL graph targets in `:git/current-repo` until the graph switch
succeeds.
9. Use `sessionStorage` for tab-local graph identity on web reloads that do not
carry a URL graph target. This preserves each browser tab's graph without
appending `graph-id` to every internal route.
## Supported URL Shapes
Canonical generated web URLs use graph ids:
- `https://logseq.com/?graph-id=<graph-uuid>`
- `https://logseq.com/page/<page-uuid>?graph-id=<graph-uuid>`
- `https://logseq.com/block/<block-uuid>?graph-id=<graph-uuid>`
Electron protocol URLs accept graph ids through the same graph identifier slot:
- `logseq://graph/<graph-uuid>`
- `logseq://graph/<graph-uuid>?page=<page-name-or-uuid>`
- `logseq://graph/<graph-uuid>?block-id=<block-uuid>`
- `logseq://new-window/<graph-uuid>?page=<page-name-or-uuid>`
Compatibility URLs remain accepted:
- `logseq://graph/<graph-name>`
- `logseq://graph/<graph-name>?page=<page-name>`
- `logseq://graph/<graph-name>?block-id=<uuid>`
- `logseq://graph/<graph-name>?file=<file>`
- `https://logseq.com/#/?graph=<graph-name>`
- `https://logseq.com/#/page/<page-name>?graph=<graph-name>`
Generated web URLs should use path routes instead of hash routes. The app may
continue accepting hash-routed URLs during migration, but new share/copy-link
helpers should emit path URLs with `graph-id`.
## Graph Registry
URL graph-id resolution must not open every graph database during startup. Keep a
small graph registry outside each graph DB.
1. Web and mobile store the graph registry in IndexedDB.
2. Electron stores the graph registry in
`(.getPath app "home")/.logseq/graphs.edn`.
3. Registry entries include at least:
- `:graph-id`
- `:repo`
- `:graph-name`
- `:local-graph-id`
- `:updated-at`
4. Do not persist a separate `:rtc-graph-id` in the registry. For remote graphs,
`:graph-id` is the remote graph UUID. For local-only graphs, `:graph-id` is
the local graph UUID.
5. Update the registry whenever a graph is created, opened, imported,
downloaded, uploaded to RTC, renamed, unlinked, or deleted.
6. If a registry lookup fails for a graph id, the app may run an explicit repair
path that enumerates local graph storage and opens graph DBs to rebuild the
registry. This repair path must not be the normal URL-open path.
## Electron Runtime Design
1. Store the registry at `(.getPath app "home")/.logseq/graphs.edn`.
2. Resolve Electron protocol graph identifiers through that registry before
falling back to existing repo/name lookup.
3. For shift-click in the graph list, open a new Electron window through the
existing `openNewWindow` IPC using the resolved repo. Electron windows still
use repo identity for window creation.
4. Keep a per-window pending target in Electron state only for work that cannot
be represented safely in the URL, such as a file redirect that requires graph
readiness. Avoid a single global `:window/once-graph-ready` callback because
concurrent windows can overwrite each other.
5. When the renderer reports `graphReady`, Electron should look up and consume
the pending target for that specific window id.
6. Existing error notifications for missing or unknown graphs remain, but they
should be sent to the selected/fallback window after graph resolution fails.
7. Direct main-process creation of route-aware graph windows remains follow-up
work; it should replace the renderer-mediated `openNewWindowOfGraph` path.
## Web Runtime Design
1. Add a parser that can read graph targets from path URLs and compatibility
hash URLs.
2. During frontend startup:
- parse the URL target;
- resolve `:graph-id` through the IndexedDB graph registry;
- resolve compatibility `:graph-identifier` against the known graph list when
no `:graph-id` is present;
- initialize from the resolved URL target when it resolves;
- otherwise use the tab-local `sessionStorage` graph target, then the stored
`:git/current-repo`, then the first linked repo.
3. During startup, apply resolved page/block URL targets after the target graph
is restored. Later in-app route changes with graph targets are follow-up
work.
4. During route generation:
- generated page routes include `graph-id` when the current graph id is
available, but startup/render-phase redirects tolerate a temporarily
unavailable graph id instead of crashing.
5. Generated web links from `frontend.util.url` should include `graph-id` on the
actual route.
6. Web "open in another tab" and shift-click from All graphs open `#/?graph-id=`
URLs for existing local graphs.
## Mobile Runtime Design
1. Use the same URL target parser and resolver as the web runtime.
2. Store and resolve the graph registry from IndexedDB.
3. Treat mobile app intents and universal links as URL sources that produce the
same target model.
## Consequences
- Resolvable graph-id URLs open the requested graph instead of silently using the
last current graph.
- Electron protocol URLs can resolve graph ids via the registry, but route-aware
direct window creation remains follow-up work.
- Web graph URLs become route-aware, so page and block links can switch graph
before navigating during startup and mobile deeplink handling.
- Web and mobile graph-id resolution is fast because it reads the graph registry
instead of scanning OPFS graph directories and opening graph databases.
- Electron graph-id resolution is fast because it reads
`(.getPath app "home")/.logseq/graphs.edn` instead of opening each graph
database.
- Bare web reloads preserve the tab's graph through `sessionStorage`, so two
tabs can remain on different graphs even when one tab changes the global
current graph.
- The implementation needs focused tests around URL parsing, graph resolution,
first-launch Electron deeplinks, second-instance deeplinks, and web/mobile
route changes.
## Follow-up Work
1. Add Electron main tests for first-launch and second-instance deeplinks.
2. Add renderer tests proving that web URLs switch graph before page/block
navigation after graph switches triggered by later route changes, not only
startup URLs.
3. Add graph registry read/write tests for IndexedDB and the Electron
`(.getPath app "home")/.logseq/graphs.edn` registry.
4. Replace `:window/once-graph-ready` with per-window pending targets.
5. Replace Electron custom protocol's renderer-mediated `openNewWindowOfGraph`
path with direct main-process route-aware window creation.

274
docs/adr/0018-rum-hooks-refactor.md Normal file
View File

@@ -0,0 +1,274 @@
# ADR 0018: Complete Rum Hooks Refactor
Date: 2026-05-29
Status: Proposed
## Context
Logseq frontend, mobile, and shui UI code still relies on Rum mixins and old
Rum lifecycle maps for state, subscriptions, query registration, event
listeners, memoization, DOM effects, and cleanup.
The current `master` tree has broad usage:
- 542 `rum/defc` components and 120 `rum/defcs` components under `src/main`
and `deps/shui/src`.
- 100 `rum/local` usages.
- Heavy use of `rum/reactive`, `rum/static`, `db-mixins/query`,
`mixins/container-id`, custom lifecycle maps, and `:should-update`.
- Shared mixin definitions in `src/main/frontend/mixins.cljs` and
`src/main/frontend/db_mixins.cljs`.
- Existing hook wrappers in `deps/shui/src/logseq/shui/hooks.cljs`, with
selected atom hooks re-exported from `src/main/frontend/rum.cljs`.
Mixins hide side effects and make component behavior harder to reason about.
The most sensitive cases are editor focus and cursor handling, block tree
rendering, reactive query refresh, plugin rendering, PDF/code integrations, and
mobile mount/unmount effects.
## Decision
Replace all existing Rum mixins and Rum lifecycle maps with hook-based component
code as a complete refactor.
The migration uses one explicit hook path for state, effects, subscriptions,
event listeners, query registration, memoization, and cleanup. The old mixin
namespaces must be removed by the end of the refactor, not left as wrappers or
compatibility shims.
The migration applies to:
- `src/main/frontend/`
- `src/main/mobile/`
- `deps/shui/src/`
The refactor is not complete until there is no behavior regression and no
performance regression on the affected UI paths. Any replacement hook must match
the old lifecycle timing, subscription semantics, cleanup behavior, and
memoization characteristics where those characteristics were observable.
Replacement code must use public React/Rum hook APIs only. It must not depend on
Rum internals such as `rum/*reactions*`, private dynamic bindings, or
reimplementations of `rum/reactive` under another name.
## Hook Foundation
Add hook APIs before migrating broad component areas. These hooks are the new
implementation path, not an adapter layer over the old mixin namespaces.
1. Add subscription hooks for render-time `rum/react`, `util/react`, and
`state/sub` usage:
- `frontend.state/use-sub`
- `frontend.state/use-sub-config`
- `frontend.state/use-sub-editing?`
- keep `frontend.rum/use-atom` and `frontend.rum/use-atom-in` as the
atom-level APIs.
2. Replace `db-mixins/query` with an explicit hook-backed query scope.
`db-mixins/query` currently binds `frontend.db.react/*query-component*` and
`*reactive-queries*` during render, then removes the component token on
unmount. The replacement should create a stable component token with
`use-ref`, register cleanup with `use-effect!`, and bind query tracking
while rendering the body.
3. Replace `rum/local` with:
- `rum/use-state` for values that affect rendering;
- `rum/use-ref` for mutable values that should not trigger rendering;
- `frontend.rum/use-atom` only where the state is intentionally atom-shaped
and shared with existing functions.
4. Replace event mixins with hook APIs:
- `use-event-listener`
- `use-window-keydown`
- `use-window-keyup`
- `use-hide-on-esc-or-outside`
- `use-modal-state`
5. Replace `mixins/container-id` with an explicit `use-container-id` hook that
derives from the same stable key currently built in the mixin.
## Lifecycle Mapping
Rum lifecycle maps are migrated directly to hooks:
- `:init`: use lazy `use-state`, `use-ref`, or pure `let` bindings. Do not move
side effects here.
- `:will-mount`: use `use-effect!` or `use-layout-effect!` with an empty
dependency vector.
- `:did-mount`: use `use-effect! []` for subscriptions, timers, and network
work; use `use-layout-effect! []` for focus, sizing, and DOM measurement.
- `:did-update`: use `use-effect!` with explicit dependencies. Use a first-render
guard only when old behavior intentionally skipped mount.
- `:will-update`: derive values during render, or use a previous-value ref plus
effect when side effects are required.
- `:will-unmount`: return cleanup from the effect that created the resource.
- `:wrap-render`: replace with explicit wrapper functions, macros, profiling
calls, or error-boundary components.
- `:should-update` and `rum/static`: replace with smaller memoized child
components, `React.memo`, or no memoization if the old optimization is not
meaningful.
- `:did-catch`: keep explicit class-based error boundaries where needed. React
hooks cannot replace render error boundaries.
## Migration Order
### Phase 1: Hook Foundation
1. Add state subscription, event listener, container id, and query-scope hooks.
2. Add focused tests for subscription cleanup and query cleanup.
3. Add a lint/search check that blocks new Rum mixin usage.
4. Migrate one small component from each behavior category:
- one `rum/local` component;
- one lifecycle component;
- one `db-mixins/query` component;
- one `rum/reactive` component.
### Phase 2: Low-Risk Leaf Components
Migrate components without editor focus, query refresh, plugin rendering, or PDF
behavior.
Good first targets:
- `src/main/frontend/components/avatar.cljs`
- `src/main/frontend/components/window_controls.cljs`
- `src/main/frontend/components/find_in_page.cljs`
- `src/main/frontend/components/selection.cljs`
- simple `src/main/mobile/components/*` components
- simple `deps/shui/src/logseq/shui/*` components
For each file, remove mixins, convert `rum/defcs` to `rum/defc` when the `state`
parameter is gone, and use hooks for subscriptions and effects.
### Phase 3: Shared Query Components
Migrate components that combine `rum/reactive` and `db-mixins/query`.
Targets include:
- page and block references;
- property values;
- query result views;
- journal and all-pages views;
- right sidebar block/page renderers.
Every `react/q` consumer should move from `util/react` or `rum/react` to a hook
subscription. Query cleanup must be verified by checking
`frontend.db.react` component maps after mount/unmount.
### Phase 4: Editor, Block Tree, and Navigation
Migrate the high-risk rendering core in small PRs by behavior:
1. Editor search and autocomplete lifecycles.
2. Editor input event listeners and cursor restoration.
3. Block content/edit switching.
4. Asset rendering and upload/download state.
5. Block container query scope and sidebar state.
6. Page mount/update side effects.
7. Header/sidebar collaboration and recent-highlight behavior.
Primary files:
- `src/main/frontend/components/editor.cljs`
- `src/main/frontend/components/block.cljs`
- `src/main/frontend/components/page.cljs`
- `src/main/frontend/components/header.cljs`
- `src/main/frontend/components/container.cljs`
Use `use-layout-effect!` for focus, selection, size, and scroll effects that
depend on mount/update ordering.
### Phase 5: Extensions and Heavy Integrations
Migrate extension components after core hooks are stable:
- `src/main/frontend/extensions/code.cljs`
- `src/main/frontend/extensions/fsrs.cljs`
- `src/main/frontend/extensions/latex.cljs`
- `src/main/frontend/extensions/pdf/core.cljs`
- `src/main/frontend/extensions/video/youtube.cljs`
These contain third-party DOM integrations, delayed rendering, timers, and
external cleanup. Prefer one integration per PR.
### Phase 6: Remove Old Mixin API
After all usage is gone:
1. Delete `src/main/frontend/mixins.cljs`; move any still-needed non-mixin
helper logic into hook-focused namespaces before deletion.
2. Delete `src/main/frontend/db_mixins.cljs`.
3. Remove unused `rum/reactive`, `rum/static`, `rum/local`, and lifecycle map
imports/usages.
4. Keep a permanent lint or CI search check blocking:
- `rum/reactive`
- `rum/static`
- `rum/local`
- `rum/defcs`
- lifecycle keys in Rum component declarations
- `frontend.mixins`
- `frontend.db-mixins`
## Consequences
- Component side effects become explicit and locally inspectable through hooks.
- Query refresh cleanup must preserve stable component identity, or query state
can leak.
- Subscription hooks need explicit dependency handling to avoid stale renders.
- Editor focus and cursor behavior can regress if layout-sensitive work moves to
normal effects.
- Removing `rum/static` and `:should-update` may change render frequency. Hot
paths need smaller memoized children or measured `React.memo` usage.
- Error boundaries remain a class-component concern and must be represented as
explicit boundary components.
- The refactor has a hard cleanup boundary: `frontend.mixins` and
`frontend.db-mixins` must not remain in the final tree.
- Performance-sensitive paths need before/after evidence when their
memoization, subscription, or render frequency changes.
## Verification
Each migration batch should run the narrowest relevant checks first, then broader
checks for shared behavior:
- focused hook tests for subscription and query cleanup;
- namespace-level unit tests where present;
- smoke compile for frontend and mobile targets;
- manual smoke checks for page load, block edit, slash command, page search,
block search, toggle block, drag block, right sidebar, asset block, query
blocks, PDF, code block rendering, flashcards, and YouTube embeds;
- app E2E tests for affected UI flows must pass. Run `bb test` from
`clj-e2e/`, or `bb -f clj-e2e/bb.edn test` from the repo root;
- CLI E2E tests must pass when a migration batch touches CLI-facing behavior,
command rendering, graph import/export behavior, or shared code used by the
CLI. Run `bb -f cli-e2e/bb.edn test --skip-build` from the repo root after
building when needed;
- render-count or profiling checks for hot paths whose `rum/static`,
`:should-update`, query subscription, or editor/block rendering behavior
changes;
- `bb dev:lint-and-test` before merging broad shared changes.
## Completion Criteria
- No `rum/reactive`, `rum/static`, `rum/local`, `rum/defcs`,
`frontend.mixins`, or `frontend.db-mixins` usage remains in application
component code.
- `src/main/frontend/mixins.cljs` and `src/main/frontend/db_mixins.cljs` are
removed from the repository.
- No Rum lifecycle maps remain in component declarations.
- Query subscriptions mount, refresh, and unmount without leaking entries.
- Editor, block tree, page navigation, right sidebar, properties, query results,
PDF, code, and mobile surfaces pass focused smoke checks.
- No known behavior regressions remain open against the affected UI paths.
- No known performance regressions remain open against the affected hot paths.
- Relevant app E2E tests pass for every affected UI flow.
- CLI E2E tests pass for batches that affect CLI-facing shared behavior.
- `bb dev:lint-and-test` passes.
## Follow-up Work
1. Implement the hook foundation and tests.
2. Add the lint/search guard for new mixin usage.
3. Migrate low-risk leaf components first.
4. Migrate query-scope components outside the editor/block tree.
5. Migrate the editor, block tree, page, header, and container components.
6. Migrate extension integrations.
7. Delete old mixin namespaces and enforce the permanent ban.
8. Capture before/after performance evidence for editor, block tree, page load,
right sidebar, query result, and PDF/code surfaces before declaring the
refactor complete.

147
docs/adr/0019-replace-rum-with-hsx.md Normal file
View File

@@ -0,0 +1,147 @@
# ADR 0019: Replace Rum With HSX
Date: 2026-06-01
Status: Accepted
## Context
ADR 0018 defined the goal of removing Rum mixins, Rum lifecycle maps, and
`rum/reactive` from Logseq's frontend component tree. During implementation we
also need to replace Rum component definitions themselves, because keeping Rum as
the component runtime leaves old lifecycle behavior, implicit reactivity, and
Rum-specific memoization in the rendering path.
Logseq now targets React 19. The old Rum runtime and the old React UMD asset
layout are a poor fit for that target:
- Rum lifecycle maps and mixins hide effect ordering and cleanup.
- `rum/reactive` depends on Rum's render-time reaction tracking.
- `db-mixins/query` relies on an implicit render scope to know which components
need to refresh after worker DB transactions.
- Rum-style Hiccup allowed several shapes that plain React element creation does
not accept directly.
- React 19 no longer provides the same browser UMD files that existing desktop
packaging expected.
The migration must keep behavior stable for editor input, block trees, right
sidebar rendering, properties, query results, settings, import, plugin surfaces,
PDF/code extensions, and mobile UI. It must also preserve hot-path render
performance that previously depended on Rum `:should-update` and `rum/static`.
## Decision
Replace Rum components with HSX components and use Logseq's fork of HSX as the
frontend component runtime.
All application component definitions should use `hsx/defc` instead of
`rum/defc`, `rum/defcs`, or plain `defn` replacements for component code. The
goal is a mechanical runtime migration with the smallest semantic change:
component names remain callable from ClojureScript, Hiccup remains the view
syntax, and hook-based state/effect/query APIs replace Rum mixins.
The runtime dependency is pinned to `https://github.com/logseq/hsx` from all
Logseq dependency roots that compile UI code. The fork is intentional and should
remain visible until its Logseq-specific compatibility changes are accepted
upstream or no longer needed.
## Query Reactivity
Rum `db-mixins/query`, `rum/reactive`, and `util/react` are replaced by explicit
query hooks.
The new query path is:
1. Build a query atom with `frontend.db.react/q`.
2. Subscribe to it from a component with `frontend.db.hooks/use-query`.
3. Let `use-query` register a stable component token with
`frontend.db.react/add-query-component!`.
4. Let worker DB refresh call `frontend.db.react/refresh!`, recompute affected
query atoms, and request re-render only for registered components.
5. Remove the component token on unmount so unused query cache entries can be
removed.
`use-query` must be called at a stable hook position. Components must not call
`db/sub-block`, `model/sub-block`, or `use-query` inside data-dependent loops,
conditionals, helper functions that run a variable number of times, or render
branches that can appear after the first render. If each list item needs its own
subscription, move that subscription into a keyed child component. If a parent
needs many entities, use one query hook that returns the collection.
This makes query reactivity explicit and keeps React hook order independent of
database result size.
## Logseq HSX Fork
The Logseq HSX fork exists to support the migration without broad application
rewrites. Its compatibility surface differs from upstream HSX in these ways:
- `hsx/defc` supports Logseq component metadata such as `^:private` and
`^:large-vars/*`.
- HSX components can be called as functions from ClojureScript component code,
matching the common Rum call style used throughout Logseq.
- Function-call component invocation preserves React component identity instead
of treating the component result as an arbitrary higher-order component.
- Legacy Hiccup child shapes used by Rum-era code are normalized so nested
vectors of children are treated as children instead of invalid React element
types.
- React keys and Clojure metadata on Hiccup forms are preserved for React list
reconciliation.
- Hook-backed memoization uses Clojure data equality where Logseq depends on it,
avoiding a separate adapter layer for immutable CLJS values.
- React 19 element creation is supported without depending on removed React
internals or old UMD asset paths.
- Known higher-order component values can be marked explicitly with `^:hoc`;
ordinary render-time anonymous functions should not be treated as HOCs.
These differences are compatibility requirements, not a license to preserve Rum
internals in application code. New component code should use direct HSX, React
hooks, and Logseq hook wrappers instead of adding more compatibility behavior to
the fork.
## Hook Rules
The migration follows normal React hook rules:
- Hooks are only called in `hsx/defc` component bodies or custom hooks.
- Hooks are called unconditionally and in the same order on every render.
- Data-dependent subscriptions are moved to keyed child components or combined
into one collection query.
- Render-time getters are used only outside component reactivity or when the
caller intentionally does not need re-rendering.
- Component views that need reactive app state use `state/use-sub`; non-view
helpers use getters.
- Component views that need reactive DB query results use `db.hooks/use-query`.
This replaces the old implicit Rum rule where `rum/reactive` could be introduced
inside helpers and still be attached to the current Rum component.
## Consequences
- Rum mixin namespaces and lifecycle maps can be removed instead of wrapped.
- Query refresh behavior becomes inspectable through
`frontend.db.react/*query-state`, `component->query-key`, and
`query-key->components`.
- Components that previously relied on `rum/static` or `:should-update` need
explicit smaller memoized children, stable keys, or collection-level queries.
- Code that used component functions as plain render helpers can keep doing so
through the Logseq HSX fork, but new code should prefer clear component
boundaries.
- Packaging must bundle React browser globals instead of copying old React UMD
files.
- Runtime compatibility issues should be fixed in HSX only when they represent
valid Logseq Hiccup/component semantics; application bugs should be fixed in
application code.
## Verification
Before merging the migration, verify:
- no `rum/defc`, `rum/defcs`, `rum/reactive`, `rum/static`, `rum/local`,
`frontend.mixins`, or `frontend.db-mixins` usage remains in application UI
code;
- no `db/sub-block`, `model/sub-block`, `state/use-sub`, or `use-query` calls
violate React hook ordering;
- query subscriptions clean up after unmount and refresh affected components
after worker DB transactions;
- block edit, split, merge, undo, redo, sidebar, page graph, properties, scoped
tags, settings, import, code blocks, PDF, plugin UI, and mobile surfaces keep
prior behavior;
- hot block-tree operations do not re-render unrelated sibling blocks;
- React console output has no hook-order, invalid element, invalid nesting, or
missing-key warnings for covered flows;
- `bb dev:lint-and-test` passes;
- app E2E tests pass, excluding only explicitly skipped RTC-related tests when
the test run is configured that way.

85
docs/adr/0020-journals-ui-stability.md Normal file
View File

@@ -0,0 +1,85 @@
# ADR 0020: Journals UI Stability
Date: 2026-06-02
Status: Accepted
## Context
The journals page renders one virtualized item per journal page. Each item can
contain a full page body, long block trees, embeds, today queries, scheduled and
deadline sections, and linked references.
Chrome verification showed the outer journals Virtuoso height and the main
scroll height changing while scrolling long journals. The same journal item also
changed height repeatedly after it was measured. Three concrete sources were
confirmed:
- `.journal-item` used bottom margin, which is outside the box measured by
ResizeObserver.
- Long journals could render a nested block-level Virtuoso inside the outer
journals Virtuoso, so two virtualizers measured the same scroll surface.
- After a long journal item was unmounted and remounted by the outer
virtualizer, dynamic content such as PDF embeds and linked references could
report a temporarily low height. That made the same measured journal shrink
and grow while scrolling back up.
Linked references must remain visible in journals. The fix cannot hide them or
rely on default collapse as a workaround.
## Decision
Use the journals Virtuoso as the only virtualized measurement owner for journal
items.
Journal item spacing is measured inside the item box. The previous
`pb-[64px] mb-[38px]` visual spacing is represented as `pb-[102px]`, removing
the margin from the measured boundary.
The outer journals Virtuoso uses `skipAnimationFrameInResizeObserver true` so
ResizeObserver updates are applied synchronously with the measurement source.
Block-level virtualization is disabled when rendering page blocks under
`:journals?`. For long journal pages, root blocks are still rendered
progressively so the initial paint remains bounded without nesting another
Virtuoso under the journal item.
Each journal item remembers its last measured height by repo and page id. When
the outer virtualizer remounts that item, the remembered height is used as a
temporary `min-height` reserve while dynamic content settles. The reserve is
released when the inner page content reaches the remembered height, when the
item receives editing input, or after a five-second fallback so real content
shrink can still be measured.
Linked references continue to render normally. Existing view logic already
disables linked-reference view virtualization under `:journals?`; this decision
keeps that behavior and verifies that linked-reference result bodies remain
visible.
## Consequences
The journals page has one scroll measurement owner for journal items, which
removes the feedback loop between nested virtualizers.
Long visible journal pages can eventually render all their visible root blocks,
but they do so progressively. This trades inner virtualization for stable outer
scroll geometry while preserving initial editing responsiveness.
Remounting a long journal no longer lets transiently low dynamic-content height
collapse a previously measured journal item during scroll restoration.
Dynamic sections such as linked references, today queries, scheduled/deadline
sections, embeds, and images can still change the measured journal item height.
Those changes are now handled by the outer journals Virtuoso without competing
with an inner block virtualizer or unmeasured margins.
## Verification
Focused clj-e2e coverage verifies:
- journal items have no bottom margin;
- long journals keep a single `#journals` Virtuoso scroller;
- long journals do not restart from a low measured height after remount;
- linked references remain visible in journals;
- the existing journals scroll-select-copy workflow still works.
Manual Chrome investigation on the affected page shape informed the decision.

260
docs/adr/0021-rfx-managed-frontend-state.md Normal file
View File

@@ -0,0 +1,260 @@
# ADR 0021: RFX-managed Frontend State
Date: 2026-06-03
Status: Accepted
## Context
ADR 0019 replaced Rum with HSX and moved the UI runtime onto React 19. The
frontend now has a React-compatible component runtime, but application state is
still managed by `frontend.state` through a large global atom plus many nested
atoms.
The current state layer has several responsibilities mixed together:
- `frontend.state/state` stores the main application map.
- Many values inside that map are themselves atoms, for example editor,
selection, RTC, PDF, command palette, scroll, and other high-churn UI refs.
- `frontend.state/use-sub` provides React subscriptions by wrapping the global
atom or a nested atom with `logseq.shui.hooks/use-atom`.
- `frontend.state/get-state`, `set-state!`, and `update-state!` are used across
handlers, components, workers, plugins, mobile code, and Electron integration.
- `frontend.state/pub-event!` publishes vectors to a `core.async` channel stored
at `:system/events`.
- `frontend.handler.events` consumes that channel and routes events through a
`defmulti`.
- Worker DB queries and component subscriptions use a separate reactive layer in
`frontend.db.react` and `frontend.db.hooks`.
- Several subsystems use Missionary flows or direct atom watches for runtime
state that is not just component state.
This makes UI reactivity explicit enough for React hooks, but the ownership is
still split between path cursors, nested atoms, a global event channel, and a
large set of imperative helpers. A code scan showed the size of the migration
surface:
- about 178 `state/use-sub` call sites under `src/main/frontend`;
- about 310 `set-state!` or `update-state!` call sites under
`src/main/frontend`;
- about 190 `pub-event!` call sites under `src/main/frontend`;
- about 102 `frontend.handler.events/handle` methods.
RFX is a better fit for the next step because it keeps the re-frame style
architecture while using React Context and hooks instead of Reagent. It provides
scoped app instances, `use-sub`, `dispatch`, `reg-sub`, `reg-event-db`,
`reg-event-fx`, effects, coeffects, snapshots, and a queue. Logseq has already
forked RFX and added root `deps.edn` support on `logseq/rfx` branch
`logseq/fork`, commit `d37aaceb37fcaf969c5cd04646dfdb985d65d3c2`.
The migration must not destabilize editor input, block trees, DB worker
transactions, RTC, plugin APIs, Electron IPC, mobile native flows, or
DataScript query subscriptions.
## Decision
Adopt RFX as the frontend application state and event runtime.
Add RFX to the main frontend dependency root from the Logseq fork:
```clojure
io.github.logseq/rfx
{:git/url "https://github.com/logseq/rfx.git"
:git/sha "d37aaceb37fcaf969c5cd04646dfdb985d65d3c2"}
```
Create a Logseq-owned RFX context, store, and registry. The context should be
provided near the root render in `frontend.core`, around `page/current-page`.
The RFX instance must be initialized during frontend startup before event
handling starts.
Replace `frontend.state` as the state owner in one coordinated migration.
Existing non-hook public entry points may remain as thin call-through APIs only
when they preserve caller compatibility without keeping state outside RFX:
- `get-state`
- `set-state!`
- `update-state!`
- `pub-event!`
- focused helpers such as `get-current-repo`, `set-current-repo!`,
`get-config`, editor helpers, selection helpers, and RTC helpers
Replace all `frontend.state/use-sub` and focused hook call sites with
`rfx/use-sub` or Logseq-owned domain hooks backed by RFX subscriptions. Do not
keep `frontend.state/use-sub` as a compatibility wrapper. `get-state`,
`set-state!`, `update-state!`, and focused non-hook helpers should read
snapshots or dispatch RFX events instead of mutating the old global atom.
Use one source of truth:
- Frontend application state is plain data in the RFX app db and is updated
through RFX events.
- Nested atoms must not remain as the representation for application state.
Convert nested atom state to plain values and path updates during the
migration.
- Missionary flows and direct atom watches must not remain as state propagation
paths for frontend application state. Replace them with RFX subscriptions,
events, effects, and coeffects.
- Imperative resources such as DOM nodes, editor handles, timers, external
worker objects, Electron handles, and plugin runtime handles stay outside the
RFX app db. They may be held in refs only when they are not renderable
application state and do not drive UI subscriptions directly.
Migrate every frontend state domain and its event handlers in the same change.
Do not land partial domain migrations. State-only migrations that still rely on
`frontend.handler.events/handle` for writes are rejected, and event-only
migrations whose render state still lives behind legacy path cursors are also
rejected.
`pub-event!` may remain as the public promise-returning compatibility API for
callers, but it should dispatch into RFX instead of writing to `:system/events`.
It must route to explicit `rfx/reg-event-db` or `rfx/reg-event-fx`
registrations and resolve or reject the deferred promise from the RFX event
result. Do not keep a bridge that invokes the existing
`frontend.handler.events/handle` multimethod.
Use RFX to replace flow-based frontend state propagation. DataScript execution
and worker-owned invalidation can remain worker responsibilities, but
UI-facing query result state and refresh orchestration should be exposed through
RFX subscriptions and events instead of Missionary flows or atom watches.
## Migration Shape
The migration should be one coordinated replacement of the frontend state and
event runtime.
1. Add the RFX dependency and create a small `frontend.rfx` namespace that owns
the app context, registry, initialization, dispatch helpers, and root
provider.
2. Move the initial plain app state map construction out of the `defonce state`
expression into a reusable function.
3. Move all frontend application state domains to plain RFX app-db values,
including route state, graph/current repo state, search, modals, sidebars,
theme flags, auth tokens, import/loading state, editor state, selection,
command palette, PDF, RTC UI state, plugin UI state, Electron UI state, and
mobile UI state.
4. Replace all `frontend.handler.events/handle` methods with explicit
`rfx/reg-event-db` or `rfx/reg-event-fx` registrations. Keep event ids
domain-scoped and register effects/coeffects for DB worker calls, RTC,
plugins, Electron IPC, mobile native calls, and async error capture.
5. Replace `:system/events` with RFX dispatch behind `pub-event!` and remove the
`core.async` event channel.
6. Replace Missionary flows and direct atom watches that propagate frontend
application state with RFX subscriptions, events, effects, and coeffects.
7. Replace all `frontend.state/use-sub` and focused state hook call sites with
`rfx/use-sub` or domain hooks backed by `rfx/reg-sub`.
8. Remove nested atoms, path cursors, and legacy hook paths for application
state. Keep refs only for imperative resources that do not represent
renderable state.
9. Update tests in the same change so every moved domain has coverage for state
reads, event-driven writes, async effects, and promise resolution.
Each domain should have a clear owner namespace, subscription ids, event ids,
effects, coeffects, and test coverage.
## Consequences
State ownership becomes explicit. Components subscribe through React-safe RFX
hooks, while non-view code can use RFX snapshots and dispatch against the same
instance.
Tests and isolated UI surfaces can create scoped RFX contexts instead of
mutating the global `frontend.state/state` atom. This matches the context model
introduced by the HSX migration.
The event system gains a uniform event/effect/coeffect shape and immediately
replaces the `core.async` channel plus `defmulti` dispatch path.
The migration cannot be a mechanical atom swap. Nested atom mutation does not
automatically notify RFX subscriptions. State must move to plain values, and
writes must move to RFX events.
Compatibility wrappers may remain only as public API aliases that call into RFX.
They must not retain separate storage, separate event dispatch, Missionary flow
state propagation, or direct atom-watch state propagation.
RFX effects that run async work must preserve existing error reporting behavior,
including `:capture-error` and promise rejection paths used by callers of
`pub-event!`.
## Rejected Alternatives
### Keep `frontend.state` As-is
This avoids migration cost but leaves Logseq with a custom subscription system
after the React 19 and HSX migration. It also keeps event routing separate from
state updates and does not give isolated RFX contexts for tests or component
development.
### Replace Only `pub-event!`
Moving events to RFX while leaving render state on path cursors would improve
event structure, but components would still subscribe through the custom atom
layer. It would not solve the state ownership problem.
### Migrate Domain-by-Domain
An incremental domain migration would leave two state and event runtimes active
for too long. That would preserve the current split ownership across RFX,
legacy path cursors, Missionary flows, atom watches, the `core.async` event
channel, and `defmulti` handlers. The migration should replace all frontend
application state domains and handlers together.
### Migrate State Without Event Handlers
Moving state keys to RFX while keeping their writes in
`frontend.handler.events/handle` would preserve the current split ownership in a
new form. RFX-managed state must be written through RFX events, so state and
event handlers should move together.
### Keep `frontend.state/use-sub` As A Wrapper
Keeping `frontend.state/use-sub` as a compatibility hook would preserve the old
subscription boundary and make it harder to see which code is using RFX
subscriptions directly. All hook call sites should move to `rfx/use-sub` or
domain hooks backed by RFX.
### Keep Missionary Flows For Frontend State
Keeping Missionary flows or atom watches as frontend state propagation would
leave RFX as only a partial store. Worker internals can still use their own
runtime mechanisms, but UI-facing state should be expressed through RFX
subscriptions and events.
### Keep DataScript Query Reactivity Outside RFX
Keeping `frontend.db.react` and `frontend.db.hooks/use-query` as a separate
UI-facing subscription layer would leave another flow-like state runtime beside
RFX. Query execution and worker invalidation may stay in the DB worker, but the
frontend-facing subscription and refresh contract should be mediated through
RFX.
## Verification
Before accepting the migration, verify:
- root rendering is wrapped in the Logseq RFX context;
- no `frontend.state/use-sub` call sites remain;
- direct `rfx/use-sub` and domain hook call sites obey React hook ordering;
- RFX-managed keys re-render components after `set-state!`, `update-state!`,
and event handlers update them;
- all frontend application state domains update through RFX events, with no
remaining legacy `defmethod handle` write path;
- no Missionary flow or direct atom watch remains as a frontend application
state propagation path;
- non-managed refs are limited to imperative resources and do not drive UI
subscriptions directly;
- `pub-event!` still returns a promise that resolves or rejects with the event
result;
- event errors are logged and reported through the existing capture path;
- graph switching, route changes, theme toggles, sidebars, modals, settings,
search, import progress, login state, plugin UI, Electron IPC, mobile tabs,
RTC indicator, editor editing, and block selection keep prior behavior;
- UI-facing DB query result subscriptions and refresh orchestration go through
RFX, while worker-owned query execution and invalidation keep prior behavior;
- no React hook-order warnings, invalid element warnings, or missed subscription
updates appear in Chrome;
- focused unit tests cover every state domain, event registration, effect,
coeffect, non-hook compatibility wrapper, and replaced flow path;
- `bb dev:lint-and-test` passes;
- relevant app or CLI E2E tests pass for graph switch, editor input, selection,
sidebar, settings, plugin, and sync surfaces touched by the migrated keys.

117
docs/adr/0022-remove-ui-db-and-refactor-editor-first.md Normal file
View File

@@ -0,0 +1,117 @@
# ADR 0022: Remove the UI DataScript DB and Refactor Editor First
Date: 2026-07-08
Status: Accepted
## Context
The db worker is the durable owner of graph data, but the renderer has still
kept a second DataScript connection as a UI read model. That split makes graph
reads harder to reason about because UI code can read stale renderer entities,
perform reverse lookups locally, or depend on lazy entity navigation that does
not exist when the worker database is the only graph database.
The renderer has used this UI DB through several shapes:
- direct `d/entity`, `d/pull`, `d/q`, `db/entity`, `db/pull`, and
`db-model/*` calls;
- reverse lookup attributes such as `:block/_alias` and `:block/_refs`;
- nested entity or pull-map reads such as reading fields from
`(:block/parent block)`;
- handler code that performs graph-backed editor calculations before
dispatching worker transactions.
The editor is the highest-risk migration surface because it combines block
tree reads, sibling and parent selection, paste, delete, merge, quick-add,
navigation, undo metadata, DOM focus, and outliner transactions. Removing the
UI DB without simplifying editor ownership would preserve the same complexity
behind async worker calls.
## Decision
Remove the renderer-owned DataScript database as a graph read model.
The db worker is the only owner of graph facts. UI code must not depend on
DataScript entities, lazy entity navigation, reverse lookup attributes, or
renderer-side pull maps for graph computation.
Refactor editor ownership first:
1. Move graph-backed editor calculations to worker APIs.
2. Add worker editor namespaces where editor-specific graph operations need a
clear home.
3. Keep UI editor handlers thin: they dispatch user intent, coordinate DOM and
focus state, and consume explicit worker payloads.
4. Return plain data from worker APIs with explicit fields needed by the UI.
Do not rely on entity identity, lazy nested refs, or reverse attributes.
5. Keep components focused on rendering and event dispatch. Component
migration comes after handler/editor contracts are explicit.
Do not add a renderer cache to replace the UI DB. If a cache becomes necessary,
it must have a narrowly documented owner, invalidation rule, and test case.
The default design is no cache.
Do not keep a compatibility layer that lets UI code continue to call
renderer-style DB helpers. Compatibility wrappers may remain only while their
call sites are being removed, and they must not hide a second graph database.
## Migration Shape
1. Add guard tests that count and reject direct UI DB reads, reverse lookups,
and nested entity or pull-map assumptions in frontend UI code.
2. Remove direct `d/entity`, `d/pull`, `d/q`, `db/entity`, `db/pull`,
`db-model/*`, and renderer `conn/get-db` reads from handlers and events.
3. Move editor calculations to worker APIs before converting component-heavy
surfaces.
4. Replace reverse lookup usage with worker-owned queries that return explicit
result fields.
5. Replace nested entity reads with explicit worker payloads, for example
parent ids, page ids, children, aliases, refs, or sibling blocks.
6. Keep outliner writes on worker-backed transaction paths and keep UI code
out of DB transaction assembly when the worker can own the calculation.
7. Convert components last so they render worker-shaped data and dispatch UI
events without graph reads.
8. Run focused tests after each migration batch, then run the broader test and
e2e suites when the UI DB removal is complete.
## Consequences
UI code becomes more explicit. It must ask the worker for the graph data or
editor decision it needs instead of navigating a local entity graph.
Worker APIs increase during the migration, especially around editor behavior.
That is intentional when the API represents a real graph-backed editor
operation. APIs should stay narrow and return plain data.
Editor behavior needs stronger focused coverage because many previously local
reads become async worker calls. Tests should cover insert, delete, merge,
paste, quick-add, sibling navigation, focused-root behavior, and reverse lookup
replacement paths.
Component migration is delayed until handler and editor worker contracts are
stable. This keeps rendering code simpler and avoids moving complex graph
calculation into components.
The removal may temporarily leave old helper namespaces in place while call
sites are drained, but the target architecture has no renderer-owned graph DB.
## Rejected Alternatives
### Keep a Renderer Read Cache
A renderer cache would preserve much of the old split-brain behavior and would
need invalidation logic for every worker transaction, sync update, import,
undo, redo, and rebase path. That adds complexity without solving ownership.
### Convert Components Before Editor
Component-first migration would move the easiest render reads first while
leaving the most complex graph-backed behavior in handlers. Editor behavior
would continue to depend on old assumptions, and later changes would force a
second pass through component contracts.
### Preserve Entity-shaped Worker Results
Returning entity-like maps with nested refs would keep the renderer coupled to
DataScript navigation semantics. Explicit worker payloads are easier to test,
serialize, and reason about across thread boundaries.