Initial import of LogDarioSeq baseline

Establishes version control for the personal Logseq DB fork before
starting the server-client architecture migration and domain features
described in docs/agent-guide and the approved implementation plan.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-25 15:32:15 +02:00
commit 40631362b3
2016 changed files with 543898 additions and 0 deletions

View File

@@ -0,0 +1,134 @@
# Remove Legacy CLJS CLI
## Problem
Logseq has one shipped CLI implementation under `cli/`, built from OCaml with
Dune, Melange, and Vite, but it still retains most of the superseded Shadow
ClojureScript implementation under `src/main/logseq/cli/` and its tests under
`src/test/logseq/cli/`.
The release migration deliberately removed the `:logseq-cli` Shadow build and
made `cli/_build/default/dist/logseq-cli.js` the only producer of the staged
`static/logseq-cli.js` artifact. `package.json`, CLI E2E preflight, npm package
preparation, and Desktop runtime staging all use that OCaml/Melange artifact.
`scripts/test-cli-release-config.mjs` rejects any active Shadow CLI build path.
The remaining legacy command tree has no production entry point. Repository-wide
namespace searches find no production consumer outside its own implementation.
It contains 31 production files and about 14,800 lines, while 26 dedicated test
files add about 16,800 more lines. Keeping this parallel implementation makes
search results, tests, and maintenance work describe a binary that users cannot
build or run.
Not every namespace under `src/main/logseq/cli/` is obsolete. Electron,
db-worker-node, and worker handlers still import shared daemon and transport
helpers. Those imports are part of the active runtime and must remain.
## Proposal
Delete only the legacy CLJS CLI closure that is unreachable from active
production builds:
- `src/main/logseq/cli/command/`;
- `src/main/logseq/cli/auth.cljs`;
- `src/main/logseq/cli/commands.cljs`;
- `src/main/logseq/cli/completion_generator.cljs`;
- `src/main/logseq/cli/config.cljs`;
- `src/main/logseq/cli/format.cljs`;
- `src/main/logseq/cli/humanize.cljs`;
- `src/main/logseq/cli/main.cljs`;
- `src/main/logseq/cli/output_mode.cljs`;
- `src/main/logseq/cli/tree_text.cljs`; and
- `src/main/logseq/cli/uuid_refs.cljs`.
Delete their implementation-specific CLJS tests:
- `src/test/logseq/cli/command/`;
- `src/test/logseq/cli/auth_test.cljs`;
- `src/test/logseq/cli/commands_test.cljs`;
- `src/test/logseq/cli/completion_generator_test.cljs`;
- `src/test/logseq/cli/config_test.cljs`;
- `src/test/logseq/cli/format_test.cljs`;
- `src/test/logseq/cli/main_test.cljs`;
- `src/test/logseq/cli/output_mode_test.cljs`; and
- `src/test/logseq/cli/uuid_refs_test.cljs`.
Retain the active CLJS runtime boundary and its tests:
- `logseq.cli.common` and `logseq.cli.common.db-worker`, used by Electron and
worker handlers;
- `logseq.cli.root-dir`, used by db-worker-node and worker logging;
- `logseq.cli.server`, used by Electron runtime management;
- `logseq.cli.transport`, used by Electron database access;
- `logseq.cli.style`, used by db-worker-node;
- `logseq.cli.log` and `logseq.cli.profile`, used transitively by the active
server and transport helpers; and
- `logseq.cli.test-helper`, which is also consumed by db-worker-node tests.
Keep the shared helpers in their existing namespace for this decision. Moving
them to a neutral namespace would add unrelated churn without reducing the
duplicate CLI implementation.
Update `docs/cli/logseq-cli.md` so its query example points to the shipped OCaml
implementation in `cli/lib/query.ml` instead of `logseq.cli.command.query`.
Preserve `static/logseq-cli.js` as a generated artifact of the existing
OCaml/Melange staging flow; it is not part of the source deletion.
## Alternatives considered
### Retain the CLJS implementation as a reference
Rejected because the OCaml implementation, its unit/parity tests, CLI E2E
manifests, and the public command reference already own shipped behavior. A
second non-buildable implementation is a stale source of truth rather than an
executable reference.
### Delete every `logseq.cli` namespace
Rejected because Electron, db-worker-node, and worker production code still
consume the daemon, root-directory, transport, formatting, and shared database
helpers listed in the proposal.
### Move the retained helpers before deleting the command tree
Rejected for this decision because namespace relocation changes many active
runtime imports but is not required to remove the parallel CLI. It can be
considered separately if the `logseq.cli` name remains confusing after the
obsolete implementation is gone.
## Acceptance criteria
- The 31 unreachable CLJS source files and 26 implementation-specific test
files listed by the proposal are removed.
- The eight active CLJS production namespaces and their focused tests remain,
and repository-wide namespace search still accounts for every Electron,
db-worker-node, worker, and test-helper consumer.
- `shadow-cljs.edn`, root build scripts, CLI E2E preflight, npm package
preparation, and Desktop staging continue to identify
`cli/_build/default/dist/logseq-cli.js` as the sole CLI producer.
- `docs/cli/logseq-cli.md` no longer points readers to a deleted CLJS command
namespace.
- `node scripts/test-cli-release-config.mjs` passes.
- `pnpm --dir cli test` passes.
- `bb -f cli-e2e/bb.edn test --skip-build` passes against freshly staged CLI
and db-worker-node artifacts.
- Focused CLJS tests for `logseq.cli.common`, `logseq.cli.common.db-worker`,
`logseq.cli.log`, `logseq.cli.profile`, `logseq.cli.root-dir`,
`logseq.cli.server`, `logseq.cli.style`, and `logseq.cli.transport` pass.
- `bb dev:lint-and-test` passes.
## Risks
- A developer may have used the old CLJS namespaces manually even though they
have no build, package, or documented public entry point. This decision
intentionally gives up that unsupported workflow.
- Removing duplicate CLJS tests reduces coverage of the deleted implementation.
Shipped CLI behavior remains covered by `cli/test/` and `cli-e2e/`; active
shared CLJS runtime tests remain in place.
- The retained helpers continue to use the `logseq.cli` namespace, so the name
will describe runtime infrastructure as well as the historical location of
the removed CLI. Renaming them is outside this simplification.
## Questions
- None.

View File

@@ -0,0 +1,88 @@
# Remove Unused db-sync Cycle Repair
## Problem
`deps/db-sync/src/logseq/db_sync/cycle.cljs` implements approximately 370 lines
of client-side DataScript repair for `:block/parent` and
`:logseq.property.class/extends` cycles. Its approximately 220-line unit test
namespace is the only current consumer.
The client remote-apply path stopped requiring and calling `fix-cycle!` in
commit `905e131770` (`fix: no need to fix parent cycle for blocks`) on
2026-03-20. Current `frontend.worker.sync.apply-txs` repairs duplicate outliner
orders but has no cycle-repair call. Repository-wide searches find no other
production, script, documentation-contract, reflection, registry, or generated
source reference. The `@logseq/db-sync` package is private and does not export
the namespace as an external API.
The unused utility therefore preserves a behavior model, transaction metadata,
and tests that are not part of runtime behavior. The implemented runtime guide
must also carry a special warning explaining that the utility is not integrated.
## Proposal
Delete:
- `deps/db-sync/src/logseq/db_sync/cycle.cljs`; and
- `deps/db-sync/test/logseq/db_sync/cycle_test.cljs`.
Remove the `Cycle repair utility` subsection from
`docs/agent-guide/implemented/architecture/2026-08-24-logseq-runtime-and-engineering-guide.md`.
The runtime guide should continue to describe the active duplicate-order repair
where relevant, without documenting a removed test-only implementation.
Do not change `frontend.worker.sync.apply-txs`, transaction application order,
outliner semantics, or server-side validation. Generated
`deps/db-sync/worker/dist/worker-test.js` is excluded from source edits and will
be regenerated by the db-sync test build.
## Alternatives considered
### Reconnect `fix-cycle!` to remote apply
Rejected as a simplification because it would add cycle-repair transactions and
change synchronized graph behavior. If runtime evidence shows cycle repair is
needed, that work belongs to a separate bugfix or architecture decision with
new behavior tests.
### Keep the utility as a test-only design reference
Rejected because version control already preserves its history, while the
current source and test suite should describe executable behavior. Keeping it
also forces the runtime guide to distinguish implemented code from the active
path.
### Extract a generic graph-cycle library
Rejected because there is no second production consumer and the extraction
would relocate unused complexity instead of deleting it.
## Acceptance criteria
- The cycle repair source and its self-contained unit test namespace are
removed.
- Repository-wide search finds no `logseq.db-sync.cycle`, `cycle/fix-cycle!`, or
equivalent build/test registration reference in tracked source.
- `frontend.worker.sync.apply-txs` retains its current remote apply and
duplicate-order repair behavior unchanged.
- The implemented runtime guide no longer documents the removed test-only
utility or suggests that active client sync repairs reference cycles.
- `pnpm --dir deps/db-sync test` passes and regenerates its test bundle without
the deleted namespace.
- `bb lint:carve` passes.
- `bb dev:lint-and-test` passes.
## Risks
- The deleted implementation includes algorithms for two kinds of cycles, so a
future runtime bugfix cannot reuse it directly from the working tree. Its Git
history and focused tests remain recoverable from commits before this
decision.
- The removal assumes that the March 2026 production-path deletion was
intentional for both supported attributes. Reintroducing any client-side
repair would be behavior-changing and requires a new decision rather than a
reversal hidden inside this simplification.
## Questions
- None.

View File

@@ -0,0 +1,872 @@
# Logseq Runtime and Engineering Guide
Status: implemented
This is a compact, source-verified engineering guide for the runtime areas that
were previously documented across docs/agent-guide. It was checked against the
repository on 2026-08-24.
The guide explains durable architecture, data flow, invariants, failure behavior,
and verification entry points. It intentionally omits implementation schedules,
branch instructions, temporary measurements, review snapshots, and product
research. Source code, tests, generated OpenAPI, and active subsystem
documentation remain the detailed contracts behind this guide.
## Problem
Engineering knowledge about the CLI, db-worker-node, db-sync, rendering,
outliner performance, search, and Graph View was distributed across plans and
reports. The useful content was mixed with proposed work, obsolete paths, and
time-sensitive observations. Engineers need one short technical map that
preserves the valid constraints without treating historical plans as current
contracts.
## Decision
### System model
Logseq uses different runtime processes. Within one local runtime, the worker
database is authoritative for graph state. On Desktop, a graph has at most one
active local writer daemon; remote db-sync storage is a synchronized copy, not a
second local writer.
| Runtime | Main responsibility | Persistent database |
| --- | --- | --- |
| Browser renderer | UI, editor state, worker client | None directly |
| Browser db-worker | Graph transactions and queries | SQLite through OPFS |
| Desktop renderer | UI and remote persistence client | None directly |
| Electron main | Window and db-worker-node lifecycle | Filesystem coordination |
| db-worker-node | Graph transactions, queries, sync, search | Filesystem SQLite |
| CLI | Command parsing, orchestration, formatting | Uses db-worker-node |
| db-sync Worker/DO | Remote graph access and ordered synchronization | D1/DO SQLite and R2 |
Important ownership rules:
- The worker database is authoritative for graph state.
- Desktop and CLI use the same graph directory, lock protocol, and
db-worker-node runtime.
- The renderer must not create a second optimistic graph authority.
- Disk SQLite is the Desktop source of truth. OPFS export is not the Desktop
primary persistence path.
- A graph must not have two active writer daemons.
Primary source map:
| Area | Source |
| --- | --- |
| Worker database and thread APIs | src/main/frontend/worker/ |
| Browser persistence | src/main/frontend/persist_db/browser.cljs |
| Desktop remote persistence | src/main/frontend/persist_db/remote.cljs |
| Electron runtime manager | src/electron/electron/db_worker.cljs |
| Shared daemon lifecycle | src/main/logseq/cli/server.cljs |
| Node daemon | src/main/frontend/worker/db_worker_node.cljs |
| CLI | cli/ |
| db-sync server | deps/db-sync/src/logseq/db_sync/worker/ |
| Renderer DB subscriptions | src/main/frontend/db/ |
### Worker platform boundary
frontend.worker.platform is the worker capability boundary. The required adapter
sections are:
- env
- storage
- kv
- broadcast
- websocket
- crypto
- timers
- sqlite
Vector and embedding capabilities are present when the runtime supports semantic
search.
The browser adapter uses OPFS, browser storage, browser WebSocket, and the worker
postMessage channel. The Node adapter uses filesystem paths, Node WebSocket,
Node crypto/storage primitives, and node:sqlite DatabaseSync.
Business namespaces should depend on frontend.worker.platform instead of testing
the host runtime repeatedly. Missing required capabilities fail during platform
validation. Optional capabilities may be absent only where the feature itself is
optional.
Interactive requests use frontend.worker.ui-request:
1. The worker creates a unique request id.
2. The host receives a typed request and payload.
3. The host resolves or rejects that exact request.
4. Cancellation rejects outstanding requests during graph/runtime shutdown.
Browser flows may request UI input. Headless Node flows must use explicit input
or return a useful error. New worker business logic should not call renderer
functions directly. One non-Node search progress path still uses
frontend.worker.state/<invoke-main-thread; do not treat that exception as a
general worker API.
### db-worker-node daemon
db-worker-node is a graph-bound localhost daemon. It starts the same worker core
used by the browser, installs the Node platform adapter, opens the requested
repo, writes graph and server-list metadata, and becomes ready only after worker
initialization succeeds.
Client-facing endpoints:
| Method | Path | Purpose |
| --- | --- | --- |
| GET | /healthz | Readiness, bound repo, process, owner, revision, host, and port |
| GET | /v1/events | Server-Sent Events from worker broadcasts |
| POST | /v1/import-db-binary?repo=... | Stream a SQLite database import |
| POST | /v1/invoke | Invoke a worker thread API with Transit arguments |
| POST | /v1/shutdown | Graceful close and process shutdown |
The daemon also answers CORS preflight requests. It binds to `127.0.0.1` on an
OS-selected port and has no remote-bind option.
Most thread APIs require the first argument to identify the repo. The daemon
rejects:
- a missing or blank repo with HTTP 400;
- a repo that differs from the daemon's bound repo with HTTP 409.
Only explicitly listed process-level APIs may omit the repo. This binding keeps a
client from accidentally sending graph work to another daemon discovered from
stale metadata.
#### Graph lock and ownership
The three discovery records have deliberately different scopes:
- `db-worker.lock` records `repo`, `pid`, `lock-id`, and `owner-source`.
- `server-list` records only `pid` and `port` pairs.
- `/healthz` reports `repo`, readiness status, host, port, pid, owner source,
root directory, and runtime revision.
Neither the graph lock nor `server-list` is a readiness or revision authority.
Lifecycle behavior:
1. Resolve and validate the graph directory under the configured root.
2. Read the graph lock and remove it only when its owner process is absent.
3. Read `server-list`, discard entries whose pid is absent, and probe remaining
entries through `/healthz`.
4. Spawn only when neither a discovered graph server nor a graph lock exists.
5. Require both a graph lock and a discovered server, then wait for readiness.
6. Reuse the daemon only when its reported revision matches the requester.
7. On revision mismatch, stop that exact server, start once, and verify the new
revision; fail if stop, startup, or the second revision check fails.
A discovered process without the expected usable lock is not adopted as a valid
runtime; startup fails instead of returning an unowned endpoint.
Normal stop and restart operations respect owner boundaries. A proven revision
mismatch may replace the exact stale daemon across owner sources because the
requester cannot safely use it. Manual CLI cleanup remains limited to eligible
CLI-owned daemons.
#### Server list
root-dir/server-list is a discovery index of pid/port pairs. Mutations are
serialized through root-dir/server-list.lock.
The write protocol:
1. Create the lock with exclusive wx semantics.
2. Write lock owner metadata including pid, lock id, and timestamp.
3. Recover the lock only when its owner process is proven absent.
4. Read and normalize current entries while holding the lock.
5. Write a uniquely named temporary file.
6. Rename the temporary file over server-list atomically.
7. Remove the lock only when its lock id still matches.
Readers tolerate missing files and ignore malformed or duplicate entries. They
must still verify each entry through /healthz.
#### Electron graph switching and recovery
Electron owns graph-scoped runtimes in electron.db-worker. A window may bind to
one graph while another window keeps another graph runtime alive.
setCurrentGraph is metadata synchronization for the window graph path. It does
not release a worker. Runtime start/switch is handled by db-worker-runtime IPC,
and release is handled by explicit lifecycle or window close. Releasing from
setCurrentGraph can stop the newly selected runtime after a switch.
The renderer remote persistence client treats server-unavailable errors as a
runtime recovery signal. The current recovery threshold is one failed request.
Recovery is graph-scoped and must not silently switch the user to another graph.
The SSE client reconnects after 1000 ms by default and keeps scheduling another
attempt while the client remains connected; `disconnect!` stops that loop.
Request recovery is different: the first server-unavailable failure triggers at
most one recovery for the active runtime session. Session and repo checks stop a
stale recovery result from replacing the currently selected graph runtime.
#### Graph backups
Desktop and CLI share the graph backup layout:
~~~text
<root-dir>/graphs/<encoded-graph>/backup/<encoded-backup-name>/
db.sqlite
metadata.edn
~~~
Backup policy:
- Snapshot through SQLite backup APIs; do not copy a live database with a plain
filesystem copy.
- Include db.sqlite only.
- Exclude WAL/SHM sidecars, search-db.sqlite, and client-ops-db.sqlite.
- Reserve the destination directory atomically.
- Add a numeric suffix when a generated backup name already exists.
- Write metadata only after the database snapshot succeeds.
- Remove an incomplete reserved directory on failure.
- Resolve shared-helper backup paths as children of the graph backup root.
metadata.edn records schema version, name, repo, source, creation time, and
database path. Retention is source-specific so automatic pruning cannot delete
manual backups.
Electron runs automatic backups hourly for active graph windows, throttles the
same source, and retains 12 automatic versions. Manual Desktop and CLI backups
are not subject to that automatic retention policy.
The shared ClojureScript helper validates path containment. The current OCaml
CLI `backup restore --src` and `backup remove --src` paths only trim the supplied
name and concatenate it below the backup root; they do not reuse that containment
validator. Do not treat those two `--src` arguments as safe for untrusted input.
Markdown mirror output keeps page-level identity but does not emit block
database ids as HTML comments.
### db-sync architecture
db-sync separates four concerns:
- index-level graph, membership, access, and E2EE operations;
- graph-scoped ordered sync in a Durable Object;
- semantic REST operations for external clients;
- asset storage and signed download access through R2.
Key files:
| Concern | Source |
| --- | --- |
| Entry and dispatch | deps/db-sync/src/logseq/db_sync/worker/dispatch.cljs |
| Sync route registry | deps/db-sync/src/logseq/db_sync/worker/routes/sync.cljs |
| Semantic route registry/OpenAPI | deps/db-sync/src/logseq/db_sync/worker/routes/semantic.cljs |
| WebSocket protocol codec | deps/db-sync/src/logseq/db_sync/protocol.cljs |
| Sync handlers | deps/db-sync/src/logseq/db_sync/worker/handler/sync.cljs |
| Semantic handlers | deps/db-sync/src/logseq/db_sync/worker/handler/semantic.cljs |
| Storage | deps/db-sync/src/logseq/db_sync/storage.cljs |
| Client sync | src/main/frontend/worker/sync/ |
| MCP and ChatGPT entry | deps/db-sync/worker/entry.mjs |
Graph-scoped sync requests use /sync/<graph-id>/... and are forwarded to the
corresponding Durable Object.
#### Sync HTTP routes
The graph-scoped route registry contains:
| Method | Internal path | Purpose |
| --- | --- | --- |
| GET | /health | Graph sync readiness |
| GET | /pull | Transactions after a server t |
| GET | /checksum/diagnostics | Incremental and recomputed checksum details |
| GET | /snapshot/download | Snapshot download |
| GET | /snapshot/stream | Framed snapshot stream |
| DELETE | /admin/reset | Administrative graph reset |
| POST | /tx/batch | Ordered transaction batch |
| POST | /snapshot/upload | Streaming snapshot upload |
Use the route registry for the exact method/path contract. Public dispatch adds
graph id, authentication, access checks, and Durable Object forwarding.
#### WebSocket protocol
Messages are JSON maps. Transaction payloads inside the maps are Transit
encoded. UUIDs are serialized as strings.
Client messages:
| type | Important fields | Result |
| --- | --- | --- |
| hello | none | hello with current t and optional checksum |
| ping | none | pong |
| presence | editing-block-uuid | broadcast presence update |
| pull | since | pull/ok with t, txs, and optional checksum |
| tx/batch | t-before, txs, optional client-revision | tx/batch/ok or tx/reject |
The server also broadcasts online-users, presence, and changed notifications.
Malformed JSON, invalid since values, and unknown types return an error message.
Transaction batches use optimistic ordering:
- t-before must be a non-negative number.
- t-before must equal the server's current t.
- snapshot upload must be complete before normal transactions apply.
- a successful logical batch advances t and returns tx/batch/ok.
- a stale request returns tx/reject with the current t.
- a partial failure reports the new t plus successful and failed transaction
identifiers when available.
- peers receive one changed notification after the batch and checksum settle.
Clients must pull/rebase after stale rejection instead of retrying the same batch
blindly.
#### Semantic REST and MCP
The semantic API lives under /api/v1 and exposes operations for:
- graph discovery;
- pages and page references;
- blocks, recursive trees, and block moves;
- typed properties and batch property updates;
- journal capture;
- Task objects;
- tags and tagged objects;
- assets;
- graph search.
The semantic layer uses Logseq outliner and property semantics. It must not
implement mutations as arbitrary raw DataScript transactions.
Security and data rules:
- Every operation declares logseq/read or logseq/write OAuth scope.
- Graph access is checked before dispatch into the graph Durable Object.
- E2EE graphs fail closed because the server cannot interpret their content.
- Rate limiting happens before expensive Durable Object work.
- Property mutations use typed DB graph properties, not file-graph key:: text.
- Task creation uses the Task class and typed status/priority/date properties,
not Markdown TODO syntax.
- Asset uploads are limited to 100 MB and require an exact size plus SHA-256
checksum.
- The generated /openapi.json is the detailed external contract.
MCP is a thin integration over the semantic API. entry.mjs owns request wiring,
chatgpt_app.mjs owns app metadata, and chatgpt_asset_upload.mjs adapts asset
uploads. Do not create a second independent mutation API for MCP tools.
#### Authentication and E2EE state
Worker auth belongs in frontend.worker.state/*state under namespaced keys:
- :auth/id-token
- :auth/access-token
- :auth/refresh-token
- :auth/oauth-token-url
- :auth/oauth-domain
- :auth/oauth-client-id
frontend.worker.state/*db-sync-config contains transport configuration such as
:ws-url and :http-base. non-auth-db-sync-config strips auth-shaped keys before
storing transport config.
E2EE password rules:
- Never store plaintext passwords in cli.edn or db-sync config.
- Verify a supplied password against the encrypted private key before saving it.
- Do not overwrite a valid stored payload after a failed verification.
- Encrypt the persisted password with material derived from the refresh token.
- Browser storage uses the platform secret store.
- The Node platform prefers the OS keychain under the Logseq E2EE service.
- If keychain access fails, Node falls back to root-dir/kv-store.json.
- CLI E2E mode uses the KV store directly so tests do not mutate the keychain.
- Headless flows fail with a missing-password error and a command hint.
- Interactive browser/Desktop flows may request a password through ui-request.
CLI cloud tokens are stored separately in ~/logseq/auth.json. Runtime sync
receives auth state in memory.
#### Large operations
Large logical transactions must remain one ordered operation while bounding
individual allocations and requests.
Client upload behavior:
- frontend.worker.sync.apply-txs groups datoms without splitting an inseparable
datom group.
- Upload requests target a 5000-datom cap; one inseparable group may exceed it.
- Chunks are sent sequentially and preserve logical order.
Server apply behavior:
- A large entry targets 500-item apply chunks; one inseparable group may exceed
the target.
- Intermediate chunks suppress per-chunk checksum updates.
- The final checksum is updated from the complete logical transaction.
- The listener is installed and removed around the operation.
- Failure payloads preserve already successful transaction ids and the failed
transaction id when known.
Do not promise full rollback across already committed chunks. Preserve explicit
partial-success reporting so clients can reconcile deterministically.
#### Checksums
logseq.db-sync.checksum owns the checksum algorithm. It normalizes eligible
entities into stable tuples and supports both full recomputation and incremental
updates.
Checksum comparison is meaningful only when local and remote transaction
positions are synchronized. The client records the latest remote checksum and
logs a mismatch only after that readiness condition.
Investigation sequence:
1. Record graph id, local t, remote t, local checksum, and remote checksum.
2. Query /checksum/diagnostics for the server recomputation.
3. Recompute the local checksum from the same logical graph state.
4. Compare normalized tuple diagnostics, not raw SQLite row order.
5. Replay a captured transaction with the scripts exposed by
deps/db-sync/package.json when the incremental transition is suspect.
6. Distinguish a true state divergence from a stale t/checksum observation.
Snapshot upload may initialize a missing checksum only before transaction history
has advanced. It must not overwrite an existing different checksum.
#### Cycle repair utility
`logseq.db-sync.cycle` implements and tests reference-cycle repair for:
- :block/parent, cardinality one;
- :logseq.property.class/extends, cardinality many.
Its algorithm:
1. Collect entities touched by remote and local transaction reports.
2. Detect a reachable cycle with DFS across the configured attribute.
3. Prefer an edge whose current value differs from the corresponding value in
the remote transaction report.
4. Retract one edge on the cycle.
5. For block parent, add a safe remote parent or page root when possible.
6. For class extension, prefer the remote reference set or Root fallback.
7. Repeat until stable, with a 16-iteration safety cap.
Repair transactions use `:outliner-op :fix-cycle`, disable undo generation, and
set `:persist-op? false`.
Current integration boundary: the client remote-apply path in
`frontend.worker.sync.apply-txs` repairs duplicate outliner orders but does not
call `logseq.db-sync.cycle/fix-cycle!`. The utility and its unit tests therefore
must not be described as active client-side cycle repair.
#### Schema and failure policy
D1 schema changes require SQL migrations in deps/db-sync/worker/migrations.
Runtime schema initialization is not a substitute for a migration.
db-sync should fail closed on invalid auth, graph access, E2EE semantic access,
stale ordering, invalid snapshots, and malformed transaction payloads. Do not
introduce defaults that hide corrupt state.
### Incremental rendering and outliner transactions
The worker commits graph state once and derives renderer state from the committed
transaction.
Current flow:
1. An outliner/thread API transacts in the worker.
2. frontend.worker.db-listener receives the complete transaction report.
3. render-affected-keys derives exact resource invalidations.
4. render-delta builds one renderer delta.
5. The worker broadcasts :sync-db-changes.
6. frontend.db.subs applies the graph-scoped revision.
7. React hooks read immutable snapshots from frontend.db.hooks.
#### Render delta contract
A delta contains:
- graph-id;
- rev, derived from the authoritative database transaction position;
- optional operation id;
- complete block replacements keyed by block UUID;
- deleted block tombstones with revision and previous db/id when available;
- child membership patches per parent;
- affected render-resource keys.
Every replacement block carries :block/tx-id. A block cannot be both replaced
and deleted in one delta.
Child patches contain base-rev and rev plus ordered remove/upsert operations.
Membership changes are derived from:
- :block/parent;
- :block/order;
- :block/closed-value-property;
- :logseq.property/created-from-property;
- :logseq.property/deleted-at.
The renderer ignores a whole delta when its graph differs from the active graph
or its revision is not newer. For child patches, a mounted slot whose base
revision cannot be reconciled is marked stale and reloaded instead of being
merged speculatively.
#### Subscription store
frontend.db.subs owns one immutable external store. Important resource
identities include:
- [:block uuid]
- [:children parent-uuid]
- [:resource resource-key]
Subscriptions watch exact keys. Applying a delta updates cached blocks and
children, marks affected resources, and notifies only slots whose snapshot
changed.
Render-resource computation is registered in
frontend.worker.handler.render-resource.engine. Families include basic graph
lookups, property resources, views, and queries. The registry and
render-affected-keys must evolve together: every cached resource needs an exact
invalidation rule.
Checksum updates, db-sync persistence, Markdown mirror work, and search-index
updates run after the graph transaction commits. Each registered listener is
wrapped so synchronous or promise failures are reported through
`:capture-error`; they do not roll back the committed graph transaction.
#### Outliner performance guardrails
Outliner mutations use worker-owned transactions and publish their committed
result through the renderer delta path. Do not add a second optimistic block
tree or publish the same mutation independently from both a command response and
the DB listener. Page-window APIs should transfer the requested window, not a
full page, and graph-open work must not add graph-wide GC or `VACUUM`.
Useful transaction-pipeline measurements that are already logged include:
- performance id, outliner operation, and transaction count;
- renderer pipeline, delta construction, and broadcast time;
- checksum, persistence, sync-main, and registered-handler time;
- total listener time.
These measurements are diagnostics, not a fixed latency guarantee.
### CLI engineering guide
The shipped CLI is implemented in OCaml and compiled with Melange:
~~~text
cli/ OCaml
-> Dune/Melange
-> Vite bundle
-> cli/_build/default/dist/logseq-cli.js
-> scripts/stage-cli-runtime.mjs
-> static/logseq-cli.js
~~~
db-worker-node remains ClojureScript and is bundled separately to
dist/db-worker-node.js. There is no shadow-cljs logseq-cli target.
CLI source map:
| Concern | Source |
| --- | --- |
| Entry/orchestration | cli/lib/cli.ml |
| Command ids | cli/lib/command_id.ml |
| Registry/help/options | cli/lib/command_registry.ml |
| Parsing | cli/lib/cli_parse.ml |
| Actions | cli/lib/cli_action.ml |
| Worker transport | cli/lib/transport.ml |
| Daemon lifecycle | cli/lib/server_runtime.ml |
| Graph/import/export/backup | cli/lib/graph.ml |
| Sync | cli/lib/sync.ml |
| AgentBridge | cli/lib/agent.ml |
| Platform effects | cli/lib/platform/ |
| Public command reference | docs/cli/logseq-cli.md |
cli/spec contains interfaces and cli/test contains unit/parity tests. The CLI
uses the existing CLJS daemon; do not duplicate the worker database or sync
implementation in OCaml.
#### Configuration and output
With the default root, runtime files may include:
~~~text
~/logseq/
cli.edn
auth.json
kv-store.json
server-list
graphs/
~~~
Configuration precedence is global flag, then environment variable, then
cli.edn. Auth tokens and E2EE passwords are not CLI config fields.
Output modes:
- human is display-oriented and may truncate table titles by display width;
- json and edn preserve full values for scripts;
- namespaced keyword keys retain namespace/name form in JSON;
- UUIDs become strings in JSON;
- verbose and profile diagnostics go to stderr;
- normal structured stdout must not contain human warnings;
- sync progress is disabled by default for structured output unless explicitly
requested.
Executed commands are normalized internally to `Cli_result.t`, with status,
data or error, optional command id, output renderer, and optional exit code.
Human output may be a table or message; JSON and EDN serialize structured
results. `skill show` deliberately renders the skill body as raw human text.
#### Data operations
The CLI exposes list, show, search, query, upsert, move, and remove commands.
Important shared rules:
- Human list titles are CJK-aware and truncate by display width, not byte count.
- Structured list output never uses human truncation.
- Selectors must be unambiguous; commands reject conflicting id, UUID, ident,
name, and page options.
- list node tag/property filters use all-of semantics.
- Block and page references are rendered from UUID-backed graph data.
- show resolves nested block references with a depth cap.
- show page hierarchy is explicit and does not replace normal page content
display.
- Library is a special hierarchy root, but ordinary page hierarchy uses the
same bounded traversal and cycle protection.
- Mutations use worker outliner APIs rather than raw database editing in the
CLI.
SQLite export asks the worker to write the snapshot directly to the destination
path. EDN export forwards validated graph export options. SQLite import has a
top-level missing-graph check; EDN import may target an existing graph.
Current limitation: `graph backup restore` imports a SQLite backup but is not
included in the CLI's top-level missing-graph check. It must not be documented as
rejecting an existing destination until that constraint is implemented.
#### Sync behavior
CLI sync reuses worker-side sync logic:
- upload resolves or creates a remote graph identity before snapshot transfer;
- download starts a create-empty daemon and refuses a non-empty target;
- download uses a long-running command timeout;
- progress uses /v1/events and :rtc.log/download;
- asset download validates the asset entity and local checksum before enqueue;
- a valid local asset skips the remote request;
- a checksum mismatch requests a new download;
- sync config accepts transport keys only;
- E2EE password verification and storage remain worker-owned.
doctor validates the daemon bundle, root permissions, running server readiness,
and revision compatibility. When startup reports missing bundled modules, rebuild
db-worker-node and verify every asset in dist/db-worker-node-assets.json.
#### AgentBridge
cli/lib/agent.ml owns graph-scoped AgentBridge routing.
Implemented routing constraints:
- one bridge lock exists per graph and agent name;
- the agent name comes from configuration or hostname and must be non-empty;
- one master Codex session is started and resumed for task and comment dispatch;
- a task is routable only when it has a stable UUID, the Task class, TODO status,
the matching assignee, and no `:logseq.property.agent/session-id`;
- an ancestor task session id is passed to the master as inherited routing
context for descendant tasks.
Prompt templates from the AgentBridge registry are loaded and validated at
startup, and invalid custom templates are replaced with defaults. The current
dispatch path discards that returned template set and constructs hard-coded
master prompts instead. Custom registry templates therefore must not be
documented as affecting dispatched prompts until the dispatch path consumes
them.
The generated master instructions say that only the master may write graph
results, subagents are read-only, and a launched child session id must be saved
on the task. Those are prompt-level policies, not enforcement by the OCaml
bridge, so they must not be presented as runtime guarantees.
Keep graph queries, lock ownership, task routing, prompt construction, and
external process effects separately testable.
### Vector search
Desktop vector search uses:
- Electron embedding server:
src/electron/electron/embedding_server.cljs;
- default sentence-transformer model: all-MiniLM-L6-v2;
- embedding dimension: 384;
- vector context version: 3;
- zvec-backed Node vector index;
- SQLite keyword search plus vector results.
Each searchable block keeps at most one vector document, keyed by block UUID.
Both indexes derive their text from that block's search title; they do not
concatenate parent, sibling, or child context. SQLite FTS stores a normalized
form for ordinary blocks. The vector document embeds and retains the
pre-normalized title for ranking context-term boosts.
Constraints:
- Do not multiply vector count with rolling windows.
- Do not add parent/sibling traversal to full rebuild or incremental indexing
without a measured need and an explicit invalidation model.
- Preserve exact keyword matches ahead of weak vector hits.
- Current vector search returns at most 10 vector candidates and requires score
greater than 0.5.
- A completed vector rebuild writes model id, dimension, and context version to
index metadata. The current rebuild decision checks the SQLite search index
`PRAGMA user_version` or an explicit force flag; it does not compare the
persisted vector metadata on open. Do not claim automatic metadata-mismatch
rebuild until that check exists.
Incremental indexing derives affected searchable entities from the transaction.
It intentionally does not reindex adjacent siblings merely because a sibling was
reordered, and it does not expand a moved block update to the new parent. Page
hierarchy changes do reindex descendant pages whose indexed titles depend on the
hierarchy.
### Graph View
Graph View is a relationship browser built from worker graph data and rendered
with Pixi.
Source map:
| Concern | Source |
| --- | --- |
| Global UI and settings | src/main/frontend/components/graph.cljs |
| Page/block sidebar graph | src/main/frontend/components/page.cljs |
| Worker graph builder | src/main/frontend/worker/graph_view.cljs |
| React/Rum bridge | src/main/frontend/extensions/graph.cljs |
| Pixi renderer | src/main/frontend/extensions/graph/pixi.cljs |
| Layout and visibility logic | src/main/frontend/extensions/graph/pixi/logic.cljs |
| Styling | src/main/frontend/extensions/graph.css |
The renderer receives plain nodes and links. The worker emits string endpoint
ids and deduplicates links by directed endpoint pair. A later nonblank label can
replace an earlier label for the same pair, and class-extension links carry an
explicit `class-extends` edge type.
Relationship sources:
- :block/tags;
- :block/refs, lifted to owning pages where appropriate;
- user ref properties, labeled with the property title;
- :logseq.property.class/extends;
- :block/parent for page hierarchy in all-pages mode.
Global modes:
- tags-and-objects is the default;
- all-pages renders the visible page relationship graph.
tags-and-objects excludes property definitions, hidden/recycled entities,
excluded entities, and core internal tag classes. It renders tag/class nodes,
their objects, class-extension links, and visible ref-property links.
all-pages excludes hidden/recycled pages and internal property pages. Journals,
built-ins, excluded pages, and orphans are controlled by options. The large-graph
path activates at 10,000 page-name datoms, bounds visible ref links to 20,000,
then appends property, parent, and extension link tuples before applying a
20,000-link cap to the combined sequence. Because ref links come first,
structural links are not guaranteed to survive when the cap is exhausted.
The page graph centers on a page and includes references, mentions, tags, class
extensions, and selected second-order relationships. The block graph centers on
a block and connects incoming/outgoing referenced pages. Both use the same Pixi
page-mode renderer.
Renderer invariants:
- visible-node sets filter rendered nodes and links independently from the
source graph data;
- label selection is bounded and screen-cell based to reduce overlap;
- all-pages and tags-and-objects share the Pixi renderer plumbing;
- the renderer exposes explicit visibility updates and destroys its label,
edge-label, and application resources during cleanup.
Per-repo settings are stored under logseq.graph.settings.<repo>. They cover view
mode, selected tags, depth, link distance, grid layout, journal visibility, and
open settings groups. Time-travel cutoff is session state and is not persisted as
the graph's initial filter.
### Frontend state and UI dependencies
frontend.state remains the primary application state namespace. Focused modules
such as frontend.graph-tab own selected domains, but there is no general
src/main/frontend/state/ facade tree. Add a focused state namespace only when it
has real ownership, not to mirror a historical split plan.
The UI stack uses Base UI and Tailwind 4. Read package.json and current component
usage before changing dependency or styling conventions.
### Engineering workflow
Before changing a subsystem:
1. Read root and directory-specific AGENTS.md.
2. Locate the owning source, its callers, and its tests.
3. Confirm runtime boundaries: browser, renderer, Electron, Node worker, CLI, or
server.
4. Write or update the narrowest behavior test.
5. Preserve one authoritative data path.
6. Fail fast on invalid internal state.
7. Remove obsolete paths instead of adding compatibility fallbacks.
8. Run focused tests before broad verification.
Useful commands:
~~~sh
# Repository lint and unit tests
bb dev:lint-and-test
# One unit test
bb dev:test -v <namespace/testcase-name>
# Already-compiled CLJS namespaces
bb dev:run-test-namespaces -n <namespace>
# CLI
(cd cli && opam exec --switch=<compatible-ocaml-switch> -- dune runtest)
pnpm cli:release
pnpm db-worker-node:release:bundle
bb -f cli-e2e/bb.edn test --skip-build
# db-sync
pnpm --dir deps/db-sync test
pnpm --dir deps/db-sync test:chatgpt-app
pnpm --dir deps/db-sync test:large-op-128m
# Agent document validation
spec-dev-tool check --all
~~~
Use docs/cli/logseq-cli.md for the current CLI command reference and generated
OpenAPI for the current semantic API contract.
## Alternatives considered
### Keep the original plans as the technical reference
Rejected because proposed steps and old file paths are indistinguishable from
current behavior during search.
### Duplicate every current source contract in prose
Rejected because exhaustive route, command, and schema duplication would become
stale quickly. This guide records durable concepts and points detailed contracts
to executable registries and tests.
## Consequences
The guide preserves useful architecture, protocol, data-flow, performance, and
debugging knowledge in one document. Historical task sequencing, temporary
benchmarks, and obsolete proposals are intentionally absent.
The guide is concise enough to read as an orientation document, but engineers
must still inspect the named source and tests before modifying behavior.

View File

@@ -0,0 +1,158 @@
# Lower CLI OCaml Minimum to 5.1.1
## Problem
The CLI does not declare an OCaml version constraint. `cli/dune-project`
depends on `ocaml` without a lower bound, so the generated
`cli/logseq-cli.opam` file does the same. At the same time, every GitHub Actions
workflow that builds the CLI selects OCaml 5.4.0. The repository therefore
neither advertises the intended minimum nor verifies it.
The local `default` opam switch uses OCaml 5.1.1 and Dune 3.23.1. Those tool
versions are sufficient for the project metadata and the pinned CLI
dependencies, but a clean CLI test build currently stops while parsing
`cli/spec/core/cli_primitive.mli`:
```text
type keyword = Melange_edn_melange.(keyword t)
^
Error: Syntax error
```
`Module.(type_expression)` is a type-level local open, an OCaml 5.2 language
feature. The [OCaml 5.2 release notes](https://github.com/ocaml/ocaml/releases/tag/5.2.0)
identify local opens in type expressions as new in that release. Parsing all
122 checked-in `.ml` and `.mli` files with the OCaml 5.1.1 compiler found this
as the only parser incompatibility. The equivalent fully qualified type
expression
`Melange_edn_melange.keyword Melange_edn_melange.t` parses under OCaml 5.1.1.
The compiler version and the installed dependency set are separate concerns.
The current `default` switch does not have Melange or the private pinned CLI
packages installed, so it cannot build the CLI without the normal dependency
installation step. A different local switch using the same OCaml 5.1.1
compiler already contains the required dependency family, including Melange,
`melange-edn-melange`, `melange-transit-melange`, `humanize`, and `rrbvec`.
The pinned `melange-edn` and `melange-transit` packages declare OCaml
`>= 5.1.1`, `humanize` declares OCaml `>= 5.0`, and Melange publishes a
compiler-specific 5.1 package variant. No dependency inspected requires OCaml
5.2 or newer.
The goal is therefore to make OCaml 5.1.1 the explicit, continuously verified
CLI minimum so the existing `default` switch can be used after installing the
project dependencies. It is not to make a switch with missing dependencies
build the CLI without bootstrapping them.
## Proposal
Declare OCaml `>= 5.1.1` in the `logseq-cli` package dependencies in
`cli/dune-project`, then regenerate and commit `cli/logseq-cli.opam` so both
files expose the same constraint.
Replace the single OCaml 5.2-only type-level local open in
`cli/spec/core/cli_primitive.mli` with the fully qualified OCaml 5.1-compatible
type expression:
```ocaml
type keyword = Melange_edn_melange.keyword Melange_edn_melange.t
```
This is a syntax-only change to an existing alias. It does not change the
public type, runtime representation, or CLI behavior. Do not add a conditional
compatibility layer or maintain separate source paths for different compiler
versions.
Change `OCAML_VERSION` from 5.4.0 to 5.1.1 in all four workflows that build the
CLI:
- `.github/workflows/build.yml`;
- `.github/workflows/build-desktop-release.yml`;
- `.github/workflows/deps-cli.yml`; and
- `.github/workflows/cli-sync-stress.yml`.
This makes the minimum compiler the single compiler baseline for CLI unit,
E2E, dependency-update, stress, Desktop release, and npm release builds. The
released JavaScript artifacts will therefore also be produced with the 5.1.1
compiler-specific Melange variant.
Keep the normal local bootstrap command as:
```sh
cd cli
opam install . --deps-only --with-test --yes
opam exec -- dune runtest
```
The implementation must verify the dependency installation in a clean 5.1.1
switch rather than relying on packages already pinned in a developer switch.
If the existing `pin-depends` metadata does not bootstrap the private packages
in that clean environment, fix that single bootstrap path as part of this
decision; do not introduce a fallback dependency-install path.
## Alternatives considered
### Keep OCaml 5.4.0 as the minimum
Rejected because the only checked-in language dependency on OCaml 5.2 or newer
is a shorthand type expression with a direct 5.1-compatible spelling. The
inspected dependencies already support OCaml 5.1.1, and requiring a separate
5.4 switch adds local setup cost without providing a needed CLI capability.
### Declare `>= 5.1.1` without changing the source
Rejected because the declaration would be false. A clean OCaml 5.1.1 build
currently fails before type checking, and an existing Dune build directory can
hide the problem by reusing artifacts produced with a newer compiler.
### Keep the package unconstrained and rely on documentation
Rejected because opam could select an unsupported older compiler and CI would
not enforce the documented minimum. The package manifest is the authoritative
place for the compiler contract.
### Maintain separate source for OCaml 5.1 and newer compilers
Rejected because the fully qualified type is accepted by both compiler lines
and preserves the same public type. A compatibility branch or preprocessor
would add complexity without retaining a distinct capability.
## Acceptance criteria
- `cli/dune-project` declares `ocaml (>= 5.1.1)`, and the generated
`cli/logseq-cli.opam` file contains the equivalent constraint.
- `cli/spec/core/cli_primitive.mli` contains no type-level local open and
exposes the same `keyword` alias through fully qualified type constructors.
- In a clean opam switch whose compiler is exactly OCaml 5.1.1,
`opam install . --deps-only --with-test --yes` succeeds from `cli/` without
manually installing a newer compiler.
- A clean, non-cached `opam exec -- dune runtest` succeeds with OCaml 5.1.1.
- `opam exec -- pnpm cli:release` succeeds with OCaml 5.1.1, and
`node dist/logseq.js --help` succeeds against the staged artifact.
- `bb -f cli-e2e/bb.edn test --skip-build` passes against artifacts freshly
built with OCaml 5.1.1.
- `.github/workflows/build.yml`, `.github/workflows/build-desktop-release.yml`,
`.github/workflows/deps-cli.yml`, and
`.github/workflows/cli-sync-stress.yml` all select OCaml 5.1.1.
- The required GitHub Actions paths collectively exercise dependency
bootstrap, CLI unit and E2E tests, release builds, smoke tests, and stress
tests with OCaml 5.1.1 so a future use of OCaml 5.2-only syntax cannot merge
or ship unnoticed.
- The existing OCaml 5.4 build remains supported because the package contract
has no upper bound.
## Risks
- Building release artifacts with OCaml 5.1.1 selects a different
compiler-specific Melange package variant than OCaml 5.4.0. Unit, smoke, and
CLI E2E tests must validate the generated JavaScript rather than assuming the
compiler downgrade is syntax-only at the artifact level.
- The private dependencies in `pin-depends` track `main`, so their future
minimum OCaml version can move independently of this repository. The clean
5.1.1 dependency-install CI step is the guard against that drift.
- Incremental Dune artifacts built under another switch can conceal parser
incompatibilities. Minimum-version verification must use a clean build
directory.
- Installing the CLI dependencies into a multipurpose `default` switch changes
that switch's package set. This decision makes the compiler version
sufficient; it does not promise that installing the dependencies is free of
solver changes to unrelated packages already present in the switch.