Establishes version control for the personal Logseq DB fork before starting the server-client architecture migration and domain features described in docs/agent-guide and the approved implementation plan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
873 lines
35 KiB
Markdown
873 lines
35 KiB
Markdown
# Logseq Runtime and Engineering Guide
|
|
|
|
Status: implemented
|
|
|
|
This is a compact, source-verified engineering guide for the runtime areas that
|
|
were previously documented across docs/agent-guide. It was checked against the
|
|
repository on 2026-08-24.
|
|
|
|
The guide explains durable architecture, data flow, invariants, failure behavior,
|
|
and verification entry points. It intentionally omits implementation schedules,
|
|
branch instructions, temporary measurements, review snapshots, and product
|
|
research. Source code, tests, generated OpenAPI, and active subsystem
|
|
documentation remain the detailed contracts behind this guide.
|
|
|
|
## Problem
|
|
|
|
Engineering knowledge about the CLI, db-worker-node, db-sync, rendering,
|
|
outliner performance, search, and Graph View was distributed across plans and
|
|
reports. The useful content was mixed with proposed work, obsolete paths, and
|
|
time-sensitive observations. Engineers need one short technical map that
|
|
preserves the valid constraints without treating historical plans as current
|
|
contracts.
|
|
|
|
## Decision
|
|
|
|
### System model
|
|
|
|
Logseq uses different runtime processes. Within one local runtime, the worker
|
|
database is authoritative for graph state. On Desktop, a graph has at most one
|
|
active local writer daemon; remote db-sync storage is a synchronized copy, not a
|
|
second local writer.
|
|
|
|
| Runtime | Main responsibility | Persistent database |
|
|
| --- | --- | --- |
|
|
| Browser renderer | UI, editor state, worker client | None directly |
|
|
| Browser db-worker | Graph transactions and queries | SQLite through OPFS |
|
|
| Desktop renderer | UI and remote persistence client | None directly |
|
|
| Electron main | Window and db-worker-node lifecycle | Filesystem coordination |
|
|
| db-worker-node | Graph transactions, queries, sync, search | Filesystem SQLite |
|
|
| CLI | Command parsing, orchestration, formatting | Uses db-worker-node |
|
|
| db-sync Worker/DO | Remote graph access and ordered synchronization | D1/DO SQLite and R2 |
|
|
|
|
Important ownership rules:
|
|
|
|
- The worker database is authoritative for graph state.
|
|
- Desktop and CLI use the same graph directory, lock protocol, and
|
|
db-worker-node runtime.
|
|
- The renderer must not create a second optimistic graph authority.
|
|
- Disk SQLite is the Desktop source of truth. OPFS export is not the Desktop
|
|
primary persistence path.
|
|
- A graph must not have two active writer daemons.
|
|
|
|
Primary source map:
|
|
|
|
| Area | Source |
|
|
| --- | --- |
|
|
| Worker database and thread APIs | src/main/frontend/worker/ |
|
|
| Browser persistence | src/main/frontend/persist_db/browser.cljs |
|
|
| Desktop remote persistence | src/main/frontend/persist_db/remote.cljs |
|
|
| Electron runtime manager | src/electron/electron/db_worker.cljs |
|
|
| Shared daemon lifecycle | src/main/logseq/cli/server.cljs |
|
|
| Node daemon | src/main/frontend/worker/db_worker_node.cljs |
|
|
| CLI | cli/ |
|
|
| db-sync server | deps/db-sync/src/logseq/db_sync/worker/ |
|
|
| Renderer DB subscriptions | src/main/frontend/db/ |
|
|
|
|
### Worker platform boundary
|
|
|
|
frontend.worker.platform is the worker capability boundary. The required adapter
|
|
sections are:
|
|
|
|
- env
|
|
- storage
|
|
- kv
|
|
- broadcast
|
|
- websocket
|
|
- crypto
|
|
- timers
|
|
- sqlite
|
|
|
|
Vector and embedding capabilities are present when the runtime supports semantic
|
|
search.
|
|
|
|
The browser adapter uses OPFS, browser storage, browser WebSocket, and the worker
|
|
postMessage channel. The Node adapter uses filesystem paths, Node WebSocket,
|
|
Node crypto/storage primitives, and node:sqlite DatabaseSync.
|
|
|
|
Business namespaces should depend on frontend.worker.platform instead of testing
|
|
the host runtime repeatedly. Missing required capabilities fail during platform
|
|
validation. Optional capabilities may be absent only where the feature itself is
|
|
optional.
|
|
|
|
Interactive requests use frontend.worker.ui-request:
|
|
|
|
1. The worker creates a unique request id.
|
|
2. The host receives a typed request and payload.
|
|
3. The host resolves or rejects that exact request.
|
|
4. Cancellation rejects outstanding requests during graph/runtime shutdown.
|
|
|
|
Browser flows may request UI input. Headless Node flows must use explicit input
|
|
or return a useful error. New worker business logic should not call renderer
|
|
functions directly. One non-Node search progress path still uses
|
|
frontend.worker.state/<invoke-main-thread; do not treat that exception as a
|
|
general worker API.
|
|
|
|
### db-worker-node daemon
|
|
|
|
db-worker-node is a graph-bound localhost daemon. It starts the same worker core
|
|
used by the browser, installs the Node platform adapter, opens the requested
|
|
repo, writes graph and server-list metadata, and becomes ready only after worker
|
|
initialization succeeds.
|
|
|
|
Client-facing endpoints:
|
|
|
|
| Method | Path | Purpose |
|
|
| --- | --- | --- |
|
|
| GET | /healthz | Readiness, bound repo, process, owner, revision, host, and port |
|
|
| GET | /v1/events | Server-Sent Events from worker broadcasts |
|
|
| POST | /v1/import-db-binary?repo=... | Stream a SQLite database import |
|
|
| POST | /v1/invoke | Invoke a worker thread API with Transit arguments |
|
|
| POST | /v1/shutdown | Graceful close and process shutdown |
|
|
|
|
The daemon also answers CORS preflight requests. It binds to `127.0.0.1` on an
|
|
OS-selected port and has no remote-bind option.
|
|
|
|
Most thread APIs require the first argument to identify the repo. The daemon
|
|
rejects:
|
|
|
|
- a missing or blank repo with HTTP 400;
|
|
- a repo that differs from the daemon's bound repo with HTTP 409.
|
|
|
|
Only explicitly listed process-level APIs may omit the repo. This binding keeps a
|
|
client from accidentally sending graph work to another daemon discovered from
|
|
stale metadata.
|
|
|
|
#### Graph lock and ownership
|
|
|
|
The three discovery records have deliberately different scopes:
|
|
|
|
- `db-worker.lock` records `repo`, `pid`, `lock-id`, and `owner-source`.
|
|
- `server-list` records only `pid` and `port` pairs.
|
|
- `/healthz` reports `repo`, readiness status, host, port, pid, owner source,
|
|
root directory, and runtime revision.
|
|
|
|
Neither the graph lock nor `server-list` is a readiness or revision authority.
|
|
|
|
Lifecycle behavior:
|
|
|
|
1. Resolve and validate the graph directory under the configured root.
|
|
2. Read the graph lock and remove it only when its owner process is absent.
|
|
3. Read `server-list`, discard entries whose pid is absent, and probe remaining
|
|
entries through `/healthz`.
|
|
4. Spawn only when neither a discovered graph server nor a graph lock exists.
|
|
5. Require both a graph lock and a discovered server, then wait for readiness.
|
|
6. Reuse the daemon only when its reported revision matches the requester.
|
|
7. On revision mismatch, stop that exact server, start once, and verify the new
|
|
revision; fail if stop, startup, or the second revision check fails.
|
|
|
|
A discovered process without the expected usable lock is not adopted as a valid
|
|
runtime; startup fails instead of returning an unowned endpoint.
|
|
|
|
Normal stop and restart operations respect owner boundaries. A proven revision
|
|
mismatch may replace the exact stale daemon across owner sources because the
|
|
requester cannot safely use it. Manual CLI cleanup remains limited to eligible
|
|
CLI-owned daemons.
|
|
|
|
#### Server list
|
|
|
|
root-dir/server-list is a discovery index of pid/port pairs. Mutations are
|
|
serialized through root-dir/server-list.lock.
|
|
|
|
The write protocol:
|
|
|
|
1. Create the lock with exclusive wx semantics.
|
|
2. Write lock owner metadata including pid, lock id, and timestamp.
|
|
3. Recover the lock only when its owner process is proven absent.
|
|
4. Read and normalize current entries while holding the lock.
|
|
5. Write a uniquely named temporary file.
|
|
6. Rename the temporary file over server-list atomically.
|
|
7. Remove the lock only when its lock id still matches.
|
|
|
|
Readers tolerate missing files and ignore malformed or duplicate entries. They
|
|
must still verify each entry through /healthz.
|
|
|
|
#### Electron graph switching and recovery
|
|
|
|
Electron owns graph-scoped runtimes in electron.db-worker. A window may bind to
|
|
one graph while another window keeps another graph runtime alive.
|
|
|
|
setCurrentGraph is metadata synchronization for the window graph path. It does
|
|
not release a worker. Runtime start/switch is handled by db-worker-runtime IPC,
|
|
and release is handled by explicit lifecycle or window close. Releasing from
|
|
setCurrentGraph can stop the newly selected runtime after a switch.
|
|
|
|
The renderer remote persistence client treats server-unavailable errors as a
|
|
runtime recovery signal. The current recovery threshold is one failed request.
|
|
Recovery is graph-scoped and must not silently switch the user to another graph.
|
|
|
|
The SSE client reconnects after 1000 ms by default and keeps scheduling another
|
|
attempt while the client remains connected; `disconnect!` stops that loop.
|
|
Request recovery is different: the first server-unavailable failure triggers at
|
|
most one recovery for the active runtime session. Session and repo checks stop a
|
|
stale recovery result from replacing the currently selected graph runtime.
|
|
|
|
#### Graph backups
|
|
|
|
Desktop and CLI share the graph backup layout:
|
|
|
|
~~~text
|
|
<root-dir>/graphs/<encoded-graph>/backup/<encoded-backup-name>/
|
|
db.sqlite
|
|
metadata.edn
|
|
~~~
|
|
|
|
Backup policy:
|
|
|
|
- Snapshot through SQLite backup APIs; do not copy a live database with a plain
|
|
filesystem copy.
|
|
- Include db.sqlite only.
|
|
- Exclude WAL/SHM sidecars, search-db.sqlite, and client-ops-db.sqlite.
|
|
- Reserve the destination directory atomically.
|
|
- Add a numeric suffix when a generated backup name already exists.
|
|
- Write metadata only after the database snapshot succeeds.
|
|
- Remove an incomplete reserved directory on failure.
|
|
- Resolve shared-helper backup paths as children of the graph backup root.
|
|
|
|
metadata.edn records schema version, name, repo, source, creation time, and
|
|
database path. Retention is source-specific so automatic pruning cannot delete
|
|
manual backups.
|
|
|
|
Electron runs automatic backups hourly for active graph windows, throttles the
|
|
same source, and retains 12 automatic versions. Manual Desktop and CLI backups
|
|
are not subject to that automatic retention policy.
|
|
|
|
The shared ClojureScript helper validates path containment. The current OCaml
|
|
CLI `backup restore --src` and `backup remove --src` paths only trim the supplied
|
|
name and concatenate it below the backup root; they do not reuse that containment
|
|
validator. Do not treat those two `--src` arguments as safe for untrusted input.
|
|
|
|
Markdown mirror output keeps page-level identity but does not emit block
|
|
database ids as HTML comments.
|
|
|
|
### db-sync architecture
|
|
|
|
db-sync separates four concerns:
|
|
|
|
- index-level graph, membership, access, and E2EE operations;
|
|
- graph-scoped ordered sync in a Durable Object;
|
|
- semantic REST operations for external clients;
|
|
- asset storage and signed download access through R2.
|
|
|
|
Key files:
|
|
|
|
| Concern | Source |
|
|
| --- | --- |
|
|
| Entry and dispatch | deps/db-sync/src/logseq/db_sync/worker/dispatch.cljs |
|
|
| Sync route registry | deps/db-sync/src/logseq/db_sync/worker/routes/sync.cljs |
|
|
| Semantic route registry/OpenAPI | deps/db-sync/src/logseq/db_sync/worker/routes/semantic.cljs |
|
|
| WebSocket protocol codec | deps/db-sync/src/logseq/db_sync/protocol.cljs |
|
|
| Sync handlers | deps/db-sync/src/logseq/db_sync/worker/handler/sync.cljs |
|
|
| Semantic handlers | deps/db-sync/src/logseq/db_sync/worker/handler/semantic.cljs |
|
|
| Storage | deps/db-sync/src/logseq/db_sync/storage.cljs |
|
|
| Client sync | src/main/frontend/worker/sync/ |
|
|
| MCP and ChatGPT entry | deps/db-sync/worker/entry.mjs |
|
|
|
|
Graph-scoped sync requests use /sync/<graph-id>/... and are forwarded to the
|
|
corresponding Durable Object.
|
|
|
|
#### Sync HTTP routes
|
|
|
|
The graph-scoped route registry contains:
|
|
|
|
| Method | Internal path | Purpose |
|
|
| --- | --- | --- |
|
|
| GET | /health | Graph sync readiness |
|
|
| GET | /pull | Transactions after a server t |
|
|
| GET | /checksum/diagnostics | Incremental and recomputed checksum details |
|
|
| GET | /snapshot/download | Snapshot download |
|
|
| GET | /snapshot/stream | Framed snapshot stream |
|
|
| DELETE | /admin/reset | Administrative graph reset |
|
|
| POST | /tx/batch | Ordered transaction batch |
|
|
| POST | /snapshot/upload | Streaming snapshot upload |
|
|
|
|
Use the route registry for the exact method/path contract. Public dispatch adds
|
|
graph id, authentication, access checks, and Durable Object forwarding.
|
|
|
|
#### WebSocket protocol
|
|
|
|
Messages are JSON maps. Transaction payloads inside the maps are Transit
|
|
encoded. UUIDs are serialized as strings.
|
|
|
|
Client messages:
|
|
|
|
| type | Important fields | Result |
|
|
| --- | --- | --- |
|
|
| hello | none | hello with current t and optional checksum |
|
|
| ping | none | pong |
|
|
| presence | editing-block-uuid | broadcast presence update |
|
|
| pull | since | pull/ok with t, txs, and optional checksum |
|
|
| tx/batch | t-before, txs, optional client-revision | tx/batch/ok or tx/reject |
|
|
|
|
The server also broadcasts online-users, presence, and changed notifications.
|
|
Malformed JSON, invalid since values, and unknown types return an error message.
|
|
|
|
Transaction batches use optimistic ordering:
|
|
|
|
- t-before must be a non-negative number.
|
|
- t-before must equal the server's current t.
|
|
- snapshot upload must be complete before normal transactions apply.
|
|
- a successful logical batch advances t and returns tx/batch/ok.
|
|
- a stale request returns tx/reject with the current t.
|
|
- a partial failure reports the new t plus successful and failed transaction
|
|
identifiers when available.
|
|
- peers receive one changed notification after the batch and checksum settle.
|
|
|
|
Clients must pull/rebase after stale rejection instead of retrying the same batch
|
|
blindly.
|
|
|
|
#### Semantic REST and MCP
|
|
|
|
The semantic API lives under /api/v1 and exposes operations for:
|
|
|
|
- graph discovery;
|
|
- pages and page references;
|
|
- blocks, recursive trees, and block moves;
|
|
- typed properties and batch property updates;
|
|
- journal capture;
|
|
- Task objects;
|
|
- tags and tagged objects;
|
|
- assets;
|
|
- graph search.
|
|
|
|
The semantic layer uses Logseq outliner and property semantics. It must not
|
|
implement mutations as arbitrary raw DataScript transactions.
|
|
|
|
Security and data rules:
|
|
|
|
- Every operation declares logseq/read or logseq/write OAuth scope.
|
|
- Graph access is checked before dispatch into the graph Durable Object.
|
|
- E2EE graphs fail closed because the server cannot interpret their content.
|
|
- Rate limiting happens before expensive Durable Object work.
|
|
- Property mutations use typed DB graph properties, not file-graph key:: text.
|
|
- Task creation uses the Task class and typed status/priority/date properties,
|
|
not Markdown TODO syntax.
|
|
- Asset uploads are limited to 100 MB and require an exact size plus SHA-256
|
|
checksum.
|
|
- The generated /openapi.json is the detailed external contract.
|
|
|
|
MCP is a thin integration over the semantic API. entry.mjs owns request wiring,
|
|
chatgpt_app.mjs owns app metadata, and chatgpt_asset_upload.mjs adapts asset
|
|
uploads. Do not create a second independent mutation API for MCP tools.
|
|
|
|
#### Authentication and E2EE state
|
|
|
|
Worker auth belongs in frontend.worker.state/*state under namespaced keys:
|
|
|
|
- :auth/id-token
|
|
- :auth/access-token
|
|
- :auth/refresh-token
|
|
- :auth/oauth-token-url
|
|
- :auth/oauth-domain
|
|
- :auth/oauth-client-id
|
|
|
|
frontend.worker.state/*db-sync-config contains transport configuration such as
|
|
:ws-url and :http-base. non-auth-db-sync-config strips auth-shaped keys before
|
|
storing transport config.
|
|
|
|
E2EE password rules:
|
|
|
|
- Never store plaintext passwords in cli.edn or db-sync config.
|
|
- Verify a supplied password against the encrypted private key before saving it.
|
|
- Do not overwrite a valid stored payload after a failed verification.
|
|
- Encrypt the persisted password with material derived from the refresh token.
|
|
- Browser storage uses the platform secret store.
|
|
- The Node platform prefers the OS keychain under the Logseq E2EE service.
|
|
- If keychain access fails, Node falls back to root-dir/kv-store.json.
|
|
- CLI E2E mode uses the KV store directly so tests do not mutate the keychain.
|
|
- Headless flows fail with a missing-password error and a command hint.
|
|
- Interactive browser/Desktop flows may request a password through ui-request.
|
|
|
|
CLI cloud tokens are stored separately in ~/logseq/auth.json. Runtime sync
|
|
receives auth state in memory.
|
|
|
|
#### Large operations
|
|
|
|
Large logical transactions must remain one ordered operation while bounding
|
|
individual allocations and requests.
|
|
|
|
Client upload behavior:
|
|
|
|
- frontend.worker.sync.apply-txs groups datoms without splitting an inseparable
|
|
datom group.
|
|
- Upload requests target a 5000-datom cap; one inseparable group may exceed it.
|
|
- Chunks are sent sequentially and preserve logical order.
|
|
|
|
Server apply behavior:
|
|
|
|
- A large entry targets 500-item apply chunks; one inseparable group may exceed
|
|
the target.
|
|
- Intermediate chunks suppress per-chunk checksum updates.
|
|
- The final checksum is updated from the complete logical transaction.
|
|
- The listener is installed and removed around the operation.
|
|
- Failure payloads preserve already successful transaction ids and the failed
|
|
transaction id when known.
|
|
|
|
Do not promise full rollback across already committed chunks. Preserve explicit
|
|
partial-success reporting so clients can reconcile deterministically.
|
|
|
|
#### Checksums
|
|
|
|
logseq.db-sync.checksum owns the checksum algorithm. It normalizes eligible
|
|
entities into stable tuples and supports both full recomputation and incremental
|
|
updates.
|
|
|
|
Checksum comparison is meaningful only when local and remote transaction
|
|
positions are synchronized. The client records the latest remote checksum and
|
|
logs a mismatch only after that readiness condition.
|
|
|
|
Investigation sequence:
|
|
|
|
1. Record graph id, local t, remote t, local checksum, and remote checksum.
|
|
2. Query /checksum/diagnostics for the server recomputation.
|
|
3. Recompute the local checksum from the same logical graph state.
|
|
4. Compare normalized tuple diagnostics, not raw SQLite row order.
|
|
5. Replay a captured transaction with the scripts exposed by
|
|
deps/db-sync/package.json when the incremental transition is suspect.
|
|
6. Distinguish a true state divergence from a stale t/checksum observation.
|
|
|
|
Snapshot upload may initialize a missing checksum only before transaction history
|
|
has advanced. It must not overwrite an existing different checksum.
|
|
|
|
#### Cycle repair utility
|
|
|
|
`logseq.db-sync.cycle` implements and tests reference-cycle repair for:
|
|
|
|
- :block/parent, cardinality one;
|
|
- :logseq.property.class/extends, cardinality many.
|
|
|
|
Its algorithm:
|
|
|
|
1. Collect entities touched by remote and local transaction reports.
|
|
2. Detect a reachable cycle with DFS across the configured attribute.
|
|
3. Prefer an edge whose current value differs from the corresponding value in
|
|
the remote transaction report.
|
|
4. Retract one edge on the cycle.
|
|
5. For block parent, add a safe remote parent or page root when possible.
|
|
6. For class extension, prefer the remote reference set or Root fallback.
|
|
7. Repeat until stable, with a 16-iteration safety cap.
|
|
|
|
Repair transactions use `:outliner-op :fix-cycle`, disable undo generation, and
|
|
set `:persist-op? false`.
|
|
|
|
Current integration boundary: the client remote-apply path in
|
|
`frontend.worker.sync.apply-txs` repairs duplicate outliner orders but does not
|
|
call `logseq.db-sync.cycle/fix-cycle!`. The utility and its unit tests therefore
|
|
must not be described as active client-side cycle repair.
|
|
|
|
#### Schema and failure policy
|
|
|
|
D1 schema changes require SQL migrations in deps/db-sync/worker/migrations.
|
|
Runtime schema initialization is not a substitute for a migration.
|
|
|
|
db-sync should fail closed on invalid auth, graph access, E2EE semantic access,
|
|
stale ordering, invalid snapshots, and malformed transaction payloads. Do not
|
|
introduce defaults that hide corrupt state.
|
|
|
|
### Incremental rendering and outliner transactions
|
|
|
|
The worker commits graph state once and derives renderer state from the committed
|
|
transaction.
|
|
|
|
Current flow:
|
|
|
|
1. An outliner/thread API transacts in the worker.
|
|
2. frontend.worker.db-listener receives the complete transaction report.
|
|
3. render-affected-keys derives exact resource invalidations.
|
|
4. render-delta builds one renderer delta.
|
|
5. The worker broadcasts :sync-db-changes.
|
|
6. frontend.db.subs applies the graph-scoped revision.
|
|
7. React hooks read immutable snapshots from frontend.db.hooks.
|
|
|
|
#### Render delta contract
|
|
|
|
A delta contains:
|
|
|
|
- graph-id;
|
|
- rev, derived from the authoritative database transaction position;
|
|
- optional operation id;
|
|
- complete block replacements keyed by block UUID;
|
|
- deleted block tombstones with revision and previous db/id when available;
|
|
- child membership patches per parent;
|
|
- affected render-resource keys.
|
|
|
|
Every replacement block carries :block/tx-id. A block cannot be both replaced
|
|
and deleted in one delta.
|
|
|
|
Child patches contain base-rev and rev plus ordered remove/upsert operations.
|
|
Membership changes are derived from:
|
|
|
|
- :block/parent;
|
|
- :block/order;
|
|
- :block/closed-value-property;
|
|
- :logseq.property/created-from-property;
|
|
- :logseq.property/deleted-at.
|
|
|
|
The renderer ignores a whole delta when its graph differs from the active graph
|
|
or its revision is not newer. For child patches, a mounted slot whose base
|
|
revision cannot be reconciled is marked stale and reloaded instead of being
|
|
merged speculatively.
|
|
|
|
#### Subscription store
|
|
|
|
frontend.db.subs owns one immutable external store. Important resource
|
|
identities include:
|
|
|
|
- [:block uuid]
|
|
- [:children parent-uuid]
|
|
- [:resource resource-key]
|
|
|
|
Subscriptions watch exact keys. Applying a delta updates cached blocks and
|
|
children, marks affected resources, and notifies only slots whose snapshot
|
|
changed.
|
|
|
|
Render-resource computation is registered in
|
|
frontend.worker.handler.render-resource.engine. Families include basic graph
|
|
lookups, property resources, views, and queries. The registry and
|
|
render-affected-keys must evolve together: every cached resource needs an exact
|
|
invalidation rule.
|
|
|
|
Checksum updates, db-sync persistence, Markdown mirror work, and search-index
|
|
updates run after the graph transaction commits. Each registered listener is
|
|
wrapped so synchronous or promise failures are reported through
|
|
`:capture-error`; they do not roll back the committed graph transaction.
|
|
|
|
#### Outliner performance guardrails
|
|
|
|
Outliner mutations use worker-owned transactions and publish their committed
|
|
result through the renderer delta path. Do not add a second optimistic block
|
|
tree or publish the same mutation independently from both a command response and
|
|
the DB listener. Page-window APIs should transfer the requested window, not a
|
|
full page, and graph-open work must not add graph-wide GC or `VACUUM`.
|
|
|
|
Useful transaction-pipeline measurements that are already logged include:
|
|
|
|
- performance id, outliner operation, and transaction count;
|
|
- renderer pipeline, delta construction, and broadcast time;
|
|
- checksum, persistence, sync-main, and registered-handler time;
|
|
- total listener time.
|
|
|
|
These measurements are diagnostics, not a fixed latency guarantee.
|
|
|
|
### CLI engineering guide
|
|
|
|
The shipped CLI is implemented in OCaml and compiled with Melange:
|
|
|
|
~~~text
|
|
cli/ OCaml
|
|
-> Dune/Melange
|
|
-> Vite bundle
|
|
-> cli/_build/default/dist/logseq-cli.js
|
|
-> scripts/stage-cli-runtime.mjs
|
|
-> static/logseq-cli.js
|
|
~~~
|
|
|
|
db-worker-node remains ClojureScript and is bundled separately to
|
|
dist/db-worker-node.js. There is no shadow-cljs logseq-cli target.
|
|
|
|
CLI source map:
|
|
|
|
| Concern | Source |
|
|
| --- | --- |
|
|
| Entry/orchestration | cli/lib/cli.ml |
|
|
| Command ids | cli/lib/command_id.ml |
|
|
| Registry/help/options | cli/lib/command_registry.ml |
|
|
| Parsing | cli/lib/cli_parse.ml |
|
|
| Actions | cli/lib/cli_action.ml |
|
|
| Worker transport | cli/lib/transport.ml |
|
|
| Daemon lifecycle | cli/lib/server_runtime.ml |
|
|
| Graph/import/export/backup | cli/lib/graph.ml |
|
|
| Sync | cli/lib/sync.ml |
|
|
| AgentBridge | cli/lib/agent.ml |
|
|
| Platform effects | cli/lib/platform/ |
|
|
| Public command reference | docs/cli/logseq-cli.md |
|
|
|
|
cli/spec contains interfaces and cli/test contains unit/parity tests. The CLI
|
|
uses the existing CLJS daemon; do not duplicate the worker database or sync
|
|
implementation in OCaml.
|
|
|
|
#### Configuration and output
|
|
|
|
With the default root, runtime files may include:
|
|
|
|
~~~text
|
|
~/logseq/
|
|
cli.edn
|
|
auth.json
|
|
kv-store.json
|
|
server-list
|
|
graphs/
|
|
~~~
|
|
|
|
Configuration precedence is global flag, then environment variable, then
|
|
cli.edn. Auth tokens and E2EE passwords are not CLI config fields.
|
|
|
|
Output modes:
|
|
|
|
- human is display-oriented and may truncate table titles by display width;
|
|
- json and edn preserve full values for scripts;
|
|
- namespaced keyword keys retain namespace/name form in JSON;
|
|
- UUIDs become strings in JSON;
|
|
- verbose and profile diagnostics go to stderr;
|
|
- normal structured stdout must not contain human warnings;
|
|
- sync progress is disabled by default for structured output unless explicitly
|
|
requested.
|
|
|
|
Executed commands are normalized internally to `Cli_result.t`, with status,
|
|
data or error, optional command id, output renderer, and optional exit code.
|
|
Human output may be a table or message; JSON and EDN serialize structured
|
|
results. `skill show` deliberately renders the skill body as raw human text.
|
|
|
|
#### Data operations
|
|
|
|
The CLI exposes list, show, search, query, upsert, move, and remove commands.
|
|
Important shared rules:
|
|
|
|
- Human list titles are CJK-aware and truncate by display width, not byte count.
|
|
- Structured list output never uses human truncation.
|
|
- Selectors must be unambiguous; commands reject conflicting id, UUID, ident,
|
|
name, and page options.
|
|
- list node tag/property filters use all-of semantics.
|
|
- Block and page references are rendered from UUID-backed graph data.
|
|
- show resolves nested block references with a depth cap.
|
|
- show page hierarchy is explicit and does not replace normal page content
|
|
display.
|
|
- Library is a special hierarchy root, but ordinary page hierarchy uses the
|
|
same bounded traversal and cycle protection.
|
|
- Mutations use worker outliner APIs rather than raw database editing in the
|
|
CLI.
|
|
|
|
SQLite export asks the worker to write the snapshot directly to the destination
|
|
path. EDN export forwards validated graph export options. SQLite import has a
|
|
top-level missing-graph check; EDN import may target an existing graph.
|
|
|
|
Current limitation: `graph backup restore` imports a SQLite backup but is not
|
|
included in the CLI's top-level missing-graph check. It must not be documented as
|
|
rejecting an existing destination until that constraint is implemented.
|
|
|
|
#### Sync behavior
|
|
|
|
CLI sync reuses worker-side sync logic:
|
|
|
|
- upload resolves or creates a remote graph identity before snapshot transfer;
|
|
- download starts a create-empty daemon and refuses a non-empty target;
|
|
- download uses a long-running command timeout;
|
|
- progress uses /v1/events and :rtc.log/download;
|
|
- asset download validates the asset entity and local checksum before enqueue;
|
|
- a valid local asset skips the remote request;
|
|
- a checksum mismatch requests a new download;
|
|
- sync config accepts transport keys only;
|
|
- E2EE password verification and storage remain worker-owned.
|
|
|
|
doctor validates the daemon bundle, root permissions, running server readiness,
|
|
and revision compatibility. When startup reports missing bundled modules, rebuild
|
|
db-worker-node and verify every asset in dist/db-worker-node-assets.json.
|
|
|
|
#### AgentBridge
|
|
|
|
cli/lib/agent.ml owns graph-scoped AgentBridge routing.
|
|
|
|
Implemented routing constraints:
|
|
|
|
- one bridge lock exists per graph and agent name;
|
|
- the agent name comes from configuration or hostname and must be non-empty;
|
|
- one master Codex session is started and resumed for task and comment dispatch;
|
|
- a task is routable only when it has a stable UUID, the Task class, TODO status,
|
|
the matching assignee, and no `:logseq.property.agent/session-id`;
|
|
- an ancestor task session id is passed to the master as inherited routing
|
|
context for descendant tasks.
|
|
|
|
Prompt templates from the AgentBridge registry are loaded and validated at
|
|
startup, and invalid custom templates are replaced with defaults. The current
|
|
dispatch path discards that returned template set and constructs hard-coded
|
|
master prompts instead. Custom registry templates therefore must not be
|
|
documented as affecting dispatched prompts until the dispatch path consumes
|
|
them.
|
|
|
|
The generated master instructions say that only the master may write graph
|
|
results, subagents are read-only, and a launched child session id must be saved
|
|
on the task. Those are prompt-level policies, not enforcement by the OCaml
|
|
bridge, so they must not be presented as runtime guarantees.
|
|
|
|
Keep graph queries, lock ownership, task routing, prompt construction, and
|
|
external process effects separately testable.
|
|
|
|
### Vector search
|
|
|
|
Desktop vector search uses:
|
|
|
|
- Electron embedding server:
|
|
src/electron/electron/embedding_server.cljs;
|
|
- default sentence-transformer model: all-MiniLM-L6-v2;
|
|
- embedding dimension: 384;
|
|
- vector context version: 3;
|
|
- zvec-backed Node vector index;
|
|
- SQLite keyword search plus vector results.
|
|
|
|
Each searchable block keeps at most one vector document, keyed by block UUID.
|
|
Both indexes derive their text from that block's search title; they do not
|
|
concatenate parent, sibling, or child context. SQLite FTS stores a normalized
|
|
form for ordinary blocks. The vector document embeds and retains the
|
|
pre-normalized title for ranking context-term boosts.
|
|
|
|
Constraints:
|
|
|
|
- Do not multiply vector count with rolling windows.
|
|
- Do not add parent/sibling traversal to full rebuild or incremental indexing
|
|
without a measured need and an explicit invalidation model.
|
|
- Preserve exact keyword matches ahead of weak vector hits.
|
|
- Current vector search returns at most 10 vector candidates and requires score
|
|
greater than 0.5.
|
|
- A completed vector rebuild writes model id, dimension, and context version to
|
|
index metadata. The current rebuild decision checks the SQLite search index
|
|
`PRAGMA user_version` or an explicit force flag; it does not compare the
|
|
persisted vector metadata on open. Do not claim automatic metadata-mismatch
|
|
rebuild until that check exists.
|
|
|
|
Incremental indexing derives affected searchable entities from the transaction.
|
|
It intentionally does not reindex adjacent siblings merely because a sibling was
|
|
reordered, and it does not expand a moved block update to the new parent. Page
|
|
hierarchy changes do reindex descendant pages whose indexed titles depend on the
|
|
hierarchy.
|
|
|
|
### Graph View
|
|
|
|
Graph View is a relationship browser built from worker graph data and rendered
|
|
with Pixi.
|
|
|
|
Source map:
|
|
|
|
| Concern | Source |
|
|
| --- | --- |
|
|
| Global UI and settings | src/main/frontend/components/graph.cljs |
|
|
| Page/block sidebar graph | src/main/frontend/components/page.cljs |
|
|
| Worker graph builder | src/main/frontend/worker/graph_view.cljs |
|
|
| React/Rum bridge | src/main/frontend/extensions/graph.cljs |
|
|
| Pixi renderer | src/main/frontend/extensions/graph/pixi.cljs |
|
|
| Layout and visibility logic | src/main/frontend/extensions/graph/pixi/logic.cljs |
|
|
| Styling | src/main/frontend/extensions/graph.css |
|
|
|
|
The renderer receives plain nodes and links. The worker emits string endpoint
|
|
ids and deduplicates links by directed endpoint pair. A later nonblank label can
|
|
replace an earlier label for the same pair, and class-extension links carry an
|
|
explicit `class-extends` edge type.
|
|
|
|
Relationship sources:
|
|
|
|
- :block/tags;
|
|
- :block/refs, lifted to owning pages where appropriate;
|
|
- user ref properties, labeled with the property title;
|
|
- :logseq.property.class/extends;
|
|
- :block/parent for page hierarchy in all-pages mode.
|
|
|
|
Global modes:
|
|
|
|
- tags-and-objects is the default;
|
|
- all-pages renders the visible page relationship graph.
|
|
|
|
tags-and-objects excludes property definitions, hidden/recycled entities,
|
|
excluded entities, and core internal tag classes. It renders tag/class nodes,
|
|
their objects, class-extension links, and visible ref-property links.
|
|
|
|
all-pages excludes hidden/recycled pages and internal property pages. Journals,
|
|
built-ins, excluded pages, and orphans are controlled by options. The large-graph
|
|
path activates at 10,000 page-name datoms, bounds visible ref links to 20,000,
|
|
then appends property, parent, and extension link tuples before applying a
|
|
20,000-link cap to the combined sequence. Because ref links come first,
|
|
structural links are not guaranteed to survive when the cap is exhausted.
|
|
|
|
The page graph centers on a page and includes references, mentions, tags, class
|
|
extensions, and selected second-order relationships. The block graph centers on
|
|
a block and connects incoming/outgoing referenced pages. Both use the same Pixi
|
|
page-mode renderer.
|
|
|
|
Renderer invariants:
|
|
|
|
- visible-node sets filter rendered nodes and links independently from the
|
|
source graph data;
|
|
- label selection is bounded and screen-cell based to reduce overlap;
|
|
- all-pages and tags-and-objects share the Pixi renderer plumbing;
|
|
- the renderer exposes explicit visibility updates and destroys its label,
|
|
edge-label, and application resources during cleanup.
|
|
|
|
Per-repo settings are stored under logseq.graph.settings.<repo>. They cover view
|
|
mode, selected tags, depth, link distance, grid layout, journal visibility, and
|
|
open settings groups. Time-travel cutoff is session state and is not persisted as
|
|
the graph's initial filter.
|
|
|
|
### Frontend state and UI dependencies
|
|
|
|
frontend.state remains the primary application state namespace. Focused modules
|
|
such as frontend.graph-tab own selected domains, but there is no general
|
|
src/main/frontend/state/ facade tree. Add a focused state namespace only when it
|
|
has real ownership, not to mirror a historical split plan.
|
|
|
|
The UI stack uses Base UI and Tailwind 4. Read package.json and current component
|
|
usage before changing dependency or styling conventions.
|
|
|
|
### Engineering workflow
|
|
|
|
Before changing a subsystem:
|
|
|
|
1. Read root and directory-specific AGENTS.md.
|
|
2. Locate the owning source, its callers, and its tests.
|
|
3. Confirm runtime boundaries: browser, renderer, Electron, Node worker, CLI, or
|
|
server.
|
|
4. Write or update the narrowest behavior test.
|
|
5. Preserve one authoritative data path.
|
|
6. Fail fast on invalid internal state.
|
|
7. Remove obsolete paths instead of adding compatibility fallbacks.
|
|
8. Run focused tests before broad verification.
|
|
|
|
Useful commands:
|
|
|
|
~~~sh
|
|
# Repository lint and unit tests
|
|
bb dev:lint-and-test
|
|
|
|
# One unit test
|
|
bb dev:test -v <namespace/testcase-name>
|
|
|
|
# Already-compiled CLJS namespaces
|
|
bb dev:run-test-namespaces -n <namespace>
|
|
|
|
# CLI
|
|
(cd cli && opam exec --switch=<compatible-ocaml-switch> -- dune runtest)
|
|
pnpm cli:release
|
|
pnpm db-worker-node:release:bundle
|
|
bb -f cli-e2e/bb.edn test --skip-build
|
|
|
|
# db-sync
|
|
pnpm --dir deps/db-sync test
|
|
pnpm --dir deps/db-sync test:chatgpt-app
|
|
pnpm --dir deps/db-sync test:large-op-128m
|
|
|
|
# Agent document validation
|
|
spec-dev-tool check --all
|
|
~~~
|
|
|
|
Use docs/cli/logseq-cli.md for the current CLI command reference and generated
|
|
OpenAPI for the current semantic API contract.
|
|
|
|
## Alternatives considered
|
|
|
|
### Keep the original plans as the technical reference
|
|
|
|
Rejected because proposed steps and old file paths are indistinguishable from
|
|
current behavior during search.
|
|
|
|
### Duplicate every current source contract in prose
|
|
|
|
Rejected because exhaustive route, command, and schema duplication would become
|
|
stale quickly. This guide records durable concepts and points detailed contracts
|
|
to executable registries and tests.
|
|
|
|
## Consequences
|
|
|
|
The guide preserves useful architecture, protocol, data-flow, performance, and
|
|
debugging knowledge in one document. Historical task sequencing, temporary
|
|
benchmarks, and obsolete proposals are intentionally absent.
|
|
|
|
The guide is concise enough to read as an orientation document, but engineers
|
|
must still inspect the named source and tests before modifying behavior.
|