d8996bd5e12e765bfe583f2fed654473c8d13f4b
19 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| d8996bd5e1 |
Fase B8+B9: stateful code kernel and live notes bound to external URLs
B8 - executable code blocks with a real kernel: - worker/handler/kernel.cljs: one persistent SCI interpreter session per page, reused across block evaluations so defs/state survive between runs. Deliberately scoped to the same privileges the existing client-side single-shot evaluator already had (no shell/fs access, no call-api). - Run button + result display wired into block.cljs's code-block UI. - New :logseq.property.code/result and /executed-at properties (65.38). B9 - live notes: pages tagged #LiveNote that fetch external content on a refresh schedule and render it via the render-resource pipeline: - worker/handler/live_notes.cljs: per-block credential storage, scheduled refresh (Node-only), thread-apis for manual refresh and auth. - worker/handler/render_resource/live_note.cljs: reads back fetched content, watching [:entity block-uuid] for reactive invalidation. - New :logseq.class/LiveNote + :logseq.property.livenote/* properties (65.39). Bugs found and fixed along the way while actually exercising page creation through page-handler/create! for the first time in a test (previously only pure-fn tests existed for CalDAV/Event/Contact): - logseq-property-namespaces (property.cljs) was missing the citation/event/ contact/caldav/livenote namespaces, causing auto-create-ontology to treat already-built-in properties as new user idents. - outliner/page.cljs's build-page-tx always added the generic #Page tag even when the caller tagged with a built-in class that extends #Page, which malli-schema rejects as conflicting; it also lost the property type when building ref-typed property values for new pages, breaking non-:default types like :number. - entity-util/page? never recognized Event/Contact/LiveNote as pages (only Journal/Whiteboard/Tag/Property/Page had dedicated recognition), so pages tagged with those classes alone weren't treated as pages anywhere in the app; added built-in-page-child? alongside the existing per-class checks. - caldav.cljs's upsert-page! passed a bare keyword as a tag where an entity was required, and two error logs passed raw error objects to glogi that triggered runaway recursion in this test environment; both fixed. |
|||
| 0158430489 |
Fase B7: CalDAV/CardDAV sync with Nextcloud, and a Calendar view type
- New built-in classes Event and Contact (schema 65.37), plus a Calendar View view type alongside table/list/gallery, backed by a new fixed month-grid calendar-view component in components/views.cljs reusing the existing scheduled/deadline properties. - New worker/handler/caldav.cljs: a deliberately-scoped CalDAV (VEVENT/VTODO) + CardDAV (VCARD) client against a single Nextcloud calendar/addressbook collection, with its own minimal iCalendar/vCard encoder-decoder and namespace-agnostic multistatus XML extraction (no external ical/rrule/xml dependency exists in this project). - RRULE <-> :logseq.property.repeat/* translation covers only the `plus` (advance-from-scheduled) repeat mode with a bare FREQ(;INTERVAL) rule -- dotted-plus/double-plus and anything with BYDAY/COUNT/UNTIL are left untranslated on purpose rather than mistranslated. - Credentials live in the platform keychain (save-secret-text!), never in the graph. Sync runs Node-only, every 15 minutes per open repo, started/stopped alongside the repo's lifecycle in db_core.cljs. Conflicts are last-write-wins via ETag: a 412 on push is skipped and retried after the next pull refreshes the stored ETag. |
|||
| dadeca7165 |
Fase B6: temporal relevance decay and vector-based "similar pages"
- combine-results now applies a bounded (never below 0.5x) exponential decay based on :block/updated-at, so more recently touched blocks/pages rank slightly higher for otherwise-equal matches. - New search/similar-blocks reuses the existing zvec index: fetch a block's own already-indexed embedding (new :fetch capability on open-vector-index, via zvec's fetchSync) and re-query the index with it instead of embedding a fresh query string. - Exposed as a :similar-pages render-resource (worker/handler/render-resource/similar.cljs), consistent with how linked/unlinked references are already wired, plus a raw :thread-api/search-similar-blocks for non-UI callers. - New "Similar pages" section on the page view, above the existing linked/unlinked references. |
|||
| d59aec181e |
Fase B5: temporal/linear graph view mode
Adds a third :temporal view mode alongside :tags-and-objects and :all-pages. The backend reuses build-all-pages-graph's node/link shape unchanged (every node already carries :block/created-at); only the client-side pixi layout differs, arranging nodes along a fixed chronological timeline (oldest first, staggered across a few lanes) instead of running the force simulation, following the existing grid-layout fixed-position precedent. |
|||
| b67f7e9f78 |
Fase B3+B4: built-in Project/Area tags and Citation class
B3: registers :logseq.class/Project and :logseq.class/Area as built-in page-like classes (the TODO in class.cljs already asked for this), so users can tag pages/journals with them out of the box. Fixed a markdown-mirror test whose "Custom Tag" fixture used to be titled "Project" and collided with the new built-in class. B4: adds :logseq.class/Citation plus /target and /context properties, following the :logseq.class/Comments pattern. Since target is a :node-typed ref, citation blocks automatically show up in the existing block/citation reference graph without any changes there. |
|||
| fcf41c7b41 |
Fase B2: give Contents root semantics alongside Library
Adds a shared common-config/contents-page-name constant and ldb/contents?/get-contents-page, mirroring the existing Library helpers. Contents already qualifies as an internal page, so the existing namespace-title split mechanism (e.g. "Contents/Proyectos") already parents pages under it via :block/parent, coexisting with Library rather than replacing it as the default root. |
|||
| c50511a7b5 |
Fase B1: chain journal pages with previous/next graph links
Adds :logseq.property.journal/previous and /next (built-in, hidden, single-cardinality node refs) and maintains them in logseq.outliner.page/create whenever a new journal page is created. Filling a gap between two already-linked journals re-links both neighbors around the new page instead of just appending. A migration backfills the chain across all existing journal pages ordered by day. |
|||
| db7f1fac7d |
Fase 9 (2/N): daemon-hosted MCP server, retire electron/mcp_server.cljs
Completes Fase 9's "consolidar MCP" half: the daemon now serves its own MCP endpoint at /mcp, exposing every semantic-routes operation as an MCP tool against the daemon's authoritative graph, and the old Electron-local MCP server is retired since it's now redundant. New: src/main/frontend/worker/mcp_server.cljs The CF-Worker's own MCP endpoint (deps/db-sync/worker/entry.mjs) generates its tools from the OpenAPI document via @cloudflare/codemode's openApiMcpServer, a package that only exists inside Cloudflare Workers, plus a pile of ChatGPT-App-specific tooling (asset image resources, upload_asset, Cognito protected-resource metadata) this daemon has no use for -- none of that ports to Node. So this registers one tool per operation by hand instead, using @modelcontextprotocol/sdk directly, the same pattern electron/mcp_server.cljs already used for its own 6 tools (registerTool with an explicit title/description/inputSchema/handler) -- just generated from semantic-routes' operation table and operation-docs (now public) instead of writing 35 one-off registrations by hand, so the MCP tool surface can't drift from the HTTP routes or the OpenAPI document. Each tool's Zod input shape is assembled from path params (auto-derived from the route's :param segments) plus a per-operation-id map of the extra query/body fields; a handful of shared fragments (pagination, time filters, the recursive block-tree shape, property-value/task-status selectors) cover the repetition across list/create/update operations. Each tool's handler makes a loopback HTTP call to this same daemon's own /v1/semantic/* route, forwarding the /mcp request's own Authorization header -- reusing the already-tested HTTP-layer logic (repo-error validation, lock-owner assertion, response shaping) instead of a second request path directly against the thread-api. Session/transport plumbing (handle-post/get/delete-request, the transports-by-session-id map) is adapted from electron/mcp_server.cljs's Fastify-based version: that one gets req/res already unwrapped-to-Node-primitives via `.raw` with the body pre-parsed by Fastify; this one is called straight from the daemon's own raw http.createServer callback, so req/res already are what StreamableHTTPServerTransport wants, and the body needs parsing here instead. No separate auth check either -- the daemon's existing bearer-token check on every route (request-authorized? in db_worker_node.cljs) already gates /mcp before this is reached. Wires POST/GET/DELETE /mcp into db_worker_node.cljs's make-server cond chain, alongside the existing /v1/semantic/* branch. Dependencies: adds @modelcontextprotocol/sdk and zod (zod/v3 -- the SDK doesn't work with zod 4's default export, same constraint electron/mcp_server.cljs already worked around) to the root package.json; previously only resources/package.json (Electron's own bundle) had them. Retired: electron/mcp_server.cljs and its wiring (electron/server.cljs's initialize-mcp-routes, its 3 POST/GET/DELETE /mcp route registrations, the :mcp-enabled? config field; components/server.cljs's "copy MCP URL" popup link; components/settings.cljs's entire AI settings tab, which had nothing in it besides the MCP-server toggle). Electron already talks to the daemon (Fase 3) rather than driving itself locally, so a second, separately-configured MCP surface bolted onto Electron's own local automation API server (which otherwise stays -- /api is a distinct, still-used concern) was redundant with the daemon's new one. Removed the 4 associated i18n keys (server/mcp-url-copied, settings/ai, settings.ai/enable-mcp-server[-desc]) from all ~26 locale dicts. New: src/test/frontend/worker/mcp_server_test.cljs Connects a real @modelcontextprotocol/sdk client over HTTP to a live daemon's /mcp endpoint: lists all registered tools, round-trips createPage/getPage and a recursive captureToToday block tree, and checks a tool call against a nonexistent page surfaces as MCP error content rather than throwing. 4 tests, 11 assertions. Verification: all 5 shadow-cljs targets compile with 0 warnings. Full `bb dev:test` run: the only failures are the same 7 pre-existing, sandboxed-network-related persist_db_test.cljs failures already documented in the Fase 8 and Fase 9 (1/N) commits -- identical test names, identical ECONNREFUSED/Invalid URL errors. Every other shard, including all three daemon-HTTP worker test files sharing one test.js process (db_worker_node_test.cljs, semantic_test.cljs, mcp_server_test.cljs), passes clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
|||
| 196de21731 |
Fase 9 (1/N): re-host the semantic API onto the daemon
Ports the 38-operation semantic API from deps/db-sync's Cloudflare Worker
handler (handler/semantic.cljs + routes/semantic.cljs) onto the daemon's
own authoritative conn, per the plan's Fase 9. deps/db-sync itself is
untouched -- this only adds the daemon-side equivalent.
New:
- worker/semantic_routes.cljs: route table + OpenAPI 3.1.0 document
generator, adapted from the CF-Worker's routes/semantic.cljs. Drops the
:graph-id URL segment and listGraphs operation entirely -- the daemon is
bound to a single repo per process (see db_worker_node.cljs's
bound-repo), unlike the CF-Worker's per-graph Durable Objects, so every
request implicitly targets the daemon's own bound repo. Security scheme
is a bearer token (the daemon's actual session auth) instead of OAuth2.
- worker/handler/semantic.cljs: the 38 operations themselves, ported from
the CF-Worker's handler/semantic.cljs. Adaptations from the original:
* ensure-conn! (lazy SQLite conn open) -> worker.state/get-datascript-conn
(the daemon already holds the authoritative conn for its bound repo).
* broadcast-change! calls dropped entirely -- any transact! against the
conn is already picked up by the listener worker.db-listener installs,
which broadcasts to every connected client, so nothing here needs to
push manually.
* HTTP body reads (p/let + body-clj) dropped -- bodies arrive already
JSON-parsed from the HTTP layer, and every operation is synchronous
(DataScript itself is synchronous), so none of this needs promises.
* http/bad-request, http/not-found, js/Response error helpers -> ex-info
with :status/:code, matching every other thread-api in this codebase.
* R2 asset storage -> platform/asset-write-bytes! (local filesystem),
using the <uuid>.<type> naming convention components/block/asset.cljs
already establishes. R2 signed URLs -> server-relative
/v1/assets/:repo/:name paths (already served, with ETag caching, by
the existing Fase 5 asset routes) since bearer-token auth already
gates those with the same ?token= fallback semantic GET routes need.
Registered as a single :thread-api/semantic-request thread-api
dispatched by handler keyword, mirroring how the CF-Worker original
dispatched via one big `case` in `handle`.
- db_worker_node.cljs: wires a /v1/semantic/* branch (JSON body parsed
for most operations; raw bytes read directly for assets-create,
mirroring the existing /v1/assets/:repo/:name PUT route) plus
/v1/semantic/openapi.json. Unlike /v1/invoke -- an internal RPC
transport whose transit-encoded results Logseq's own clients already
know how to decode -- /v1/semantic/* is the public HTTP API documented
by the OpenAPI doc and meant for arbitrary HTTP/MCP clients, so results
are decoded server-side into plain JSON, the same way the existing
/v1/assets/:repo/:name GET route already does for the other route with
external HTTP consumers.
- src/test/frontend/worker/semantic_test.cljs: exercises pages,
blocks/properties, capture, tags/tasks, search, the OpenAPI document,
401-without-auth, and 404-on-unknown-route against a live daemon over
real HTTP. 7 tests, 31 assertions.
Verification: all 5 shadow-cljs targets compile with 0 warnings. Full
`bb dev:test` run has zero regressions -- the only failures are the same
7 pre-existing, sandboxed-network-related persist_db_test.cljs failures
(ECONNREFUSED / Invalid URL) already documented as environmental in the
Fase 8 commit.
Found and fixed in-session: semantic_test.cljs originally set its own
LOGSEQ_DAEMON_USERNAME/PASSWORD env vars at namespace load time, colliding
with db_worker_node_test.cljs doing the same (both get bundled into the
same test.js process) -- whichever namespace loaded last won the race and
locked the other file's daemons out of their own bootstrapped credential,
cascading into ~150 unrelated failures. Fixed by reusing the same literal
credential constants instead of introducing a second one.
Known follow-up, not yet done: the plan's "consolidar MCP" half of Fase 9
-- an MCP JSON-RPC endpoint on the daemon (the CF-Worker's equivalent,
deps/db-sync/worker/entry.mjs, leans on @cloudflare/codemode's
openApiMcpServer plus ChatGPT-app-specific tooling that doesn't port to
Node) and the decision on electron/mcp_server.cljs's fate are still open.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|||
| 1e7e9a3c13 |
Fase 8 (2/N): delete the client-side RTC/E2EE/sync UI and finish the teardown
Removes the client-side half of the remote CRDT sync protocol left after
the worker-side deletion in
|
|||
| 565c2114be |
Fase 8 (1/N): delete the worker-side remote sync protocol
Deletes frontend.worker.sync/* wholesale (apply_txs, asset_db_listener,
assets, auth, client_op, const, crypt, download, handle_message,
large_title, log_and_state, presence, temp_sqlite, transport, upload,
util), worker/sync.cljs, worker/handler/sync.cljs, and
worker/ui_request.cljs -- the remote CRDT sync protocol and its E2EE
password-request plumbing, both made obsolete by the server-client
daemon architecture built in Fases 1-7. Deviates from the plan's KEEP
list on sync/presence.cljs and sync/transport.cljs: neither has any
caller left once the rest of the cluster is gone (confirmed by grep),
because Fase 4's WS resume-by-revision work already built its own
self-contained module (db_worker_node_ws.cljs) instead of adopting
these as the plan assumed it would.
This touched far more than a mechanical deletion:
- frontend.worker.sync.client-op wasn't sync-only. Its client_ops
SQLite table is also the idempotency marker for db_core.cljs's
built-in-property repair migration and the forward/inverse-outliner-
ops relay undo/redo reads back through history-action-ops-by-tx-id.
Trimmed (not deleted) into frontend.worker.local-tx-log, keeping only
the tx-entry/local-tx functions those two callers need and dropping
the genuinely sync-only surface (pending-upload counts, checksums,
graph-uuid, sync-conflict tracking, asset-op queues).
- frontend.worker.sync.apply-txs's apply-history-action! looked
sync-specific (it lived next to the remote-tx-apply/rebase machinery)
but turned out to be the *only* implementation ever registered for
worker-undo-redo/*apply-history-action! -- undo_redo.cljs calls it on
every db-transact undo/redo, local edits included, not just
remote-origin ones. Deleting it outright would have silently broken
undo/redo app-wide the next time anyone hit Cmd-Z. Ported the
general-purpose slice (replaying a recorded entry's forward/inverse
outliner ops against the current db state, plus the raw-tx-data
fallback for edits that bypass the structured outliner-op layer) into
a new frontend.worker.undo-redo.replay, along with persist-local-tx!
(derive history ops on every local commit, stash them in
local-tx-log, hand off to gen-undo-ops!) -- dropping only the
remote-rebase-specific rest (CRDT conflict resolution, upload-queue
bookkeeping). Wired into db_listener.cljs as a renamed
:persist-local-tx handler (was :db-sync, a name that stopped meaning
anything once sync was gone).
Caught by running the full existing undo_redo_test.cljs suite (30
deftests written for the old mechanism) against the new module after
fixing its test harness to use the real db_listener.cljs dispatch
instead of a hand-rolled listener -- all 29 tests port over passing
with no behavior change, which is the actual evidence this preserves
undo/redo correctness rather than just compiling.
- db_core.cljs's own 8 :thread-api/db-sync-* handlers (remote-graph-
download import, close-db/invalidate-search-db/recreate-lock
sync-specific variants) are deleted along with their sync-crypt/
sync-download calls; :thread-api/unsafe-unlink-db, :close-db, and
:release-access-handles keep their real (non-sync) bodies.
db_worker_node.cljs's non-repo-methods set drops from 22 to 7
entries (the 15 removed were all e2ee/db-sync/ui-request keywords
backed by deleted files).
- worker/handler/export.cljs's :thread-api/recompute-checksum-diagnostics
(and worker/db/validate.cljs's backing recompute-checksum-diagnostics,
which pulled in logseq-db-sync.checksum) compared a local checksum
against a remote one that no longer exists -- deleted outright, not
ported, since there's nothing left to diff against.
- worker/handler/block.cljs's :block.temp/sync-conflicts metadata
(backed by the deleted client-op/get-all-sync-conflicts) is gone;
its client-side consumer (a retry-upload/clear-conflicts UI in
components/block.cljs) is dead code now and still needs a follow-up
pass -- flagging here since worker-side cleanup alone can't remove
it without breaking that UI's thread-api calls at runtime.
Verified: app/electron/db-worker-node/mobile all compile clean. Full
worker test suite green (db_core_test 131, db_listener_test 11,
db_worker_test 8, db_worker_node_test 59, db_worker_node_lock_test 10,
local_tx_log_test 4, undo_redo_test 29, subs_test 50, subs_loader_test
2, config_test 6 -- 0 failures across all of them). The entire
src/test/frontend/worker/sync/ test directory (7 files) and
db_sync_test.cljs/db_sync_sim_test.cljs/node_sync_test.cljs (all
testing only-deleted namespaces) are deleted wholesale rather than
ported.
Known follow-up, not yet done: the client-side (frontend.handler.*,
frontend.components.*) RTC/E2EE/db-sync UI and its dead thread-api call
sites (components/block.cljs, components/repo.cljs,
components/settings.cljs, handler/user.cljs, handler/assets.cljs,
db/restore.cljs, handler/common/developer.cljs, and the
handler/db-based/{sync,rtc-flows,rtc-background-tasks}.cljs +
handler/events/rtc.cljs + components/rtc/* + db/rtc/* files themselves)
-- a separate commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|||
| c96b8a7a68 |
Fase 7: cache the daemon's per-invoke lock check, harden mobile server config storage
assert-lock-owner! ran on every write invoke and did fs.existsSync + fs.readFileSync + JSON.parse + a process.kill(pid, 0) syscall each time -- all synchronous on the Node event loop, once per request. Now cached in db_worker_node_lock.cljs: the lock file's content is cached and invalidated by watching its containing directory (survives an unlink+recreate, unlike watching the file path directly), and the liveness check is skipped entirely when the recorded pid is this process's own -- the dominant case, since assert-lock-owner! is the owner checking its own lock, not a stranger checking someone else's. Writes this process makes itself (create-lock!/update-lock!/ remove-lock!) invalidate the cache synchronously in the same call, so the only staleness window is for a lock file changed by another process -- a few ms until the fs.watch callback fires, which doesn't weaken the two-daemons-can't-both-hold-a-live-lock guarantee. Verified against a real daemon (5 consecutive write invokes, no lock-watch-failed warnings) and with two new tests: one proving repeated assert-lock-owner! calls keep agreeing with an unchanged lock, one proving a lock file rewritten by "another process" (a raw fs.writeFileSync bypassing update-lock!'s own invalidation) is picked up within the polling window instead of being served stale forever. All 8 pre-existing lock tests and all 61 db-worker-node-test tests still pass. frontend.config's db-worker-server-url/token getters/setters stay synchronous localStorage reads everywhere (retrofitting their many callers to await an async native store is out of scope), but on native mobile, writes now also mirror into frontend.mobile.secure-storage (OS keychain/keystore) as a durability backstop -- WebView localStorage can be evicted under storage pressure or not survive a reinstall. mobile.init/init! now fires config/<restore-server-config-from-secure- storage! at startup to copy a surviving value back into localStorage if it came back empty. Guarded so this is a true no-op outside native mobile (including under Node tests, where it's additionally gated by the same util/node-test? check the existing localStorage calls already used) -- verified via the :mobile shadow-cljs target compiling clean and the full test suite (subs-test, config-test, db-worker-node-test, db-worker-node-lock-test, subs-loader-test) staying green. iOS ATS and Android cleartext traffic, also called out in this phase of the plan, turned out to already be configured permissively (NSAllowsArbitraryLoads in Info.plist, cleartextTrafficPermitted in network_security_config.xml) -- no changes needed there. Investigated whether db-core's .remoteInvoke serializes concurrent invocations, per the plan's own open question: it does not -- frontend.common.thread-api/remote-function dispatches straight into the handler with no per-repo queueing, so two concurrent write invokes to the same repo could in principle interleave at any await point inside the handler. Deliberately did NOT add a write queue for this: the plan gates that on Fase 0 telemetry showing real invoke overlap in practice, and doing it without solid evidence of need risks a mistake in read-vs-write method classification for uncertain benefit -- querying methods (:thread-api/q, :thread-api/get-render-snapshots, etc.) share the same repo-scoped lock-check path Fase 6 just optimized for read latency, and serializing those too would undo that work. Left as an explicitly open question rather than guessed at. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
|||
| 2164bbdf1e |
Fase 6: close the subs-loader in-flight dedup gap, raise batch limits, add /v1/invoke-batch
Fase 0's telemetry work had already flagged that subs-loader deduped
load! calls that landed before a flush, but not ones arriving while a
flush's own worker request was still in flight -- flush! cleared *batch
via take-batch! before the request it just sent even resolved, so a
load! for the same slot in that window fired a duplicate request. Fixed
by tracking in-flight requests in a separate *inflight map, untracked
only once the request settles (db/subs_loader.cljs), with a real e2e
test proving the dedup and that a later load! after settlement does
fire a fresh request (db/subs_loader_test.cljs).
Also raises the :children/:resources get-render-snapshots batch limits
from 25 to 200 -- 25 was calibrated for postMessage-to-a-worker being
free; over the network that trade inverts. Two existing subs-test.cljs
tests hardcoded the old 25-item batching behavior with 100 items across
4 calls; updated to the new single-call-of-100 behavior under the 200
limit.
Deliberately did NOT widen db/subs.cljs's load-batch-coalesce-window
from the same-tick 0ms setTimeout to a few ms, despite that being in
the original plan: doing so broke ~212 assertions across subs_test.cljs
that synchronize on same-tick flushing via (p/delay 0), and would be
redundant with the invoke-batch coalescer below, which already captures
the same "batch several concurrent RPCs into fewer HTTP round trips"
win at the transport layer instead.
New POST /v1/invoke-batch on the daemon (db_worker_node.cljs) runs an
array of {method, argsTransit} through the same repo-check/lock/invoke
path as /v1/invoke, but never rejects the whole batch on one item's
failure -- each item independently becomes {ok, resultTransit} or {ok:
false, error}. Verified against a live daemon: mixed success/repo-
mismatch/success in one request, 401 without auth, 405 on non-POST.
On the client, frontend.persist-db.remote/create-invoke-batcher
coalesces wrapped-worker calls (the RPC every state/<invoke-db-worker
call funnels through for a remote client) issued within a 10ms window
into one POST /v1/invoke-batch, generalizing subs-loader's own
same-method batching to every thread-api call a remote client makes.
Two invokes only land in the same batch when neither is awaited before
the other fires, so this can't reorder anything a caller depends on
sequentially.
New db_worker_node_test.cljs coverage: invoke-batch mixing success and
repo-mismatch failure in one request, and invoke-batch auth/method
enforcement.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|||
| 902bf7f568 |
Fase 5: serve and accept assets over HTTP on the daemon
Closes the gap the plan flagged as blocking the sync deletion: a web/mobile client had no way to read or write asset bytes at all (only db-sync's now end-of-life handlers touched platform/asset-*). Adds GET/PUT /v1/assets/:repo/:name on db_worker_node.cljs, backed by three new thread-api ops (asset-read-bytes/write-bytes/stat) that just call the existing Node platform.cljs asset-*-bytes! functions -- reusing the same JSON+transit /v1/invoke path already proven for export-db-binary, so no new binary RPC mechanism was needed. GET responses carry a real Content-Type (guessed from extension), a sha1 ETag, and immutable Cache-Control -- assets are content-addressed by filename, so this is safe to cache forever once fetched. Auth accepts ?token= as a fallback for a direct <img src>/browser navigation, same rationale as the SSE interim token, but as a genuine fallback alongside the Authorization header rather than replacing it (an early version of this made query-token exclusive with header auth, which 401'd every JS-driven fetch -- caught before committing, real end-to-end curl testing against a live daemon also caught a 500-instead-of-404 on missing assets and a never-firing 304 from a hyphenated JS property access that needed `aget` instead of `.-`). Client side: frontend.handler.assets/resolve-asset-real-path-url and <make-asset-url now resolve to the daemon's asset URL (with the server token) instead of an unreachable file:// path when a web/mobile client has a server configured (frontend.config/get-server-base-url) -- covers viewing existing assets. frontend.fs/write-asset-file! gets the matching upload path via a new frontend.persist-db.remote/upload-asset!, so pasting an asset from a web/mobile client no longer silently lands in frontend.fs.memory-fs and vanishes on reload. Server-side routes and round-trip verified end-to-end against a real daemon (curl) and with new automated tests (byte equality, content-type, etag/304, 404, repo-mismatch, auth-required). The client-side URL resolution changes could not be visually verified -- no browser in this environment -- only reviewed and compiled clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
|||
| cf61c46c35 |
Fase 4: migrate the daemon event channel from SSE to WebSocket with resume-by-revision
Adds a WS upgrade handler alongside the existing SSE stream on /v1/events
(kept alive for instant rollback), authenticated via Sec-WebSocket-Protocol
instead of a query-string token. The daemon now keeps a ~500-entry circular
buffer of broadcast sync-db-changes deltas indexed by renderer revision; a
client reconnecting sends {type: subscribe, graphId, sinceRev} and either
gets a gap-free replay + ack, or a {type: resync} instructing it to discard
all cached renderer state and reload (frontend.db.subs/reset-graph!, wired
through a new :db/resync event). The web/Electron client transport
(frontend.persist-db.remote) now opens the WS channel by default; the SSE
path (connect-events!) is untouched and still fully wired.
Also fixes new/pre-existing test breakage surfaced by switching the default
transport: persist_db_test.cljs's shared Electron-failover fixture now mocks
open-ws-fn (not just open-sse-fn), and a `new` applied to an inline (or ...)
expression in make-server's WebSocketServer construction is replaced with
the same let-bound-symbol pattern deps/db-sync's node/server.cljs already
uses -- inlining it silently dropped `new`, breaking every daemon test.
Confirmed 7 unrelated persist_db_test.cljs failures (electron-ensure-remote-*
IPC bookkeeping) already fail identically on the pre-Fase-4 baseline -- left
untouched, out of scope here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|||
| be7abd52a3 |
Fase 3: make the remote transport config-driven, not Electron-only
Every PersistentDB dispatch site in persist_db.cljs branched on electron-runtime?, so a web or mobile client could never take the remote path -- only Electron's IPC-managed local daemon could. The remote client itself (persist_db/remote.cljs) was already platform-agnostic (plain js/fetch and js/EventSource), so this is a transport-selection fix, not a new architecture. - frontend.persist-db/remote-runtime?: true for Electron OR a non-Electron client with config/get-server-base-url configured. Replaces every electron-runtime? check at the seven PersistentDB call sites (<start-runtime!, <list-db, <unsafe-delete, <close-db, <export-db, <import-db, <open-and-fetch-schema, <new). - Fixes the known <export-db bug: it always went through the local OPFS impl regardless of transport, so exporting a remote graph silently exported nothing. It now follows the same remote-runtime? dispatch as every sibling function. - <resolve-remote-runtime!/<release-remote-runtime-slot! split the Electron-IPC path from a new web path that just reads the locally configured base-url/token -- there's no daemon to spawn from a browser, only one to log into. - config.cljs: get/set-server-base-url!, get/set-server-token!, clear-server-config!, mirroring the existing custom-sync-server-url localStorage pattern. - remote.cljs: login! POSTs to the daemon's /v1/auth/login (added in Fase 1) and returns a bearer token. - A minimal settings panel (components/settings.cljs, non-Electron only): server URL + username + password, calls login!, saves the result, and reloads. Functional but not visually verified -- no browser available in this environment; the request/response wiring is confirmed below against a real daemon, not the rendered UI itself. Verified by running a real db-worker-node daemon and exercising the exact sequence the new settings panel and remote.cljs drive: unauthenticated invoke -> 401, login with correct/wrong credentials -> token/401, healthz unauthenticated with :pid/:root-dir absent, authenticated invoke (list-db) -> 200, per-origin CORS reflection, bad Host -> 403, and SSE both without a token (401) and with one as a query param (200). Plus clojure -M:cljs compile app db-worker-node electron (0 warnings) and the full logseq.db-worker.daemon-test, logseq.cli.server-test, frontend.worker.db-worker-node-test, electron.db-worker-manager-test, frontend.state-test, frontend.persist-db.remote-test suites (134 tests, 541 assertions, 0 failures). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
|||
| 49d9f7da84 |
Fase 1: require session auth on the db-worker-node HTTP/SSE server
Prepares the daemon to be reachable off loopback as part of the server-client migration (see the approved implementation plan). Today it trusts any caller that can reach the port; this makes every route except health checks and login require a bearer session token. - New frontend.worker.db-worker-node-auth: scrypt-hashed username/password credential persisted at <root-dir>/db-worker-auth.json (mode 0600), bootstrapped from LOGSEQ_DAEMON_USERNAME/PASSWORD on first launch since there's no interactive setup yet. In-memory bearer sessions with constant-time comparison, a failed-login lockout, and list/revoke by device. The daemon now refuses to start with no credential configured at all -- not just when binding beyond loopback -- since without one no token could ever be issued. - db_worker_node.cljs: POST /v1/auth/login and GET/DELETE /v1/auth/sessions; every other route requires Authorization: Bearer except /healthz, which stays open (readiness polling happens before any session exists) but now omits :pid/:root-dir from its payload. Adds a Host-header allowlist ahead of auth to guard against DNS rebinding, and configurable --host/--port/--allowed-origin/--allowed-host in place of the hardcoded 127.0.0.1 bind and single CORS origin. - EventSource can't send custom headers, so /v1/events accepts the bearer token as a query param instead; persist_db/remote.cljs's events-url carries it there now (interim until the Fase 4 WebSocket migration). - Electron always sent :auth-token nil (verified by reading db_worker.cljs -- one of the plan's open questions). daemon/spawn-server! now seeds a per-root-dir local device credential (<root-dir>/db-worker-local-credential.json, reused across restarts) via LOGSEQ_DAEMON_USERNAME/PASSWORD, and cli/server.cljs logs in once the spawned or reused daemon is ready, threading the resulting token through ensure-server! to Electron's runtime config. Not done: the OCaml CLI (cli/lib/transport.ml) also calls /v1/invoke directly and will start receiving 401s once this ships -- confirmed with the user to leave that gap for now rather than touch the OCaml toolchain in this pass. Verified with clojure -M:cljs compile app db-worker-node electron (0 warnings) and the full logseq.db-worker.daemon-test, logseq.cli.server-test, frontend.worker.db-worker-node-test, electron.db-worker-manager-test suites (113 tests, 480 assertions, 0 failures) plus frontend.state-test and frontend.persist-db.remote-test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
|||
| 52971f8ec9 |
Add Fase 0 telemetry for the server-client migration baseline
Adds dev-only counters, with no behavior change, to answer the open
questions from the migration plan before touching the transport:
- frontend.state/get-invoke-stats: per-thread-api-method count/total-ms/
max-ms for every <invoke-db-worker call, to census how many invokes a
typical navigation generates and their latency distribution.
- db-worker-node: per-repo in-flight /v1/invoke counter that logs a new
concurrency high-water mark, to check whether concurrent client writes
actually overlap before deciding on a write queue.
- subs-loader/get-stats: dedup-hit vs new-entry counts for load!, and
batch-size distribution per slot kind at flush, to check whether
in-flight snapshot requests are already deduplicated and whether the
{:children 25 :resources 25} limits are worth raising for network RTT.
Verified with clojure -M:cljs compile app and compile db-worker-node
(0 warnings on both).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|||
| 40631362b3 |
Initial import of LogDarioSeq baseline
Establishes version control for the personal Logseq DB fork before starting the server-client architecture migration and domain features described in docs/agent-guide and the approved implementation plan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
