Closes the gap the plan flagged as blocking the sync deletion: a web/mobile
client had no way to read or write asset bytes at all (only db-sync's now
end-of-life handlers touched platform/asset-*). Adds GET/PUT
/v1/assets/:repo/:name on db_worker_node.cljs, backed by three new
thread-api ops (asset-read-bytes/write-bytes/stat) that just call the
existing Node platform.cljs asset-*-bytes! functions -- reusing the same
JSON+transit /v1/invoke path already proven for export-db-binary, so no new
binary RPC mechanism was needed.
GET responses carry a real Content-Type (guessed from extension), a sha1
ETag, and immutable Cache-Control -- assets are content-addressed by
filename, so this is safe to cache forever once fetched. Auth accepts
?token= as a fallback for a direct <img src>/browser navigation, same
rationale as the SSE interim token, but as a genuine fallback alongside the
Authorization header rather than replacing it (an early version of this
made query-token exclusive with header auth, which 401'd every JS-driven
fetch -- caught before committing, real end-to-end curl testing against a
live daemon also caught a 500-instead-of-404 on missing assets and a
never-firing 304 from a hyphenated JS property access that needed `aget`
instead of `.-`).
Client side: frontend.handler.assets/resolve-asset-real-path-url and
<make-asset-url now resolve to the daemon's asset URL (with the server
token) instead of an unreachable file:// path when a web/mobile client has
a server configured (frontend.config/get-server-base-url) -- covers viewing
existing assets. frontend.fs/write-asset-file! gets the matching upload
path via a new frontend.persist-db.remote/upload-asset!, so pasting an
asset from a web/mobile client no longer silently lands in
frontend.fs.memory-fs and vanishes on reload.
Server-side routes and round-trip verified end-to-end against a real
daemon (curl) and with new automated tests (byte equality, content-type,
etag/304, 404, repo-mismatch, auth-required). The client-side URL
resolution changes could not be visually verified -- no browser in this
environment -- only reviewed and compiled clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a WS upgrade handler alongside the existing SSE stream on /v1/events
(kept alive for instant rollback), authenticated via Sec-WebSocket-Protocol
instead of a query-string token. The daemon now keeps a ~500-entry circular
buffer of broadcast sync-db-changes deltas indexed by renderer revision; a
client reconnecting sends {type: subscribe, graphId, sinceRev} and either
gets a gap-free replay + ack, or a {type: resync} instructing it to discard
all cached renderer state and reload (frontend.db.subs/reset-graph!, wired
through a new :db/resync event). The web/Electron client transport
(frontend.persist-db.remote) now opens the WS channel by default; the SSE
path (connect-events!) is untouched and still fully wired.
Also fixes new/pre-existing test breakage surfaced by switching the default
transport: persist_db_test.cljs's shared Electron-failover fixture now mocks
open-ws-fn (not just open-sse-fn), and a `new` applied to an inline (or ...)
expression in make-server's WebSocketServer construction is replaced with
the same let-bound-symbol pattern deps/db-sync's node/server.cljs already
uses -- inlining it silently dropped `new`, breaking every daemon test.
Confirmed 7 unrelated persist_db_test.cljs failures (electron-ensure-remote-*
IPC bookkeeping) already fail identically on the pre-Fase-4 baseline -- left
untouched, out of scope here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every PersistentDB dispatch site in persist_db.cljs branched on
electron-runtime?, so a web or mobile client could never take the remote
path -- only Electron's IPC-managed local daemon could. The remote client
itself (persist_db/remote.cljs) was already platform-agnostic (plain
js/fetch and js/EventSource), so this is a transport-selection fix, not a
new architecture.
- frontend.persist-db/remote-runtime?: true for Electron OR a non-Electron
client with config/get-server-base-url configured. Replaces every
electron-runtime? check at the seven PersistentDB call sites
(<start-runtime!, <list-db, <unsafe-delete, <close-db, <export-db,
<import-db, <open-and-fetch-schema, <new).
- Fixes the known <export-db bug: it always went through the local OPFS
impl regardless of transport, so exporting a remote graph silently
exported nothing. It now follows the same remote-runtime? dispatch as
every sibling function.
- <resolve-remote-runtime!/<release-remote-runtime-slot! split the
Electron-IPC path from a new web path that just reads the locally
configured base-url/token -- there's no daemon to spawn from a browser,
only one to log into.
- config.cljs: get/set-server-base-url!, get/set-server-token!,
clear-server-config!, mirroring the existing custom-sync-server-url
localStorage pattern.
- remote.cljs: login! POSTs to the daemon's /v1/auth/login (added in Fase
1) and returns a bearer token.
- A minimal settings panel (components/settings.cljs, non-Electron only):
server URL + username + password, calls login!, saves the result, and
reloads. Functional but not visually verified -- no browser available in
this environment; the request/response wiring is confirmed below against
a real daemon, not the rendered UI itself.
Verified by running a real db-worker-node daemon and exercising the exact
sequence the new settings panel and remote.cljs drive: unauthenticated
invoke -> 401, login with correct/wrong credentials -> token/401, healthz
unauthenticated with :pid/:root-dir absent, authenticated invoke (list-db)
-> 200, per-origin CORS reflection, bad Host -> 403, and SSE both without a
token (401) and with one as a query param (200). Plus clojure -M:cljs
compile app db-worker-node electron (0 warnings) and the full
logseq.db-worker.daemon-test, logseq.cli.server-test,
frontend.worker.db-worker-node-test, electron.db-worker-manager-test,
frontend.state-test, frontend.persist-db.remote-test suites (134 tests,
541 assertions, 0 failures).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Prepares the daemon to be reachable off loopback as part of the
server-client migration (see the approved implementation plan). Today it
trusts any caller that can reach the port; this makes every route except
health checks and login require a bearer session token.
- New frontend.worker.db-worker-node-auth: scrypt-hashed username/password
credential persisted at <root-dir>/db-worker-auth.json (mode 0600),
bootstrapped from LOGSEQ_DAEMON_USERNAME/PASSWORD on first launch since
there's no interactive setup yet. In-memory bearer sessions with
constant-time comparison, a failed-login lockout, and list/revoke by
device. The daemon now refuses to start with no credential configured at
all -- not just when binding beyond loopback -- since without one no
token could ever be issued.
- db_worker_node.cljs: POST /v1/auth/login and GET/DELETE
/v1/auth/sessions; every other route requires Authorization: Bearer
except /healthz, which stays open (readiness polling happens before any
session exists) but now omits :pid/:root-dir from its payload. Adds a
Host-header allowlist ahead of auth to guard against DNS rebinding, and
configurable --host/--port/--allowed-origin/--allowed-host in place of
the hardcoded 127.0.0.1 bind and single CORS origin.
- EventSource can't send custom headers, so /v1/events accepts the bearer
token as a query param instead; persist_db/remote.cljs's events-url
carries it there now (interim until the Fase 4 WebSocket migration).
- Electron always sent :auth-token nil (verified by reading db_worker.cljs
-- one of the plan's open questions). daemon/spawn-server! now seeds a
per-root-dir local device credential
(<root-dir>/db-worker-local-credential.json, reused across restarts) via
LOGSEQ_DAEMON_USERNAME/PASSWORD, and cli/server.cljs logs in once the
spawned or reused daemon is ready, threading the resulting token through
ensure-server! to Electron's runtime config.
Not done: the OCaml CLI (cli/lib/transport.ml) also calls /v1/invoke
directly and will start receiving 401s once this ships -- confirmed with
the user to leave that gap for now rather than touch the OCaml toolchain
in this pass.
Verified with clojure -M:cljs compile app db-worker-node electron (0
warnings) and the full logseq.db-worker.daemon-test,
logseq.cli.server-test, frontend.worker.db-worker-node-test,
electron.db-worker-manager-test suites (113 tests, 480 assertions, 0
failures) plus frontend.state-test and frontend.persist-db.remote-test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds dev-only counters, with no behavior change, to answer the open
questions from the migration plan before touching the transport:
- frontend.state/get-invoke-stats: per-thread-api-method count/total-ms/
max-ms for every <invoke-db-worker call, to census how many invokes a
typical navigation generates and their latency distribution.
- db-worker-node: per-repo in-flight /v1/invoke counter that logs a new
concurrency high-water mark, to check whether concurrent client writes
actually overlap before deciding on a write queue.
- subs-loader/get-stats: dedup-hit vs new-entry counts for load!, and
batch-size distribution per slot kind at flush, to check whether
in-flight snapshot requests are already deduplicated and whether the
{:children 25 :resources 25} limits are worth raising for network RTT.
Verified with clojure -M:cljs compile app and compile db-worker-node
(0 warnings on both).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Establishes version control for the personal Logseq DB fork before
starting the server-client architecture migration and domain features
described in docs/agent-guide and the approved implementation plan.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
