Fase 0's telemetry work had already flagged that subs-loader deduped
load! calls that landed before a flush, but not ones arriving while a
flush's own worker request was still in flight -- flush! cleared *batch
via take-batch! before the request it just sent even resolved, so a
load! for the same slot in that window fired a duplicate request. Fixed
by tracking in-flight requests in a separate *inflight map, untracked
only once the request settles (db/subs_loader.cljs), with a real e2e
test proving the dedup and that a later load! after settlement does
fire a fresh request (db/subs_loader_test.cljs).
Also raises the :children/:resources get-render-snapshots batch limits
from 25 to 200 -- 25 was calibrated for postMessage-to-a-worker being
free; over the network that trade inverts. Two existing subs-test.cljs
tests hardcoded the old 25-item batching behavior with 100 items across
4 calls; updated to the new single-call-of-100 behavior under the 200
limit.
Deliberately did NOT widen db/subs.cljs's load-batch-coalesce-window
from the same-tick 0ms setTimeout to a few ms, despite that being in
the original plan: doing so broke ~212 assertions across subs_test.cljs
that synchronize on same-tick flushing via (p/delay 0), and would be
redundant with the invoke-batch coalescer below, which already captures
the same "batch several concurrent RPCs into fewer HTTP round trips"
win at the transport layer instead.
New POST /v1/invoke-batch on the daemon (db_worker_node.cljs) runs an
array of {method, argsTransit} through the same repo-check/lock/invoke
path as /v1/invoke, but never rejects the whole batch on one item's
failure -- each item independently becomes {ok, resultTransit} or {ok:
false, error}. Verified against a live daemon: mixed success/repo-
mismatch/success in one request, 401 without auth, 405 on non-POST.
On the client, frontend.persist-db.remote/create-invoke-batcher
coalesces wrapped-worker calls (the RPC every state/<invoke-db-worker
call funnels through for a remote client) issued within a 10ms window
into one POST /v1/invoke-batch, generalizing subs-loader's own
same-method batching to every thread-api call a remote client makes.
Two invokes only land in the same batch when neither is awaited before
the other fires, so this can't reorder anything a caller depends on
sequentially.
New db_worker_node_test.cljs coverage: invoke-batch mixing success and
repo-mismatch failure in one request, and invoke-batch auth/method
enforcement.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes the gap the plan flagged as blocking the sync deletion: a web/mobile
client had no way to read or write asset bytes at all (only db-sync's now
end-of-life handlers touched platform/asset-*). Adds GET/PUT
/v1/assets/:repo/:name on db_worker_node.cljs, backed by three new
thread-api ops (asset-read-bytes/write-bytes/stat) that just call the
existing Node platform.cljs asset-*-bytes! functions -- reusing the same
JSON+transit /v1/invoke path already proven for export-db-binary, so no new
binary RPC mechanism was needed.
GET responses carry a real Content-Type (guessed from extension), a sha1
ETag, and immutable Cache-Control -- assets are content-addressed by
filename, so this is safe to cache forever once fetched. Auth accepts
?token= as a fallback for a direct <img src>/browser navigation, same
rationale as the SSE interim token, but as a genuine fallback alongside the
Authorization header rather than replacing it (an early version of this
made query-token exclusive with header auth, which 401'd every JS-driven
fetch -- caught before committing, real end-to-end curl testing against a
live daemon also caught a 500-instead-of-404 on missing assets and a
never-firing 304 from a hyphenated JS property access that needed `aget`
instead of `.-`).
Client side: frontend.handler.assets/resolve-asset-real-path-url and
<make-asset-url now resolve to the daemon's asset URL (with the server
token) instead of an unreachable file:// path when a web/mobile client has
a server configured (frontend.config/get-server-base-url) -- covers viewing
existing assets. frontend.fs/write-asset-file! gets the matching upload
path via a new frontend.persist-db.remote/upload-asset!, so pasting an
asset from a web/mobile client no longer silently lands in
frontend.fs.memory-fs and vanishes on reload.
Server-side routes and round-trip verified end-to-end against a real
daemon (curl) and with new automated tests (byte equality, content-type,
etag/304, 404, repo-mismatch, auth-required). The client-side URL
resolution changes could not be visually verified -- no browser in this
environment -- only reviewed and compiled clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a WS upgrade handler alongside the existing SSE stream on /v1/events
(kept alive for instant rollback), authenticated via Sec-WebSocket-Protocol
instead of a query-string token. The daemon now keeps a ~500-entry circular
buffer of broadcast sync-db-changes deltas indexed by renderer revision; a
client reconnecting sends {type: subscribe, graphId, sinceRev} and either
gets a gap-free replay + ack, or a {type: resync} instructing it to discard
all cached renderer state and reload (frontend.db.subs/reset-graph!, wired
through a new :db/resync event). The web/Electron client transport
(frontend.persist-db.remote) now opens the WS channel by default; the SSE
path (connect-events!) is untouched and still fully wired.
Also fixes new/pre-existing test breakage surfaced by switching the default
transport: persist_db_test.cljs's shared Electron-failover fixture now mocks
open-ws-fn (not just open-sse-fn), and a `new` applied to an inline (or ...)
expression in make-server's WebSocketServer construction is replaced with
the same let-bound-symbol pattern deps/db-sync's node/server.cljs already
uses -- inlining it silently dropped `new`, breaking every daemon test.
Confirmed 7 unrelated persist_db_test.cljs failures (electron-ensure-remote-*
IPC bookkeeping) already fail identically on the pre-Fase-4 baseline -- left
untouched, out of scope here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every PersistentDB dispatch site in persist_db.cljs branched on
electron-runtime?, so a web or mobile client could never take the remote
path -- only Electron's IPC-managed local daemon could. The remote client
itself (persist_db/remote.cljs) was already platform-agnostic (plain
js/fetch and js/EventSource), so this is a transport-selection fix, not a
new architecture.
- frontend.persist-db/remote-runtime?: true for Electron OR a non-Electron
client with config/get-server-base-url configured. Replaces every
electron-runtime? check at the seven PersistentDB call sites
(<start-runtime!, <list-db, <unsafe-delete, <close-db, <export-db,
<import-db, <open-and-fetch-schema, <new).
- Fixes the known <export-db bug: it always went through the local OPFS
impl regardless of transport, so exporting a remote graph silently
exported nothing. It now follows the same remote-runtime? dispatch as
every sibling function.
- <resolve-remote-runtime!/<release-remote-runtime-slot! split the
Electron-IPC path from a new web path that just reads the locally
configured base-url/token -- there's no daemon to spawn from a browser,
only one to log into.
- config.cljs: get/set-server-base-url!, get/set-server-token!,
clear-server-config!, mirroring the existing custom-sync-server-url
localStorage pattern.
- remote.cljs: login! POSTs to the daemon's /v1/auth/login (added in Fase
1) and returns a bearer token.
- A minimal settings panel (components/settings.cljs, non-Electron only):
server URL + username + password, calls login!, saves the result, and
reloads. Functional but not visually verified -- no browser available in
this environment; the request/response wiring is confirmed below against
a real daemon, not the rendered UI itself.
Verified by running a real db-worker-node daemon and exercising the exact
sequence the new settings panel and remote.cljs drive: unauthenticated
invoke -> 401, login with correct/wrong credentials -> token/401, healthz
unauthenticated with :pid/:root-dir absent, authenticated invoke (list-db)
-> 200, per-origin CORS reflection, bad Host -> 403, and SSE both without a
token (401) and with one as a query param (200). Plus clojure -M:cljs
compile app db-worker-node electron (0 warnings) and the full
logseq.db-worker.daemon-test, logseq.cli.server-test,
frontend.worker.db-worker-node-test, electron.db-worker-manager-test,
frontend.state-test, frontend.persist-db.remote-test suites (134 tests,
541 assertions, 0 failures).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Prepares the daemon to be reachable off loopback as part of the
server-client migration (see the approved implementation plan). Today it
trusts any caller that can reach the port; this makes every route except
health checks and login require a bearer session token.
- New frontend.worker.db-worker-node-auth: scrypt-hashed username/password
credential persisted at <root-dir>/db-worker-auth.json (mode 0600),
bootstrapped from LOGSEQ_DAEMON_USERNAME/PASSWORD on first launch since
there's no interactive setup yet. In-memory bearer sessions with
constant-time comparison, a failed-login lockout, and list/revoke by
device. The daemon now refuses to start with no credential configured at
all -- not just when binding beyond loopback -- since without one no
token could ever be issued.
- db_worker_node.cljs: POST /v1/auth/login and GET/DELETE
/v1/auth/sessions; every other route requires Authorization: Bearer
except /healthz, which stays open (readiness polling happens before any
session exists) but now omits :pid/:root-dir from its payload. Adds a
Host-header allowlist ahead of auth to guard against DNS rebinding, and
configurable --host/--port/--allowed-origin/--allowed-host in place of
the hardcoded 127.0.0.1 bind and single CORS origin.
- EventSource can't send custom headers, so /v1/events accepts the bearer
token as a query param instead; persist_db/remote.cljs's events-url
carries it there now (interim until the Fase 4 WebSocket migration).
- Electron always sent :auth-token nil (verified by reading db_worker.cljs
-- one of the plan's open questions). daemon/spawn-server! now seeds a
per-root-dir local device credential
(<root-dir>/db-worker-local-credential.json, reused across restarts) via
LOGSEQ_DAEMON_USERNAME/PASSWORD, and cli/server.cljs logs in once the
spawned or reused daemon is ready, threading the resulting token through
ensure-server! to Electron's runtime config.
Not done: the OCaml CLI (cli/lib/transport.ml) also calls /v1/invoke
directly and will start receiving 401s once this ships -- confirmed with
the user to leave that gap for now rather than touch the OCaml toolchain
in this pass.
Verified with clojure -M:cljs compile app db-worker-node electron (0
warnings) and the full logseq.db-worker.daemon-test,
logseq.cli.server-test, frontend.worker.db-worker-node-test,
electron.db-worker-manager-test suites (113 tests, 480 assertions, 0
failures) plus frontend.state-test and frontend.persist-db.remote-test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Establishes version control for the personal Logseq DB fork before
starting the server-client architecture migration and domain features
described in docs/agent-guide and the approved implementation plan.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
