Initial import of LogDarioSeq baseline

Establishes version control for the personal Logseq DB fork before
starting the server-client architecture migration and domain features
described in docs/agent-guide and the approved implementation plan.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-25 15:32:15 +02:00
commit 40631362b3
2016 changed files with 543898 additions and 0 deletions

38
cli-e2e/AGENTS.md Normal file
View File

@@ -0,0 +1,38 @@
# cli-e2e
Shell-first end-to-end tests for logseq CLI.
## Test cli-e2e itself
- Run internal cli-e2e harness unit tests: `bb unit-test`
## cleanup first
- Execute cleanup: terminate stale db-worker-node processes, terminate stale db-sync listeners on port `18080`, and remove cli-e2e temp roots: `bb cleanup`
- Preview cleanup actions only (no kill/delete): `bb cleanup --dry-run`
## Run non-sync suite
- List declared non-sync case ids: `bb list-cases`
- Run non-sync cases with build preflight unless `--skip-build` is provided: `bb test`
- `bb test --help` for options
- Increase case-level parallelism with `--jobs N` (default: `4`), for example: `bb test --skip-build --jobs 4`
- Parallelism is case-scoped only; each case still runs setup/main/cleanup sequentially in the existing ephemeral shell model
## Run sync suite
- List declared sync case ids: `bb list-sync-cases`
- Run sync cases with build preflight by default: `bb test-sync`
- Add `--skip-build` only when you intentionally want to reuse existing artifacts
- `bb test-sync --help` for options
- Increase case-level parallelism with `--jobs N` (default: `4`), for example: `bb test-sync --jobs 4`
- Parallelism is case-scoped only; each sync case still runs its own setup/main/cleanup sequentially
- The local db-sync server is shared per suite and starts once before cases begin, then stops once after all cases complete
- Each sync case gets its own isolated temp root, data directories, home directory, auth copy, and generated config files
- Configure sync E2EE password: `--e2ee-password <value>` (default: `11111`)
- Run only sync MVP case: `bb test-sync --case sync-upload-download-mvp`
### Sync suite prerequisites
- CLI auth file must exist at `~/logseq/auth.json` (generated by `logseq login`).
- Sync suite starts/stops one shared local db-sync server per suite run.
- Required build artifact for local db-sync server:
- `deps/db-sync/worker/dist/node-adapter.js`
- If artifact is missing, build it before running sync suite:
- `pnpm --dir deps/db-sync build:node-adapter`

1
cli-e2e/README.md Executable file
View File

@@ -0,0 +1 @@
AGENTS.md

47
cli-e2e/bb.edn Normal file
View File

@@ -0,0 +1,47 @@
{:paths ["src" "test"]
:deps {cheshire/cheshire {:mvn/version "5.12.0"}}
:tasks
{:requires ([babashka.cli :as cli]
[logseq.cli.e2e.main :as main]
[logseq.cli.e2e.test-runner :as test-runner])
:init (def cli-opts
(cli/parse-opts
*command-line-args*
{:alias {:i :include
:h :help}
:spec {:jobs {:default 4}}
:coerce {:include []
:help :boolean
:dry-run :boolean
:skip-build :boolean
:verbose :boolean
:timings :boolean
:jobs :long}}))
unit-test
{:doc "Run internal cli-e2e harness unit tests"
:task (test-runner/run! cli-opts)}
build
{:doc "Compile required CLI artifacts and verify expected outputs exist"
:task (main/build! cli-opts)}
list-cases
{:doc "List declared non-sync cli-e2e case ids"
:task (main/list-cases! cli-opts)}
list-sync-cases
{:doc "List declared sync cli-e2e case ids"
:task (main/list-sync-cases! cli-opts)}
test
{:doc "Run non-sync cli-e2e cases with build preflight unless --skip-build is provided"
:task (main/test! cli-opts)}
test-sync
{:doc "Run sync cli-e2e cases with build preflight unless --skip-build is provided; use --jobs for case-level parallelism"
:task (main/test-sync! cli-opts)}
cleanup
{:doc "Terminate cli-e2e db-worker-node processes, terminate stale db-sync listeners, and remove cli-e2e temp roots"
:task (main/cleanup! cli-opts)}}}

View File

@@ -0,0 +1,322 @@
#!/usr/bin/env bash
set -euo pipefail
usage() {
cat <<'USAGE'
Usage: agent_bridge_demo.sh [options]
Creates a fresh Logseq graph, starts `logseq agent bridge`, writes a routable
task after the bridge listener is ready, and verifies that a fake Codex master
session received the task dispatch.
Options:
--cli PATH Path to static/logseq-cli.js. Default: <repo-root>/static/logseq-cli.js
--root-dir DIR Logseq CLI root dir. Default: a new temp dir
--graph NAME Graph name. Default: agent-bridge-demo-<timestamp>
--repo-root DIR Repository root. Default: inferred from this script
--timeout-sec N Wait timeout for bridge events. Default: 45
-h, --help Show this help
USAGE
}
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
repo_root="$(cd "$script_dir/../.." && pwd)"
cli_path=""
root_dir=""
graph="agent-bridge-demo-$(date +%s)"
timeout_sec=45
agent_name="AgentBridgeDemo"
task_title="AgentBridge demo task: mark this block in review"
expected_session="thread-agent-bridge-demo"
bridge_pid=""
graph_created=0
while [[ $# -gt 0 ]]; do
case "$1" in
--cli)
cli_path="$2"
shift 2
;;
--root-dir)
root_dir="$2"
shift 2
;;
--graph)
graph="$2"
shift 2
;;
--repo-root)
repo_root="$2"
shift 2
;;
--timeout-sec)
timeout_sec="$2"
shift 2
;;
-h|--help)
usage
exit 0
;;
*)
echo "Unknown option: $1" >&2
usage >&2
exit 2
;;
esac
done
cli_path="${cli_path:-$repo_root/static/logseq-cli.js}"
root_dir="${root_dir:-$(mktemp -d "${TMPDIR:-/tmp}/logseq-agent-bridge-demo.XXXXXX")}"
config_path="$root_dir/cli.edn"
work_dir="$root_dir/agent-bridge-demo"
fake_bin="$work_dir/fake-bin"
bridge_log="$work_dir/agent-bridge.log"
bridge_err="$work_dir/agent-bridge.err"
codex_log="$work_dir/codex-invocations.jsonl"
cleanup() {
local status=$?
if [[ -n "${bridge_pid:-}" ]] && kill -0 "$bridge_pid" 2>/dev/null; then
kill "$bridge_pid" 2>/dev/null || true
wait "$bridge_pid" 2>/dev/null || true
fi
if [[ "$graph_created" -eq 1 ]]; then
node "$cli_path" --root-dir "$root_dir" --config "$config_path" --output json server stop --graph "$graph" >/dev/null 2>&1 || true
fi
exit "$status"
}
trap cleanup EXIT INT TERM
require_command() {
if ! command -v "$1" >/dev/null 2>&1; then
echo "Required command is missing: $1" >&2
exit 2
fi
}
json_result() {
python3 -c 'import json,sys; print(json.load(sys.stdin)["data"]["result"])'
}
json_result_first() {
python3 -c 'import json,sys; r=json.load(sys.stdin)["data"]["result"]; print(r[0] if isinstance(r, list) else r)'
}
run_cli_json() {
node "$cli_path" --root-dir "$root_dir" --config "$config_path" --output json "$@"
}
wait_for_file_text() {
local path="$1"
local text="$2"
local deadline=$((SECONDS + timeout_sec))
while (( SECONDS < deadline )); do
if [[ -f "$path" ]] && grep -Fq "$text" "$path"; then
return 0
fi
if [[ -n "${bridge_pid:-}" ]] && ! kill -0 "$bridge_pid" 2>/dev/null; then
echo "agent bridge exited before '$text'" >&2
[[ -f "$bridge_log" ]] && cat "$bridge_log" >&2
[[ -f "$bridge_err" ]] && cat "$bridge_err" >&2
exit 1
fi
sleep 0.2
done
echo "Timed out waiting for '$text' in $path" >&2
[[ -f "$bridge_log" ]] && cat "$bridge_log" >&2
[[ -f "$bridge_err" ]] && cat "$bridge_err" >&2
exit 1
}
query_task_status() {
run_cli_json query --graph "$graph" --query "[:find ?status-ident . :where [$1 :logseq.property/status ?status] [?status :db/ident ?status-ident]]" | json_result
}
query_agent_session() {
local payload
payload="$(run_cli_json query --graph "$graph" --query "[:find ?session . :where [$1 :logseq.property.agent/session-id ?session]]")"
python3 - "$root_dir" "$config_path" "$graph" "$cli_path" "$payload" <<'PY'
import json
import subprocess
import sys
payload = json.loads(sys.argv[5])
value = payload.get("data", {}).get("result")
if isinstance(value, int):
root_dir, config_path, graph, cli_path = sys.argv[1:5]
query = f"[:find ?title . :where [{value} :block/title ?title]]"
resolved = subprocess.check_output(
[
"node",
cli_path,
"--root-dir",
root_dir,
"--config",
config_path,
"--output",
"json",
"query",
"--graph",
graph,
"--query",
query,
],
text=True,
)
value = json.loads(resolved)["data"]["result"]
print("" if value is None else value)
PY
}
write_fake_codex() {
mkdir -p "$fake_bin"
cat > "$fake_bin/codex" <<'FAKE_CODEX'
#!/usr/bin/env bash
set -euo pipefail
if [[ "$#" -eq 1 && "$1" == "--version" ]]; then
echo "codex-cli 0.0.0-agent-bridge-demo"
exit 0
fi
if [[ "$#" -ge 6 && "$1" == "--sandbox" && "$2" == "danger-full-access" && "$3" == "exec" && "$4" == "--json" && "$5" == "--skip-git-repo-check" ]]; then
prompt="$6"
python3 - "$CODEX_FAKE_LOG" "$prompt" "$@" <<'PY'
import json
import pathlib
import sys
log_path = pathlib.Path(sys.argv[1])
prompt = sys.argv[2]
args = sys.argv[3:]
log_path.parent.mkdir(parents=True, exist_ok=True)
with log_path.open("a", encoding="utf8") as f:
f.write(json.dumps({"event": "start", "args": args, "prompt": prompt}, ensure_ascii=False) + "\n")
PY
printf '{"type":"thread.started","thread_id":"thread-agent-bridge-demo"}\n'
exit 0
fi
if [[ "$#" -ge 8 && "$1" == "--sandbox" && "$2" == "danger-full-access" && "$3" == "exec" && "$4" == "resume" && "$5" == "--json" && "$6" == "--skip-git-repo-check" ]]; then
session_id="$7"
prompt="$8"
python3 - "$CODEX_FAKE_LOG" "$session_id" "$prompt" "$@" <<'PY'
import json
import pathlib
import sys
log_path = pathlib.Path(sys.argv[1])
session_id = sys.argv[2]
prompt = sys.argv[3]
args = sys.argv[4:]
log_path.parent.mkdir(parents=True, exist_ok=True)
with log_path.open("a", encoding="utf8") as f:
f.write(json.dumps({"event": "resume", "session": session_id, "args": args, "prompt": prompt}, ensure_ascii=False) + "\n")
PY
block_uuid="$(python3 - "$prompt" <<'PY'
import re
import sys
match = re.search(r"^Block UUID:\s*([0-9a-fA-F-]+)\s*$", sys.argv[1], re.MULTILINE)
if not match:
raise SystemExit("Block UUID not found in AgentBridge prompt")
print(match.group(1))
PY
)"
node "$DEMO_CLI" --root-dir "$DEMO_ROOT_DIR" --config "$DEMO_CONFIG" --output json upsert task --graph "$DEMO_GRAPH" --uuid "$block_uuid" --status in-review >/dev/null
node "$DEMO_CLI" --root-dir "$DEMO_ROOT_DIR" --config "$DEMO_CONFIG" --output json upsert block --graph "$DEMO_GRAPH" --uuid "$block_uuid" --update-properties '{"Agent Session ID" "thread-agent-bridge-demo"}' >/dev/null
printf '{"type":"thread.started","thread_id":"%s"}\n' "$session_id"
exit 0
fi
echo "unexpected codex args: $*" >&2
exit 2
FAKE_CODEX
chmod +x "$fake_bin/codex"
}
require_command node
require_command python3
mkdir -p "$work_dir"
printf '{:output-format :json :agent-name "%s"}\n' "$agent_name" > "$config_path"
write_fake_codex
echo "creating graph: $graph"
echo "root dir: $root_dir"
run_cli_json graph create --graph "$graph" >/dev/null
graph_created=1
echo "starting agent bridge"
PATH="$fake_bin:$PATH" \
CODEX_FAKE_LOG="$codex_log" \
DEMO_CLI="$cli_path" \
DEMO_ROOT_DIR="$root_dir" \
DEMO_CONFIG="$config_path" \
DEMO_GRAPH="$graph" \
node "$cli_path" --root-dir "$root_dir" --config "$config_path" --output human agent bridge --graph "$graph" >"$bridge_log" 2>"$bridge_err" &
bridge_pid=$!
wait_for_file_text "$bridge_log" "listening graph changes"
task_id="$(run_cli_json upsert task --graph "$graph" --target-page AgentBridgeDemo --content "$task_title" --status todo | json_result_first)"
run_cli_json upsert block --graph "$graph" --id "$task_id" --update-properties "{\"Assignee\" \"$agent_name\"}" >/dev/null
deadline=$((SECONDS + timeout_sec))
task_status=""
agent_session=""
while (( SECONDS < deadline )); do
task_status="$(query_task_status "$task_id")"
agent_session="$(query_agent_session "$task_id")"
if [[ "$task_status" == "logseq.property/status.in-review" && "$agent_session" == "$expected_session" ]]; then
break
fi
if [[ -n "${bridge_pid:-}" ]] && ! kill -0 "$bridge_pid" 2>/dev/null; then
echo "agent bridge exited before verification completed" >&2
cat "$bridge_log" >&2
cat "$bridge_err" >&2
exit 1
fi
sleep 0.5
done
if [[ "$task_status" != "logseq.property/status.in-review" ]]; then
echo "Expected task status in-review, got: ${task_status:-<empty>}" >&2
cat "$bridge_log" >&2
cat "$bridge_err" >&2
exit 1
fi
if [[ "$agent_session" != "$expected_session" ]]; then
echo "Expected agent-session-id $expected_session, got: ${agent_session:-<empty>}" >&2
cat "$bridge_log" >&2
cat "$bridge_err" >&2
exit 1
fi
python3 - "$codex_log" "$task_title" "$graph" <<'PY'
import json
import pathlib
import sys
log_path = pathlib.Path(sys.argv[1])
task_title = sys.argv[2]
graph = sys.argv[3]
lines = [json.loads(line) for line in log_path.read_text(encoding="utf8").splitlines() if line.strip()]
resume_lines = [line for line in lines if line.get("event") == "resume"]
if len(resume_lines) != 1:
raise SystemExit(f"expected one Codex resume invocation, got {len(resume_lines)}")
prompt = resume_lines[0]["prompt"]
if task_title not in prompt:
raise SystemExit("task title missing from Codex prompt")
if f"Graph: {graph}" not in prompt:
raise SystemExit("graph missing from Codex prompt")
if "Block UUID:" not in prompt:
raise SystemExit("block uuid missing from Codex prompt")
if "Request kind: task" not in prompt:
raise SystemExit("request kind missing from Codex prompt")
PY
echo "task status: in-review"
echo "agent-session-id: $agent_session"
echo "agent bridge demo completed"

View File

@@ -0,0 +1,685 @@
#!/usr/bin/env python3
import argparse
import json
import os
import pathlib
import subprocess
import time
TASK_TITLE = "Test AgentBridge task routing and completion status"
EXPECTED_SESSION = "thread-e2e-agent-bridge"
PARALLEL_TASK_TITLES = [
"测试 agent bridge 并行执行任务 1",
"测试 agent bridge 并行执行任务 2",
]
def write_fake_codex(fake_bin):
fake_codex = fake_bin / "codex"
fake_codex.parent.mkdir(parents=True, exist_ok=True)
fake_codex.write_text(
"""#!/usr/bin/env python3
import json
import os
import pathlib
import re
import sys
import time
args = sys.argv[1:]
if args == ["--version"]:
print("codex-cli 0.0.0-e2e")
sys.exit(0)
if args[:5] == ["--sandbox", "danger-full-access", "exec", "--json", "--skip-git-repo-check"]:
prompt = args[5] if len(args) > 5 else ""
block_uuid_match = re.search(r"^Block UUID: (.+)$", prompt, re.MULTILINE)
block_uuid = block_uuid_match.group(1) if block_uuid_match else None
log_path = pathlib.Path(os.environ["CODEX_FAKE_LOG"])
log_path.parent.mkdir(parents=True, exist_ok=True)
with log_path.open("a", encoding="utf8") as f:
f.write(json.dumps(
{"event": "start", "time": time.time(), "args": args, "prompt": prompt, "block_uuid": block_uuid},
ensure_ascii=False) + "\\n")
delay = float(os.environ.get("CODEX_FAKE_DELAY_SECONDS", "0"))
if delay > 0:
time.sleep(delay)
if os.environ.get("CODEX_FAKE_SESSION_BY_UUID") == "1" and block_uuid:
session_id = "thread-e2e-agent-bridge-" + re.sub(r"[^A-Za-z0-9]", "", block_uuid)[-12:]
else:
session_id = "thread-e2e-agent-bridge"
with log_path.open("a", encoding="utf8") as f:
f.write(json.dumps(
{"event": "session", "time": time.time(), "session": session_id, "block_uuid": block_uuid},
ensure_ascii=False) + "\\n")
print(json.dumps({"type": "thread.started", "thread_id": session_id}), flush=True)
sys.exit(0)
if args[:6] == ["--sandbox", "danger-full-access", "exec", "resume", "--json", "--skip-git-repo-check"]:
session_id = args[6] if len(args) > 6 else ""
prompt = args[7] if len(args) > 7 else ""
comment_uuid_match = re.search(r"^Comment UUID: (.+)$", prompt, re.MULTILINE)
comment_uuid = comment_uuid_match.group(1) if comment_uuid_match else None
block_uuid_match = re.search(r"^Block UUID: (.+)$", prompt, re.MULTILINE)
block_uuid = block_uuid_match.group(1) if block_uuid_match else None
log_path = pathlib.Path(os.environ["CODEX_FAKE_LOG"])
log_path.parent.mkdir(parents=True, exist_ok=True)
with log_path.open("a", encoding="utf8") as f:
f.write(json.dumps(
{"event": "resume", "time": time.time(), "args": args, "prompt": prompt, "comment_uuid": comment_uuid, "block_uuid": block_uuid, "session": session_id},
ensure_ascii=False) + "\\n")
delay = float(os.environ.get("CODEX_FAKE_DELAY_SECONDS", "0"))
if delay > 0:
time.sleep(delay)
with log_path.open("a", encoding="utf8") as f:
f.write(json.dumps(
{"event": "resume-complete", "time": time.time(), "comment_uuid": comment_uuid, "block_uuid": block_uuid, "session": session_id},
ensure_ascii=False) + "\\n")
print(json.dumps({"type": "thread.started", "thread_id": session_id}), flush=True)
sys.exit(0)
print("unexpected codex args: " + repr(args), file=sys.stderr)
sys.exit(2)
""",
encoding="utf8",
)
fake_codex.chmod(0o755)
def run_json(cli, repo_root, root_dir, config, graph, query):
result = subprocess.run(
cli
+ [
"--root-dir",
root_dir,
"--config",
config,
"--output",
"json",
"query",
"--graph",
graph,
"--query",
query,
],
cwd=repo_root,
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
if result.returncode != 0:
return None
return json.loads(result.stdout).get("data", {}).get("result")
def run_cli(cli, repo_root, root_dir, config, graph, extra_args):
result = subprocess.run(
cli
+ [
"--root-dir",
root_dir,
"--config",
config,
"--output",
"json",
]
+ extra_args,
cwd=repo_root,
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
if result.returncode != 0:
raise SystemExit(
"CLI command failed: {}\nstdout:\n{}\nstderr:\n{}".format(
" ".join(extra_args), result.stdout, result.stderr
)
)
return json.loads(result.stdout) if result.stdout.strip() else None
def run_cli_with_env(cli, repo_root, root_dir, config, extra_args, env):
result = subprocess.run(
cli
+ [
"--root-dir",
root_dir,
"--config",
config,
"--output",
"json",
]
+ extra_args,
cwd=repo_root,
env=env,
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
return result
def deref_session_value(cli, repo_root, root_dir, config, graph, value):
if not isinstance(value, int):
return value
return run_json(
cli,
repo_root,
root_dir,
config,
graph,
f"[:find ?title . :where [{value} :block/title ?title]]",
)
def read_text(path):
return path.read_text(encoding="utf8", errors="replace")
def wait_for_log(path, text, process):
deadline = time.time() + 30
while time.time() < deadline:
if path.exists() and text in read_text(path):
return
if process.poll() is not None:
raise SystemExit(
"agent bridge exited before {}\nstdout:\n{}".format(
text, read_text(path)
)
)
time.sleep(0.2)
raise SystemExit("agent bridge did not log {!r}\nstdout:\n{}".format(text, read_text(path)))
def find_task_id(cli, repo_root, root_dir, config, graph, title):
task_id = run_json(
cli,
repo_root,
root_dir,
config,
graph,
'[:find ?e . :where [?e :block/title "{}"]]'.format(title),
)
if task_id is None:
raise SystemExit("task block was not found: {}".format(title))
return task_id
def find_task_uuid(cli, repo_root, root_dir, config, graph, title):
task_uuid = run_json(
cli,
repo_root,
root_dir,
config,
graph,
'[:find ?uuid . :where [?e :block/title "{}"] [?e :block/uuid ?uuid]]'.format(title),
)
if task_uuid is None:
raise SystemExit("task block uuid was not found: {}".format(title))
return task_uuid
def create_task(cli, repo_root, root_dir, config, graph, title):
run_cli(
cli,
repo_root,
root_dir,
config,
graph,
[
"upsert",
"task",
"--graph",
graph,
"--target-page",
"AgentBridgeE2E",
"--content",
title,
"--status",
"todo",
],
)
def assign_task(cli, repo_root, root_dir, config, graph, title=TASK_TITLE):
task_id = find_task_id(cli, repo_root, root_dir, config, graph, title)
run_cli(
cli,
repo_root,
root_dir,
config,
graph,
[
"upsert",
"block",
"--graph",
graph,
"--id",
str(task_id),
"--update-properties",
'{{"Assignee" "{}"}}'.format(os.uname().nodename),
],
)
def read_codex_events(codex_log):
if not codex_log.exists():
return []
return [
json.loads(line)
for line in codex_log.read_text(encoding="utf8").splitlines()
if line.strip()
]
def session_query_for_title(title):
return (
'[:find ?session . :where [?e :block/title "{}"] '
"[?e :logseq.property.agent/session-id ?session]]"
).format(title)
def wait_for_task_sessions(cli, repo_root, root_dir, config, graph, titles, bridge, bridge_log, bridge_err):
deadline = time.time() + 45
sessions = {}
while time.time() < deadline:
sessions = {
title: deref_session_value(
cli,
repo_root,
root_dir,
config,
graph,
run_json(cli, repo_root, root_dir, config, graph, session_query_for_title(title)),
)
for title in titles
}
if all(sessions.values()):
return sessions
if bridge.poll() is not None:
raise SystemExit(
"agent bridge exited early with {}\nstdout:\n{}\nstderr:\n{}".format(
bridge.returncode, read_text(bridge_log), read_text(bridge_err)
)
)
time.sleep(0.5)
raise SystemExit(
"agent-session-id was not written for every task; last sessions={!r}\nstdout:\n{}\nstderr:\n{}".format(
sessions, read_text(bridge_log), read_text(bridge_err)
)
)
def wait_for_codex_event(codex_log, event_name, bridge, bridge_log, bridge_err):
deadline = time.time() + 30
events = []
while time.time() < deadline:
events = read_codex_events(codex_log)
matches = [event for event in events if event.get("event") == event_name]
if matches:
return matches
if bridge.poll() is not None:
raise SystemExit(
"agent bridge exited early with {}\nstdout:\n{}\nstderr:\n{}".format(
bridge.returncode, read_text(bridge_log), read_text(bridge_err)
)
)
time.sleep(0.5)
raise SystemExit("codex event {!r} was not observed; events={!r}".format(event_name, events))
def wait_for_codex_events(codex_log, event_name, count, bridge, bridge_log, bridge_err):
deadline = time.time() + 45
events = []
while time.time() < deadline:
events = read_codex_events(codex_log)
matches = [event for event in events if event.get("event") == event_name]
if len(matches) >= count:
return matches
if bridge.poll() is not None:
raise SystemExit(
"agent bridge exited early with {}\nstdout:\n{}\nstderr:\n{}".format(
bridge.returncode, read_text(bridge_log), read_text(bridge_err)
)
)
time.sleep(0.5)
raise SystemExit("codex event {!r} count {} was not observed; events={!r}".format(event_name, count, events))
def write_comment_blocks_file(path, task_uuid, hostname):
path.write_text(
"""[{{:block/title "Comments"
:block/tags [:logseq.class/Comments]
:logseq.property.comments/blocks [[:block/uuid #uuid "{task_uuid}"]]
:block/children [{{:block/title "[[{hostname}]] please continue from the comment"
:block/tags [:logseq.class/Comment]}}]}}]""".format(
task_uuid=task_uuid,
hostname=hostname,
),
encoding="utf8",
)
def run_comment_mention_check(cli, repo_root, root_dir, config, graph, tmp_dir):
create_task(cli, repo_root, root_dir, config, graph, TASK_TITLE)
fake_bin = tmp_dir / "fake-bin"
bridge_log = tmp_dir / "agent-bridge.log"
bridge_err = tmp_dir / "agent-bridge.err"
codex_log = tmp_dir / "codex-invocations.jsonl"
env = os.environ.copy()
env["PATH"] = str(fake_bin) + os.pathsep + env.get("PATH", "")
env["CODEX_FAKE_LOG"] = str(codex_log)
with bridge_log.open("wb") as out, bridge_err.open("wb") as err:
bridge = subprocess.Popen(
cli
+ [
"--root-dir",
root_dir,
"--config",
config,
"--output",
"human",
"agent",
"bridge",
"--graph",
graph,
],
cwd=repo_root,
env=env,
stdout=out,
stderr=err,
)
try:
wait_for_log(bridge_log, "listening graph changes", bridge)
assign_task(cli, repo_root, root_dir, config, graph)
start_events = wait_for_codex_event(codex_log, "start", bridge, bridge_log, bridge_err)
session_events = wait_for_codex_event(codex_log, "session", bridge, bridge_log, bridge_err)
session = session_events[0].get("session")
resume_events = wait_for_codex_event(codex_log, "resume", bridge, bridge_log, bridge_err)
task_prompt = resume_events[0].get("prompt", "")
assert resume_events[0].get("session") == session, resume_events[0]
assert "Request kind: task" in task_prompt, task_prompt
assert TASK_TITLE in task_prompt, task_prompt
task_id = find_task_id(cli, repo_root, root_dir, config, graph, TASK_TITLE)
task_uuid = find_task_uuid(cli, repo_root, root_dir, config, graph, TASK_TITLE)
blocks_file = tmp_dir / "comment-blocks.edn"
write_comment_blocks_file(blocks_file, task_uuid, os.uname().nodename)
run_cli(
cli,
repo_root,
root_dir,
config,
graph,
[
"upsert",
"block",
"--graph",
graph,
"--target-id",
str(task_id),
"--pos",
"last-child",
"--blocks-file",
str(blocks_file),
],
)
resume_events = wait_for_codex_events(codex_log, "resume", 2, bridge, bridge_log, bridge_err)
comment_event = resume_events[-1]
if comment_event.get("session") != session:
raise SystemExit("comment resumed the wrong master session: {!r}".format(comment_event))
prompt = comment_event.get("prompt", "")
assert "You are handling a Logseq AgentBridge master dispatch request." in prompt, prompt
assert "Request kind: comment" in prompt, prompt
assert "Comment target context:" in prompt, prompt
assert "Requesting comment:" in prompt, prompt
print("agent bridge dispatched comment request to master " + session)
finally:
if bridge.poll() is None:
bridge.terminate()
try:
bridge.wait(timeout=5)
except subprocess.TimeoutExpired:
bridge.kill()
bridge.wait(timeout=5)
def run_parallel_assignment_check(cli, repo_root, root_dir, config, graph, tmp_dir):
for title in PARALLEL_TASK_TITLES:
create_task(cli, repo_root, root_dir, config, graph, title)
fake_bin = tmp_dir / "fake-bin"
bridge_log = tmp_dir / "agent-bridge.log"
bridge_err = tmp_dir / "agent-bridge.err"
codex_log = tmp_dir / "codex-invocations.jsonl"
env = os.environ.copy()
env["PATH"] = str(fake_bin) + os.pathsep + env.get("PATH", "")
env["CODEX_FAKE_LOG"] = str(codex_log)
env["CODEX_FAKE_DELAY_SECONDS"] = "5"
env["CODEX_FAKE_SESSION_BY_UUID"] = "1"
with bridge_log.open("wb") as out, bridge_err.open("wb") as err:
bridge = subprocess.Popen(
cli
+ [
"--root-dir",
root_dir,
"--config",
config,
"--output",
"human",
"agent",
"bridge",
"--graph",
graph,
],
cwd=repo_root,
env=env,
stdout=out,
stderr=err,
)
try:
wait_for_log(bridge_log, "listening graph changes", bridge)
for title in PARALLEL_TASK_TITLES:
assign_task(cli, repo_root, root_dir, config, graph, title)
events = read_codex_events(codex_log)
start_events = [event for event in events if event.get("event") == "start"]
resume_events = wait_for_codex_events(codex_log, "resume", 2, bridge, bridge_log, bridge_err)
complete_events = wait_for_codex_events(codex_log, "resume-complete", 2, bridge, bridge_log, bridge_err)
if len(start_events) != 1 or len(resume_events) != 2:
raise SystemExit("expected one master start and two master resumes, got: {!r}".format(read_codex_events(codex_log)))
first_complete_time = min(event["time"] for event in complete_events)
latest_resume_time = max(event["time"] for event in resume_events)
if latest_resume_time >= first_complete_time:
raise SystemExit(
"master dispatch resumes did not overlap; resumes={!r}, completes={!r}".format(
[event["time"] for event in resume_events],
[event["time"] for event in complete_events],
)
)
prompts = "\n".join(event["prompt"] for event in resume_events)
for title in PARALLEL_TASK_TITLES:
assert title in prompts, prompts
print("agent bridge dispatched tasks concurrently to master")
finally:
if bridge.poll() is None:
bridge.terminate()
try:
bridge.wait(timeout=5)
except subprocess.TimeoutExpired:
bridge.kill()
bridge.wait(timeout=5)
def run_duplicate_bridge_check(cli, repo_root, root_dir, config, graph, tmp_dir):
fake_bin = tmp_dir / "fake-bin"
bridge_log = tmp_dir / "agent-bridge.log"
bridge_err = tmp_dir / "agent-bridge.err"
codex_log = tmp_dir / "codex-invocations.jsonl"
env = os.environ.copy()
env["PATH"] = str(fake_bin) + os.pathsep + env.get("PATH", "")
env["CODEX_FAKE_LOG"] = str(codex_log)
with bridge_log.open("wb") as out, bridge_err.open("wb") as err:
bridge = subprocess.Popen(
cli
+ [
"--root-dir",
root_dir,
"--config",
config,
"--output",
"human",
"agent",
"bridge",
"--graph",
graph,
],
cwd=repo_root,
env=env,
stdout=out,
stderr=err,
)
try:
wait_for_log(bridge_log, "listening graph changes", bridge)
duplicate = subprocess.run(
cli
+ [
"--root-dir",
root_dir,
"--config",
config,
"--output",
"json",
"agent",
"bridge",
"--graph",
graph,
],
cwd=repo_root,
env=env,
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
timeout=15,
)
output = duplicate.stdout + duplicate.stderr
if duplicate.returncode == 0:
raise SystemExit("duplicate bridge unexpectedly succeeded:\n{}".format(output))
if "agent-bridge-already-running" not in output and "already running" not in output:
raise SystemExit("duplicate bridge did not report the expected lock error:\n{}".format(output))
print("agent bridge rejected duplicate process")
finally:
if bridge.poll() is None:
bridge.terminate()
try:
bridge.wait(timeout=5)
except subprocess.TimeoutExpired:
bridge.kill()
bridge.wait(timeout=5)
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--cli", required=True)
parser.add_argument("--root-dir", required=True)
parser.add_argument("--config", required=True)
parser.add_argument("--graph", required=True)
parser.add_argument("--tmp-dir", required=True)
parser.add_argument("--repo-root", required=True)
parser.add_argument("--prepare-fake-codex-only", action="store_true")
parser.add_argument("--assign-after-start", action="store_true")
parser.add_argument("--parallel-assignment-check", action="store_true")
parser.add_argument("--comment-mention-check", action="store_true")
parser.add_argument("--duplicate-bridge-check", action="store_true")
args = parser.parse_args()
repo_root = pathlib.Path(args.repo_root)
tmp_dir = pathlib.Path(args.tmp_dir)
fake_bin = tmp_dir / "fake-bin"
bridge_log = tmp_dir / "agent-bridge.log"
bridge_err = tmp_dir / "agent-bridge.err"
codex_log = tmp_dir / "codex-invocations.jsonl"
cli = ["node", args.cli]
write_fake_codex(fake_bin)
if args.prepare_fake_codex_only:
return
if args.parallel_assignment_check:
run_parallel_assignment_check(cli, repo_root, args.root_dir, args.config, args.graph, tmp_dir)
return
if args.comment_mention_check:
run_comment_mention_check(cli, repo_root, args.root_dir, args.config, args.graph, tmp_dir)
return
if args.duplicate_bridge_check:
run_duplicate_bridge_check(cli, repo_root, args.root_dir, args.config, args.graph, tmp_dir)
return
env = os.environ.copy()
env["PATH"] = str(fake_bin) + os.pathsep + env.get("PATH", "")
env["CODEX_FAKE_LOG"] = str(codex_log)
with bridge_log.open("wb") as out, bridge_err.open("wb") as err:
bridge = subprocess.Popen(
cli
+ [
"--root-dir",
args.root_dir,
"--config",
args.config,
"--output",
"human",
"agent",
"bridge",
"--graph",
args.graph,
],
cwd=repo_root,
env=env,
stdout=out,
stderr=err,
)
try:
if args.assign_after_start:
wait_for_log(bridge_log, "listening graph changes", bridge)
assign_task(cli, repo_root, args.root_dir, args.config, args.graph)
start_events = [event for event in read_codex_events(codex_log) if event.get("event") == "start"]
resume_events = wait_for_codex_event(codex_log, "resume", bridge, bridge_log, bridge_err)
assert len(start_events) == 1, start_events
prompt = resume_events[0]["prompt"]
assert TASK_TITLE in prompt, prompt
assert "Graph: " + args.graph in prompt, prompt
assert "Block UUID:" in prompt, prompt
assert "Request kind: task" in prompt, prompt
print("agent bridge dispatched task to master " + resume_events[0]["session"])
finally:
if bridge.poll() is None:
bridge.terminate()
try:
bridge.wait(timeout=5)
except subprocess.TimeoutExpired:
bridge.kill()
bridge.wait(timeout=5)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,181 @@
#!/usr/bin/env python3
"""Compare normalized query payloads between two cli graph contexts."""
from __future__ import annotations
import argparse
import json
import subprocess
import sys
from pathlib import Path
from typing import Any, Dict
IGNORED_KEYS = {
"db/id",
"db/created-at",
"db/updated-at",
"block/uuid",
"block/updated-at",
"block/created-at",
}
def fail(message: str, **context: object) -> None:
payload = {"status": "error", "message": message}
if context:
payload["context"] = context
print(json.dumps(payload), file=sys.stderr)
raise SystemExit(1)
def normalize(value: Any) -> Any:
if isinstance(value, dict):
normalized = {}
for key in sorted(value.keys(), key=str):
key_str = str(key)
if key_str in IGNORED_KEYS:
continue
normalized[key_str] = normalize(value[key])
return normalized
if isinstance(value, list):
normalized_list = [normalize(item) for item in value]
try:
return sorted(normalized_list, key=lambda item: json.dumps(item, sort_keys=True))
except TypeError:
return normalized_list
return value
def run_query(cli_path: Path, config_path: Path, root_dir: Path, graph: str, query: str) -> Dict[str, Any]:
command = [
"node",
str(cli_path),
"--root-dir",
str(root_dir),
"--config",
str(config_path),
"--output",
"json",
"query",
"--graph",
graph,
"--query",
query,
]
result = subprocess.run(command, capture_output=True, text=True)
if result.returncode != 0:
fail(
"query command failed",
command=command,
exit=result.returncode,
stdout=result.stdout,
stderr=result.stderr,
)
try:
payload = json.loads(result.stdout)
except json.JSONDecodeError as error:
fail(
"query command did not return valid JSON",
command=command,
stdout=result.stdout,
stderr=result.stderr,
detail=str(error),
)
if payload.get("status") != "ok":
fail("query command returned non-ok status", command=command, payload=payload)
data = payload.get("data") or {}
return {
"payload": payload,
"result": data.get("result"),
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Compare normalized query payloads between two cli contexts")
parser.add_argument("--cli", required=True, help="Path to static/logseq-cli.js")
parser.add_argument("--graph", required=True)
parser.add_argument("--query", required=True, action="append")
parser.add_argument("--config-a", required=True)
parser.add_argument("--root-dir-a", required=True)
parser.add_argument("--config-b", required=True)
parser.add_argument("--root-dir-b", required=True)
parser.add_argument("--require-result", action="store_true")
return parser.parse_args()
def main() -> None:
args = parse_args()
cli_path = Path(args.cli).expanduser().resolve()
if not cli_path.exists():
fail("cli entry file does not exist", cli=str(cli_path))
queries = args.query
left_config = Path(args.config_a).expanduser().resolve()
left_root_dir = Path(args.root_dir_a).expanduser().resolve()
right_config = Path(args.config_b).expanduser().resolve()
right_root_dir = Path(args.root_dir_b).expanduser().resolve()
normalized_results = {}
for query in queries:
left = run_query(
cli_path,
left_config,
left_root_dir,
args.graph,
query,
)
right = run_query(
cli_path,
right_config,
right_root_dir,
args.graph,
query,
)
left_result = left["result"]
right_result = right["result"]
if args.require_result and (left_result is None or right_result is None):
fail(
"query result is empty",
query=query,
left_result=left_result,
right_result=right_result,
)
left_normalized = normalize(left_result)
right_normalized = normalize(right_result)
if left_normalized != right_normalized:
fail(
"normalized query results differ",
query=query,
left_result=left_normalized,
right_result=right_normalized,
left_payload=left["payload"],
right_payload=right["payload"],
)
normalized_results[query] = left_normalized
payload_key = "result" if len(normalized_results) == 1 else "results"
payload_value = next(iter(normalized_results.values())) if len(normalized_results) == 1 else normalized_results
print(
json.dumps(
{
"status": "ok",
payload_key: payload_value,
}
)
)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,297 @@
#!/usr/bin/env python3
"""Manage local db-sync server process for cli-e2e sync suite."""
from __future__ import annotations
import argparse
import base64
import ctypes
import json
import os
import signal
import socket
import subprocess
import sys
import time
import urllib.error
import urllib.request
from pathlib import Path
from typing import Any, Dict, Optional
DEFAULT_COGNITO_ISSUER = "https://cognito-idp.us-east-2.amazonaws.com/us-east-2_kAqZcxIeM"
DEFAULT_COGNITO_CLIENT_ID = "1qi1uijg8b6ra70nejvbptis0q"
def fail(message: str, **context: object) -> None:
payload = {"status": "error", "message": message}
if context:
payload["context"] = context
print(json.dumps(payload), file=sys.stderr)
raise SystemExit(1)
def load_pid(pid_file: Path) -> Optional[int]:
if not pid_file.exists():
return None
raw = pid_file.read_text(encoding="utf-8").strip()
if not raw:
return None
try:
return int(raw)
except ValueError:
return None
def process_running(pid: int) -> bool:
if pid <= 0:
return False
if os.name == "nt":
kernel32 = ctypes.windll.kernel32
access = 0x00100000 | 0x1000 # SYNCHRONIZE | PROCESS_QUERY_LIMITED_INFORMATION
handle = kernel32.OpenProcess(access, False, pid)
if handle:
kernel32.CloseHandle(handle)
return True
return ctypes.get_last_error() == 5
try:
os.kill(pid, 0)
except ProcessLookupError:
return False
except PermissionError:
return True
else:
return True
def terminate_process(pid: int, force: bool) -> bool:
if pid <= 0:
return True
if os.name == "nt":
cmd = ["taskkill", "/PID", str(pid), "/T"]
if force:
cmd.append("/F")
result = subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False)
return result.returncode == 0 or not process_running(pid)
try:
os.kill(pid, signal.SIGKILL if force else signal.SIGTERM)
except ProcessLookupError:
return True
except PermissionError:
return False
return True
def parse_json_file(path: Path) -> Dict[str, Any]:
if not path.exists():
fail("sync auth file is missing", auth_path=str(path), hint="Run `logseq login` first.")
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except json.JSONDecodeError as error:
fail("sync auth file is invalid JSON", auth_path=str(path), detail=str(error))
if not isinstance(payload, dict):
fail("sync auth file must be a JSON object", auth_path=str(path))
return payload
def decode_jwt_claims(token: str) -> Optional[Dict[str, Any]]:
if not isinstance(token, str):
return None
parts = token.split(".")
if len(parts) != 3:
return None
payload = parts[1]
padded = payload + "=" * ((4 - (len(payload) % 4)) % 4)
try:
decoded = base64.urlsafe_b64decode(padded.encode("utf-8")).decode("utf-8")
claims = json.loads(decoded)
except (ValueError, UnicodeDecodeError, json.JSONDecodeError):
return None
return claims if isinstance(claims, dict) else None
def auth_cognito_from_auth_file(auth_path: Path) -> Dict[str, str]:
payload = parse_json_file(auth_path)
token = payload.get("id-token") or payload.get("access-token")
claims = decode_jwt_claims(token) if isinstance(token, str) else None
issuer = claims.get("iss") if isinstance(claims, dict) else None
client_id = None
if isinstance(claims, dict):
client_id = claims.get("aud") or claims.get("client_id")
return {
"issuer": issuer if isinstance(issuer, str) and issuer else "",
"client_id": client_id if isinstance(client_id, str) and client_id else "",
}
def wait_health(base_url: str, timeout_s: float, interval_s: float) -> bool:
deadline = time.time() + timeout_s
url = base_url.rstrip("/") + "/health"
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
while time.time() < deadline:
try:
with opener.open(url, timeout=2) as response:
if response.status == 200:
return True
except (urllib.error.URLError, TimeoutError, socket.timeout):
pass
time.sleep(interval_s)
return False
def add_node_path(env: Dict[str, str], path: Path) -> None:
existing = env.get("NODE_PATH")
next_path = str(path)
env["NODE_PATH"] = next_path if not existing else next_path + os.pathsep + existing
def start_server(args: argparse.Namespace) -> None:
repo_root = Path(args.repo_root).expanduser().resolve()
entry = repo_root / "deps" / "db-sync" / "worker" / "dist" / "node-adapter.js"
if not entry.exists():
fail("db-sync node adapter build artifact is missing", entry=str(entry), hint="Run: yarn --cwd deps/db-sync build:node-adapter")
pid_file = Path(args.pid_file).expanduser().resolve()
log_file = Path(args.log_file).expanduser().resolve()
data_dir = Path(args.data_dir).expanduser().resolve()
pid_file.parent.mkdir(parents=True, exist_ok=True)
log_file.parent.mkdir(parents=True, exist_ok=True)
data_dir.mkdir(parents=True, exist_ok=True)
existing_pid = load_pid(pid_file)
if existing_pid and process_running(existing_pid):
fail("db-sync server already running", pid=existing_pid, pid_file=str(pid_file))
auth_path = Path(args.auth_path).expanduser().resolve() if args.auth_path else None
auth_derived = auth_cognito_from_auth_file(auth_path) if auth_path else {"issuer": "", "client_id": ""}
issuer = args.cognito_issuer or auth_derived.get("issuer") or DEFAULT_COGNITO_ISSUER
client_id = args.cognito_client_id or auth_derived.get("client_id") or DEFAULT_COGNITO_CLIENT_ID
jwks_url = args.cognito_jwks_url or f"{issuer}/.well-known/jwks.json"
env = os.environ.copy()
add_node_path(env, repo_root / "deps" / "db-sync" / "node_modules")
env.update(
{
"DB_SYNC_PORT": str(args.port),
"DB_SYNC_DATA_DIR": str(data_dir),
"COGNITO_ISSUER": issuer,
"COGNITO_CLIENT_ID": client_id,
"COGNITO_JWKS_URL": jwks_url,
# CLI e2e sync suite should remain runnable without outbound internet.
"DB_SYNC_ALLOW_UNVERIFIED_JWT_CLAIMS": "true",
}
)
with log_file.open("a", encoding="utf-8") as stream:
stream.write(f"\n=== cli-e2e db-sync start {time.strftime('%Y-%m-%d %H:%M:%S')} ===\n")
stream.flush()
process = subprocess.Popen(
["node", str(entry)],
stdout=stream,
stderr=stream,
cwd=str(repo_root),
env=env,
start_new_session=True,
)
base_url = f"http://{args.host}:{args.port}"
if not wait_health(base_url, args.startup_timeout_s, args.poll_interval_s):
terminate_process(process.pid, force=False)
fail(
"db-sync server failed health check before timeout",
base_url=base_url,
pid=process.pid,
log_file=str(log_file),
)
pid_file.write_text(f"{process.pid}\n", encoding="utf-8")
print(
json.dumps(
{
"status": "ok",
"pid": process.pid,
"pid_file": str(pid_file),
"log_file": str(log_file),
"base_url": base_url,
"data_dir": str(data_dir),
"cognito_issuer": issuer,
"cognito_client_id": client_id,
"auth_path": str(auth_path) if auth_path else None,
}
)
)
def stop_server(args: argparse.Namespace) -> None:
pid_file = Path(args.pid_file).expanduser().resolve()
pid = load_pid(pid_file)
if pid is None:
print(json.dumps({"status": "ok", "stopped": False, "reason": "pid-file-missing-or-empty", "pid_file": str(pid_file)}))
return
if not process_running(pid):
try:
pid_file.unlink(missing_ok=True)
except OSError:
pass
print(json.dumps({"status": "ok", "stopped": False, "reason": "process-not-running", "pid": pid, "pid_file": str(pid_file)}))
return
if not terminate_process(pid, force=False):
fail("db-sync server stop failed", pid=pid, signal="SIGTERM", pid_file=str(pid_file))
deadline = time.time() + args.shutdown_timeout_s
while time.time() < deadline:
if not process_running(pid):
pid_file.unlink(missing_ok=True)
print(json.dumps({"status": "ok", "stopped": True, "signal": "SIGTERM", "pid": pid, "pid_file": str(pid_file)}))
return
time.sleep(args.poll_interval_s)
if not terminate_process(pid, force=True):
fail("db-sync server force stop failed", pid=pid, signal="SIGKILL", pid_file=str(pid_file))
pid_file.unlink(missing_ok=True)
print(json.dumps({"status": "ok", "stopped": True, "signal": "SIGKILL", "pid": pid, "pid_file": str(pid_file)}))
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description="Manage db-sync local server for cli-e2e sync tests")
subparsers = parser.add_subparsers(dest="command", required=True)
start = subparsers.add_parser("start", help="Start server and wait for /health")
start.add_argument("--repo-root", required=True)
start.add_argument("--pid-file", required=True)
start.add_argument("--log-file", required=True)
start.add_argument("--data-dir", required=True)
start.add_argument("--host", default="127.0.0.1")
start.add_argument("--port", type=int, default=8080)
start.add_argument("--startup-timeout-s", type=float, default=25.0)
start.add_argument("--poll-interval-s", type=float, default=0.5)
start.add_argument("--auth-path", default="~/logseq/auth.json")
start.add_argument("--cognito-issuer")
start.add_argument("--cognito-client-id")
start.add_argument("--cognito-jwks-url")
stop = subparsers.add_parser("stop", help="Stop server if running")
stop.add_argument("--pid-file", required=True)
stop.add_argument("--shutdown-timeout-s", type=float, default=10.0)
stop.add_argument("--poll-interval-s", type=float, default=0.25)
return parser
def main() -> None:
args = build_parser().parse_args()
if args.command == "start":
start_server(args)
elif args.command == "stop":
stop_server(args)
else:
fail("unknown command", command=args.command)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,86 @@
#!/usr/bin/env python3
"""Prepare per-case CLI config for sync e2e tests."""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
DEFAULT_OAUTH_TOKEN_ENDPOINT = "https://logseq-prod.auth.us-east-1.amazoncognito.com/oauth2/token"
def fail(message: str, **context: object) -> None:
payload = {"status": "error", "message": message}
if context:
payload["context"] = context
print(json.dumps(payload), file=sys.stderr)
raise SystemExit(1)
def read_auth(auth_path: Path) -> dict:
if not auth_path.exists():
fail("sync auth file is missing", auth_path=str(auth_path), hint="Run `logseq login` first.")
try:
payload = json.loads(auth_path.read_text(encoding="utf-8"))
except json.JSONDecodeError as error:
fail("sync auth file is invalid JSON", auth_path=str(auth_path), detail=str(error))
has_token = any(payload.get(key) for key in ("refresh-token", "id-token", "access-token"))
if not has_token:
fail(
"sync auth file does not contain usable tokens",
auth_path=str(auth_path),
required_any_of=["refresh-token", "id-token", "access-token"],
)
return payload
def write_config(output_path: Path, http_base: str, ws_url: str, oauth_token_endpoint: str) -> None:
output_path.parent.mkdir(parents=True, exist_ok=True)
payload = "\n".join(
[
"{",
" :output-format :json",
f' :http-base "{http_base}"',
f' :ws-url "{ws_url}"',
f' :oauth-token-endpoint "{oauth_token_endpoint}"',
"}",
"",
]
)
output_path.write_text(payload, encoding="utf-8")
def main() -> None:
parser = argparse.ArgumentParser(description="Prepare cli.edn for sync e2e")
parser.add_argument("--output", required=True)
parser.add_argument("--auth-path", default="~/logseq/auth.json")
parser.add_argument("--http-base", required=True)
parser.add_argument("--ws-url", required=True)
parser.add_argument("--oauth-token-endpoint", default=DEFAULT_OAUTH_TOKEN_ENDPOINT)
args = parser.parse_args()
auth_path = Path(args.auth_path).expanduser().resolve()
_auth = read_auth(auth_path)
output_path = Path(args.output).expanduser().resolve()
write_config(output_path, args.http_base, args.ws_url, args.oauth_token_endpoint)
print(
json.dumps(
{
"status": "ok",
"auth_path": str(auth_path),
"config_path": str(output_path),
"http_base": args.http_base,
"ws_url": args.ws_url,
"oauth_token_endpoint": args.oauth_token_endpoint,
}
)
)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,480 @@
#!/usr/bin/env python3
"""Run randomized bidirectional block operations on two synced graph peers."""
from __future__ import annotations
import argparse
import json
import random
import subprocess
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Dict, List
class CliCommandError(RuntimeError):
"""Raised when a CLI command does not complete successfully."""
def __init__(self, message: str, *, context: Dict[str, Any]) -> None:
super().__init__(message)
self.context = context
@dataclass(frozen=True)
class ClientContext:
name: str
config: Path
root_dir: Path
def fail(message: str, **context: object) -> None:
payload = {"status": "error", "message": message}
if context:
payload["context"] = context
print(json.dumps(payload), file=sys.stderr)
raise SystemExit(1)
def run_cli_json(
*,
cli_path: Path,
graph: str,
client: ClientContext,
args: List[str],
) -> Dict[str, Any]:
command = [
"node",
str(cli_path),
"--root-dir",
str(client.root_dir),
"--config",
str(client.config),
"--output",
"json",
*args,
"--graph",
graph,
]
result = subprocess.run(command, capture_output=True, text=True)
if result.returncode != 0:
raise CliCommandError(
"cli command exited with non-zero status",
context={
"client": client.name,
"command": command,
"exit": result.returncode,
"stdout": result.stdout,
"stderr": result.stderr,
},
)
try:
payload = json.loads(result.stdout)
except json.JSONDecodeError as error:
raise CliCommandError(
"cli command did not return valid json",
context={
"client": client.name,
"command": command,
"stdout": result.stdout,
"stderr": result.stderr,
"detail": str(error),
},
) from error
if payload.get("status") != "ok":
raise CliCommandError(
"cli command returned non-ok status",
context={
"client": client.name,
"command": command,
"payload": payload,
},
)
return payload
def page_block_ids(
*,
cli_path: Path,
graph: str,
client: ClientContext,
page_title: str,
) -> List[int]:
query = (
"[:find [?e ...] "
":where "
f"[?p :block/title {json.dumps(page_title)}] "
"[?e :block/page ?p] "
"[?e :block/uuid]]"
)
payload = run_cli_json(
cli_path=cli_path,
graph=graph,
client=client,
args=["query", "--query", query],
)
result = (payload.get("data") or {}).get("result")
if not isinstance(result, list):
return []
output: List[int] = []
for item in result:
try:
output.append(int(item))
except (TypeError, ValueError):
continue
return output
def upsert_page(
*,
cli_path: Path,
graph: str,
client: ClientContext,
page_title: str,
) -> None:
run_cli_json(
cli_path=cli_path,
graph=graph,
client=client,
args=["upsert", "page", "--page", page_title],
)
def create_block(
*,
cli_path: Path,
graph: str,
client: ClientContext,
page_title: str,
content: str,
ids: List[int],
rng: random.Random,
) -> None:
if ids and rng.random() < 0.6:
target_id = str(rng.choice(ids))
run_cli_json(
cli_path=cli_path,
graph=graph,
client=client,
args=[
"upsert",
"block",
"--target-id",
target_id,
"--pos",
"first-child",
"--content",
content,
],
)
return
run_cli_json(
cli_path=cli_path,
graph=graph,
client=client,
args=[
"upsert",
"block",
"--target-page",
page_title,
"--content",
content,
],
)
def move_block(
*,
cli_path: Path,
graph: str,
client: ClientContext,
page_title: str,
ids: List[int],
rng: random.Random,
) -> bool:
if not ids:
return False
source_id = str(rng.choice(ids))
run_cli_json(
cli_path=cli_path,
graph=graph,
client=client,
args=[
"upsert",
"block",
"--id",
source_id,
"--target-page",
page_title,
"--pos",
"last-child",
],
)
return True
def delete_block(
*,
cli_path: Path,
graph: str,
client: ClientContext,
ids: List[int],
rng: random.Random,
) -> bool:
if len(ids) <= 2:
return False
source_id = str(rng.choice(ids))
run_cli_json(
cli_path=cli_path,
graph=graph,
client=client,
args=["remove", "block", "--id", source_id],
)
return True
def feasible(operation: str, ids: List[int]) -> bool:
if operation == "create":
return True
if operation == "move":
return len(ids) >= 1
if operation == "delete":
return len(ids) > 2
return False
def choose_operation(op_counts: Dict[str, int], ids: List[int], rng: random.Random) -> str:
for required in ("create", "move", "delete"):
if op_counts.get(required, 0) == 0 and feasible(required, ids):
return required
candidates = ["create", "create", "move"]
if feasible("delete", ids):
candidates.append("delete")
if feasible("move", ids):
candidates.append("move")
return rng.choice(candidates)
def apply_operation(
*,
cli_path: Path,
graph: str,
client: ClientContext,
page_title: str,
op_counts: Dict[str, int],
round_index: int,
rng: random.Random,
) -> None:
last_error: Dict[str, Any] | None = None
for attempt in range(1, 7):
ids = page_block_ids(
cli_path=cli_path,
graph=graph,
client=client,
page_title=page_title,
)
operation = choose_operation(op_counts, ids, rng)
content = f"{client.name}-rnd-{round_index:03d}-{rng.randint(100000, 999999)}"
try:
if operation == "create":
create_block(
cli_path=cli_path,
graph=graph,
client=client,
page_title=page_title,
content=content,
ids=ids,
rng=rng,
)
op_counts["create"] = op_counts.get("create", 0) + 1
return
if operation == "move":
moved = move_block(
cli_path=cli_path,
graph=graph,
client=client,
page_title=page_title,
ids=ids,
rng=rng,
)
if moved:
op_counts["move"] = op_counts.get("move", 0) + 1
return
continue
if operation == "delete":
deleted = delete_block(
cli_path=cli_path,
graph=graph,
client=client,
ids=ids,
rng=rng,
)
if deleted:
op_counts["delete"] = op_counts.get("delete", 0) + 1
return
continue
except CliCommandError as error:
last_error = {
"attempt": attempt,
"operation": operation,
"context": error.context,
}
continue
fail(
"failed to apply random operation after retries",
client=client.name,
round_index=round_index,
last_error=last_error,
)
def ensure_non_empty_page(
*,
cli_path: Path,
graph: str,
client: ClientContext,
page_title: str,
rng: random.Random,
) -> None:
ids = page_block_ids(
cli_path=cli_path,
graph=graph,
client=client,
page_title=page_title,
)
if ids:
return
content = f"{client.name}-reseed-{rng.randint(100000, 999999)}"
create_block(
cli_path=cli_path,
graph=graph,
client=client,
page_title=page_title,
content=content,
ids=[],
rng=rng,
)
PROFILE_DEFAULT_ROUNDS = {
"default": 40,
"high-stress": 100,
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Run randomized bidirectional block operations on two synced graph clients"
)
parser.add_argument("--cli", required=True, help="Path to static/logseq-cli.js")
parser.add_argument("--graph", required=True)
parser.add_argument("--config-a", required=True)
parser.add_argument("--root-dir-a", required=True)
parser.add_argument("--config-b", required=True)
parser.add_argument("--root-dir-b", required=True)
parser.add_argument("--page", required=True)
parser.add_argument(
"--profile",
choices=sorted(PROFILE_DEFAULT_ROUNDS.keys()),
default="default",
help="Execution profile controlling default stress level",
)
parser.add_argument("--rounds-per-client", type=int, default=None)
parser.add_argument("--seed", type=int, default=424242)
args = parser.parse_args()
if args.rounds_per_client is None:
args.rounds_per_client = PROFILE_DEFAULT_ROUNDS[args.profile]
return args
def main() -> None:
args = parse_args()
cli_path = Path(args.cli).expanduser().resolve()
if not cli_path.exists():
fail("cli path does not exist", cli=str(cli_path))
rng = random.Random(args.seed)
client_a = ClientContext(
name="a",
config=Path(args.config_a).expanduser().resolve(),
root_dir=Path(args.root_dir_a).expanduser().resolve(),
)
client_b = ClientContext(
name="b",
config=Path(args.config_b).expanduser().resolve(),
root_dir=Path(args.root_dir_b).expanduser().resolve(),
)
clients = [client_a, client_b]
for client in clients:
upsert_page(
cli_path=cli_path,
graph=args.graph,
client=client,
page_title=args.page,
)
# Seed both peers so move/delete have available targets from the beginning.
for seed_round in range(3):
for client in clients:
create_block(
cli_path=cli_path,
graph=args.graph,
client=client,
page_title=args.page,
content=f"{client.name}-seed-{seed_round}-{rng.randint(100000, 999999)}",
ids=[],
rng=rng,
)
op_stats: Dict[str, Dict[str, int]] = {
client.name: {"create": 0, "move": 0, "delete": 0} for client in clients
}
for round_index in range(args.rounds_per_client):
for client in clients:
apply_operation(
cli_path=cli_path,
graph=args.graph,
client=client,
page_title=args.page,
op_counts=op_stats[client.name],
round_index=round_index,
rng=rng,
)
for client in clients:
ensure_non_empty_page(
cli_path=cli_path,
graph=args.graph,
client=client,
page_title=args.page,
rng=rng,
)
print(
json.dumps(
{
"status": "ok",
"graph": args.graph,
"page": args.page,
"rounds_per_client": args.rounds_per_client,
"seed": args.seed,
"stats": op_stats,
"total_operations": args.rounds_per_client * len(clients),
}
)
)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,249 @@
#!/usr/bin/env python3
"""Exercise `logseq sync asset download` in an isolated sync e2e graph."""
from __future__ import annotations
import argparse
import hashlib
import json
import subprocess
import sys
import time
from pathlib import Path
from typing import Any, Dict, Iterable, List
def fail(message: str, **context: object) -> None:
payload = {"status": "error", "message": message}
if context:
payload["context"] = context
print(json.dumps(payload), file=sys.stderr)
raise SystemExit(1)
def cli_base(cli: Path, root_dir: Path, config: Path) -> List[str]:
return [
"node",
str(cli),
"--root-dir",
str(root_dir),
"--config",
str(config),
"--output",
"json",
]
def run_cli_json(command: List[str], *, allow_error: bool = False) -> Dict[str, Any]:
result = subprocess.run(command, capture_output=True, text=True)
if result.returncode != 0 and not allow_error:
fail(
"cli command failed",
command=command,
exit=result.returncode,
stdout=result.stdout,
stderr=result.stderr,
)
try:
payload = json.loads(result.stdout)
except json.JSONDecodeError as error:
fail(
"cli command did not return valid JSON",
command=command,
exit=result.returncode,
stdout=result.stdout,
stderr=result.stderr,
detail=str(error),
)
if payload.get("status") != "ok" and not allow_error:
fail("cli command returned non-ok status", command=command, payload=payload)
return payload
def contains_key(value: Any, key: str) -> bool:
if isinstance(value, dict):
return key in value or any(contains_key(child, key) for child in value.values())
if isinstance(value, list):
return any(contains_key(child, key) for child in value)
return False
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(8192), b""):
digest.update(chunk)
return digest.hexdigest()
def wait_for_checksum(path: Path, checksum: str, timeout_s: float = 60.0) -> None:
deadline = time.time() + timeout_s
last_state = "missing"
while time.time() < deadline:
if path.exists():
current = sha256_file(path)
if current == checksum:
return
last_state = f"checksum:{current}"
time.sleep(1.0)
fail("asset file did not reach expected checksum", path=str(path), expected=checksum, last_state=last_state)
def get_asset(cli: Path, root_dir: Path, config: Path, graph: str, title: str) -> Dict[str, Any]:
query = (
'[:find (pull ?e [:db/id :block/uuid '
':logseq.property.asset/type :logseq.property.asset/checksum '
':logseq.property.asset/remote-metadata]) . '
f':where [?e :block/title "{title}"]]'
)
payload = run_cli_json(
cli_base(cli, root_dir, config)
+ ["query", "--graph", graph, "--query", query]
)
asset = (payload.get("data") or {}).get("result")
if not isinstance(asset, dict):
fail("asset query did not return an entity", payload=payload)
required = [
"db/id",
"block/uuid",
"logseq.property.asset/type",
"logseq.property.asset/checksum",
"logseq.property.asset/remote-metadata",
]
missing = [key for key in required if not asset.get(key)]
if missing:
fail("asset entity is missing required fields", asset=asset, missing=missing)
return asset
def sync_asset_download_by_id(cli: Path, root_dir: Path, config: Path, graph: str, asset_id: Any) -> Dict[str, Any]:
return run_cli_json(
cli_base(cli, root_dir, config)
+ ["sync", "asset", "download", "--graph", graph, "--id", str(asset_id)]
)
def sync_asset_download_by_uuid(cli: Path, root_dir: Path, config: Path, graph: str, asset_uuid: str) -> Dict[str, Any]:
return run_cli_json(
cli_base(cli, root_dir, config)
+ ["sync", "asset", "download", "--graph", graph, "--uuid", asset_uuid]
)
def assert_data(payload: Dict[str, Any], expected: Dict[str, Any]) -> None:
data = payload.get("data") or {}
for key, value in expected.items():
if data.get(key) != value:
fail("unexpected sync asset download data", expected=expected, actual=data, payload=payload)
if contains_key(payload, "local-path"):
fail("sync asset download output leaked local-path", payload=payload)
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Run sync asset download e2e assertions")
parser.add_argument("--cli", required=True)
parser.add_argument("--graph", required=True)
parser.add_argument("--asset-title", required=True)
parser.add_argument("--config-b", required=True)
parser.add_argument("--root-dir-b", required=True)
return parser.parse_args()
def main() -> None:
args = parse_args()
cli = Path(args.cli).expanduser().resolve()
root_b = Path(args.root_dir_b).expanduser().resolve()
config_b = Path(args.config_b).expanduser().resolve()
asset = get_asset(cli, root_b, config_b, args.graph, args.asset_title)
asset_id = asset["db/id"]
asset_uuid = asset["block/uuid"]
asset_type = asset["logseq.property.asset/type"]
checksum = asset["logseq.property.asset/checksum"]
asset_path = root_b / "graphs" / args.graph / "assets" / f"{asset_uuid}.{asset_type}"
if asset_path.exists():
asset_path.unlink()
requested = sync_asset_download_by_uuid(cli, root_b, config_b, args.graph, asset_uuid)
assert_data(
requested,
{
"asset-id": asset_id,
"asset-uuid": asset_uuid,
"asset-type": asset_type,
"download-requested?": True,
"checksum-status": "missing",
},
)
wait_for_checksum(asset_path, checksum)
skipped = sync_asset_download_by_id(cli, root_b, config_b, args.graph, asset_id)
assert_data(
skipped,
{
"asset-id": asset_id,
"asset-uuid": asset_uuid,
"asset-type": asset_type,
"download-requested?": False,
"checksum-status": "match",
"skipped-reason": "already-downloaded",
},
)
asset_path.write_text("corrupted local asset", encoding="utf-8")
mismatched = sync_asset_download_by_id(cli, root_b, config_b, args.graph, asset_id)
assert_data(
mismatched,
{
"asset-id": asset_id,
"asset-uuid": asset_uuid,
"asset-type": asset_type,
"download-requested?": True,
"checksum-status": "mismatch",
},
)
hint = (mismatched.get("data") or {}).get("hint") or ""
if "checksum" not in hint:
fail("checksum mismatch hint is missing", payload=mismatched)
wait_for_checksum(asset_path, checksum)
run_cli_json(cli_base(cli, root_b, config_b) + ["sync", "stop", "--graph", args.graph])
inactive = run_cli_json(
cli_base(cli, root_b, config_b)
+ ["sync", "asset", "download", "--graph", args.graph, "--id", str(asset_id)],
allow_error=True,
)
if inactive.get("status") != "error":
fail("inactive sync command unexpectedly succeeded", payload=inactive)
error = inactive.get("error") or {}
if error.get("code") != "sync-not-started":
fail("inactive sync command returned wrong error", payload=inactive)
if "sync start" not in (error.get("hint") or ""):
fail("inactive sync command did not include start hint", payload=inactive)
print(
json.dumps(
{
"status": "ok",
"data": {
"asset-id": asset_id,
"asset-uuid": asset_uuid,
"asset-type": asset_type,
"download-requested?": True,
"checksum-status": "mismatch",
"pending-local": 0,
"pending-asset": 0,
"pending-server": 0,
"last-error": None,
},
}
)
)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,302 @@
#!/usr/bin/env python3
"""Poll `logseq sync status` until queues settle and tx converges."""
from __future__ import annotations
import argparse
import json
import subprocess
import sys
import time
from pathlib import Path
from typing import Any, Dict
def fail(message: str, **context: object) -> None:
payload = {"status": "error", "message": message}
if context:
payload["context"] = context
print(json.dumps(payload), file=sys.stderr)
raise SystemExit(1)
def parse_int(value: Any) -> int:
if value is None:
return 0
if isinstance(value, bool):
return int(value)
if isinstance(value, (int, float)):
return int(value)
try:
return int(str(value))
except (TypeError, ValueError):
return 0
def status_command(args: argparse.Namespace) -> list[str]:
existing = getattr(args, "status_command", None)
if existing is not None:
return existing
command = [
"node",
str(Path(args.cli).expanduser().resolve()),
"--root-dir",
str(Path(args.root_dir).expanduser().resolve()),
"--config",
str(Path(args.config).expanduser().resolve()),
"--output",
"json",
"sync",
"status",
"--graph",
args.graph,
]
args.status_command = command
return command
def run_status(args: argparse.Namespace) -> Dict[str, Any]:
command = status_command(args)
result = subprocess.run(command, capture_output=True, text=True)
if result.returncode != 0:
fail(
"sync status command failed",
command=command,
exit=result.returncode,
stdout=result.stdout,
stderr=result.stderr,
)
try:
payload = json.loads(result.stdout)
except json.JSONDecodeError as error:
fail(
"sync status command did not return valid JSON",
command=command,
stdout=result.stdout,
stderr=result.stderr,
detail=str(error),
)
if payload.get("status") != "ok":
fail("sync status returned non-ok status", payload=payload)
return payload
def pending_counts(status_payload: Dict[str, Any]) -> Dict[str, int]:
data = status_payload.get("data") or {}
return {
"pending-local": parse_int(data.get("pending-local")),
"pending-asset": parse_int(data.get("pending-asset")),
"pending-server": parse_int(data.get("pending-server")),
}
def all_settled(counts: Dict[str, int]) -> bool:
required_keys = ("pending-local", "pending-asset", "pending-server")
return all(counts.get(key, 0) == 0 for key in required_keys)
def parse_required_int(value: Any) -> int | None:
if value is None:
return None
if isinstance(value, str) and value.strip() == "":
return None
if isinstance(value, bool):
return int(value)
if isinstance(value, (int, float)):
return int(value)
try:
return int(str(value))
except (TypeError, ValueError):
return None
def tx_sync_status(status_payload: Dict[str, Any]) -> Dict[str, Any]:
data = status_payload.get("data") or {}
local_tx = parse_required_int(data.get("local-tx"))
remote_tx = parse_required_int(data.get("remote-tx"))
synced = (
local_tx is not None
and remote_tx is not None
and local_tx == remote_tx
)
return {
"local-tx": local_tx,
"remote-tx": remote_tx,
"synced": synced,
}
def tx_deltas(
tx_status: Dict[str, Any],
baseline_tx: Dict[str, Any],
) -> Dict[str, int | None]:
local_tx = tx_status.get("local-tx")
remote_tx = tx_status.get("remote-tx")
baseline_local_tx = baseline_tx.get("local-tx")
baseline_remote_tx = baseline_tx.get("remote-tx")
local_tx_delta = (
None
if local_tx is None or baseline_local_tx is None
else local_tx - baseline_local_tx
)
remote_tx_delta = (
None
if remote_tx is None or baseline_remote_tx is None
else remote_tx - baseline_remote_tx
)
return {
"local-tx-delta": local_tx_delta,
"remote-tx-delta": remote_tx_delta,
}
def min_tx_delta_reached(
tx_status: Dict[str, Any],
baseline_tx: Dict[str, Any],
min_tx_delta: int,
) -> bool:
if min_tx_delta <= 0:
return True
deltas = tx_deltas(tx_status, baseline_tx)
local_tx_delta = deltas.get("local-tx-delta")
remote_tx_delta = deltas.get("remote-tx-delta")
if local_tx_delta is None or remote_tx_delta is None:
return False
return local_tx_delta >= min_tx_delta and remote_tx_delta >= min_tx_delta
def main() -> None:
parser = argparse.ArgumentParser(
description="Wait for sync status to settle"
)
parser.add_argument(
"--cli",
required=True,
help="Path to static/logseq-cli.js",
)
parser.add_argument("--root-dir", required=True)
parser.add_argument("--config", required=True)
parser.add_argument("--graph", required=True)
parser.add_argument("--timeout-s", type=float, default=120.0)
parser.add_argument("--interval-s", type=float, default=1.0)
parser.add_argument(
"--min-tx-delta",
type=int,
default=0,
help=(
"Require synced tx delta to be >= this value: "
"(new-tx - old-tx) >= min-tx-delta "
"(default: 0)"
),
)
parser.add_argument(
"--baseline-tx",
type=int,
default=None,
help=(
"Optional explicit baseline tx used as old-tx for both local "
"and remote; if omitted, first observed tx values "
"are used"
),
)
args = parser.parse_args()
if args.min_tx_delta < 0:
invalid_min_tx_delta = args.min_tx_delta
fail(
"min-tx-delta must be non-negative",
min_tx_delta=invalid_min_tx_delta,
)
started = time.time()
deadline = started + args.timeout_s
last_payload: Dict[str, Any] | None = None
baseline_tx: Dict[str, Any] | None = (
{
"local-tx": args.baseline_tx,
"remote-tx": args.baseline_tx,
}
if args.baseline_tx is not None
else None
)
while time.time() < deadline:
payload = run_status(args)
last_payload = payload
data = payload.get("data") or {}
last_error = data.get("last-error")
if last_error is not None:
fail("sync status reports last-error", payload=payload)
counts = pending_counts(payload)
tx_status = tx_sync_status(payload)
if baseline_tx is None:
baseline_tx = {
"local-tx": tx_status.get("local-tx"),
"remote-tx": tx_status.get("remote-tx"),
}
deltas = tx_deltas(tx_status, baseline_tx)
if (
all_settled(counts)
and tx_status["synced"]
and min_tx_delta_reached(
tx_status,
baseline_tx,
args.min_tx_delta,
)
):
print(
json.dumps(
{
"status": "ok",
"elapsed_s": round(time.time() - started, 3),
"counts": counts,
"tx": {
"local-tx": tx_status["local-tx"],
"remote-tx": tx_status["remote-tx"],
},
"tx-delta": deltas,
"baseline-tx": baseline_tx,
"min_tx_delta": args.min_tx_delta,
"payload": payload,
}
)
)
return
time.sleep(max(args.interval_s, 0.0))
last_tx_status = tx_sync_status(last_payload or {})
last_baseline_tx = baseline_tx or {"local-tx": None, "remote-tx": None}
fail(
(
"sync status polling timed out before queues settled, tx synced, "
"and min tx delta reached"
),
timeout_s=args.timeout_s,
min_tx_delta=args.min_tx_delta,
baseline_tx=last_baseline_tx,
last_payload=last_payload,
last_counts=pending_counts(last_payload or {}),
last_tx=last_tx_status,
last_tx_delta=tx_deltas(
last_tx_status,
last_baseline_tx,
),
)
if __name__ == "__main__":
main()

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,174 @@
{:excluded-command-prefixes ["sync" "login" "logout"]
:scopes
{:global
{:options ["--help"
"--version"
"--config"
"--graph"
"--root-dir"
"--output"
"--verbose"]}
:graph
{:commands ["graph list"
"graph create"
"graph switch"
"graph remove"
"graph validate"
"graph info"
"graph export"
"graph import"
"graph backup list"
"graph backup create"
"graph backup restore"
"graph backup remove"]
:options ["--fix"
"--type"
"--file"
"--edn-options"
"--pretty-print"
"--input"
"--name"
"--src"
"--dst"]}
:list
{:commands ["list page"
"list tag"
"list property"
"list task"
"list node"
"list asset"]
:options ["--expand"
"--fields"
"--limit"
"--offset"
"--sort"
"--order"
"--status"
"--priority"
"--content"
"--tags"
"--properties"
"--journal-only"
"--with-properties"
"--with-extends"
"--with-classes"
"--with-type"]}
:upsert
{:commands ["upsert block"
"upsert page"
"upsert task"
"upsert asset"
"upsert tag"
"upsert property"]
:options ["--id"
"--uuid"
"--target-id"
"--target-uuid"
"--target-page"
"--pos"
"--content"
"--path"
"--blocks-file"
"--status"
"--priority"
"--scheduled"
"--deadline"
"--no-status"
"--no-priority"
"--no-scheduled"
"--no-deadline"
"--update-tags"
"--update-properties"
"--remove-tags"
"--remove-properties"
"--add-properties"
"--remove-properties"
"--page"
"--name"
"--type"
"--cardinality"
"--hide"
"--public"]}
:remove
{:commands ["remove block"
"remove page"
"remove tag"
"remove property"]
:options ["--id"
"--uuid"
"--name"]}
:query
{:commands ["query"
"query list"]
:options ["--query"
"--name"
"--inputs"]}
:search
{:commands ["search block"
"search page"
"search property"
"search tag"]
:options ["--content"]}
:show
{:commands ["show"]
:options ["--id"
"--uuid"
"--page"
"--page-hierarchy"
"--linked-references"
"--level"
"stdin:--id"]}
:debug
{:commands ["debug pull"]
:options ["--id"
"--uuid"
"--ident"]}
:example
{:commands ["example list"
"example list page"
"example upsert"
"example upsert page"
"example remove"
"example remove page"
"example query"
"example query list"
"example search"
"example search block"
"example show"]
:options []}
:server
{:commands ["server list"
"server cleanup"
"server start"
"server stop"
"server restart"]
:options ["--graph"]}
:doctor
{:commands ["doctor"]
:options ["--dev-script"]}
:agent
{:commands ["agent bridge"]
:options []}
:completion
{:commands ["completion"]
:options ["bash"
"zsh"
"--shell"]}
:skill
{:commands ["skill show"
"skill install"]
:options ["--global"]}}}

229
cli-e2e/spec/sync_cases.edn Normal file
View File

@@ -0,0 +1,229 @@
{:templates
{:sync/base
{:cleanup
["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{graph-arg}}"
"{{cli}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json server stop --graph {{graph-arg}}"],
:tags [:sync],
:vars
{:sync-port "18080",
:sync-http-base "http://127.0.0.1:18080",
:sync-ws-url "ws://127.0.0.1:18080/sync/%s",
:home-dir "{{tmp-dir}}/home",
:auth-path "{{tmp-dir}}/home/logseq/auth.json",
:cli-home "HOME='{{tmp-dir}}/home' {{cli}}"},
:setup
["mkdir -p '{{tmp-dir}}/graphs-b'"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync ensure-keys --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"]},
:sync/create-enable-base
{:cleanup
["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{graph-arg}}"
"{{cli}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json server stop --graph {{graph-arg}}"],
:tags [:sync],
:vars
{:sync-port "18080",
:sync-http-base "http://127.0.0.1:18080",
:sync-ws-url "ws://127.0.0.1:18080/sync/%s",
:home-dir "{{tmp-dir}}/home",
:auth-path "{{tmp-dir}}/home/logseq/auth.json",
:cli-home "HOME='{{tmp-dir}}/home' {{cli}}"},
:setup
["mkdir -p '{{tmp-dir}}/graphs-b'"]},
:sync/common
{:extends :sync/base,
:cmds
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync upload --graph {{graph-arg}}"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync start --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{root-dir}}' --config '{{config-path}}' --graph '{{graph}}' --timeout-s 120 --interval-s 1"
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync download --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync start --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 120 --interval-s 1"],
:expect
{:exit 0,
:stdout-json-paths
{[:status] "ok",
[:data :pending-local] 0,
[:data :pending-asset] 0,
[:data :pending-server] 0,
[:data :last-error] nil}},
:covers
{:commands ["sync upload" "sync download" "sync status"],
:options {:global ["--config" "--graph" "--root-dir" "--output"]}}}}
:cases
[{:tags [:happy-path :bootstrap :create-enable-sync :a-to-b],
:extends :sync/create-enable-base,
:setup
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} --enable-sync --e2ee-password {{e2ee-password-arg}}"
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{root-dir}}' --config '{{config-path}}' --graph '{{graph}}' --timeout-s 120 --interval-s 1"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncCreateEnableHome >/dev/null"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncCreateEnableHome --content '{{marker-content}}' >/dev/null"
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{root-dir}}' --config '{{config-path}}' --graph '{{graph}}' --timeout-s 120 --interval-s 1"
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync download --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync start --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 120 --interval-s 1"],
:cmds
["python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{marker-content}}\")] ]' --require-result"
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync status --graph {{graph-arg}}"],
:expect
{:exit 0,
:stdout-json-paths
{[:status] "ok",
[:data :pending-local] 0,
[:data :pending-asset] 0,
[:data :pending-server] 0,
[:data :last-error] nil}},
:covers
{:commands ["graph create" "sync download" "sync status"],
:options {:global ["--config" "--graph" "--root-dir" "--output"],
:graph-create ["--enable-sync" "--e2ee-password"]}},
:id "sync-graph-create-enable-sync-a-to-b",
:graph "sync-e2e-create-enable-sync-a-to-b",
:vars {:marker-content "sync-create-enable-sync-a-to-b-marker"}}
{:tags [:happy-path :bootstrap :a-to-b],
:extends :sync/common,
:setup
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncBootstrapHome >/dev/null"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncBootstrapHome --content '{{marker-content}}' >/dev/null"],
:cmds
["python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{marker-content}}\")] ]' --require-result"
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync status --graph {{graph-arg}}"],
:id "sync-bootstrap-upload-download-a-to-b",
:graph "sync-e2e-bootstrap-a-to-b",
:vars {:marker-content "sync-happy-bootstrap-a-to-b-marker"}}
{:tags [:happy-path :incremental :a-to-b],
:extends :sync/common,
:setup
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncIncrementalHome >/dev/null"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncIncrementalHome --content '{{seed-marker}}' >/dev/null"],
:cmds
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncIncrementalHome --content '{{incremental-marker}}' >/dev/null"
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 120 --interval-s 1"
"python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{incremental-marker}}\")] ]' --require-result"
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync status --graph {{graph-arg}}"],
:id "sync-incremental-update-a-to-b",
:graph "sync-e2e-incremental-a-to-b",
:vars
{:seed-marker "sync-happy-incremental-seed-marker",
:incremental-marker "sync-happy-incremental-update-marker"}}
{:tags [:happy-path :bidirectional :roundtrip],
:extends :sync/common,
:setup
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncRoundtripAHome >/dev/null"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncRoundtripAHome --content '{{marker-a}}' >/dev/null"],
:cmds
["{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json upsert page --graph {{graph-arg}} --page SyncRoundtripBHome >/dev/null"
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json upsert block --graph {{graph-arg}} --target-page SyncRoundtripBHome --content '{{marker-b}}' >/dev/null"
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{root-dir}}' --config '{{config-path}}' --graph '{{graph}}' --timeout-s 120 --interval-s 1"
"python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{marker-b}}\")] ]' --require-result"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync status --graph {{graph-arg}}"],
:id "sync-bidirectional-roundtrip",
:graph "sync-e2e-bidirectional-roundtrip",
:vars
{:marker-a "sync-happy-roundtrip-a-seed-marker",
:marker-b "sync-happy-roundtrip-b-origin-marker"}}
{:tags [:happy-path :multi-batch :a-to-b],
:extends :sync/common,
:setup
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncMultiBatchHome >/dev/null"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncMultiBatchHome --content '{{seed-marker}}' >/dev/null"],
:cmds
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncMultiBatchOne >/dev/null"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncMultiBatchOne --content '{{batch-marker-1}}' >/dev/null"
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 30 --interval-s 1"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncMultiBatchTwo >/dev/null"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncMultiBatchTwo --content '{{batch-marker-2}}' >/dev/null"
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 30 --interval-s 1"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncMultiBatchThree >/dev/null"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncMultiBatchThree --content '{{batch-marker-3}}' >/dev/null"
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 30 --interval-s 1"
"python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{batch-marker-1}}\")] ]' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{batch-marker-2}}\")] ]' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{batch-marker-3}}\")] ]' --require-result"
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync status --graph {{graph-arg}}"],
:id "sync-multi-batch-operations",
:graph "sync-e2e-multi-batch-operations",
:vars
{:seed-marker "sync-happy-multi-batch-seed-marker",
:batch-marker-3 "sync-happy-multi-batch-marker-3",
:batch-marker-1 "sync-happy-multi-batch-marker-1",
:batch-marker-2 "sync-happy-multi-batch-marker-2"}}
{:tags [:negative :upload :duplicate],
:extends :sync/base,
:setup
["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{graph-arg}}"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync upload --graph {{graph-arg}} >/dev/null"],
:cmds
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync upload --graph {{graph-arg}}"],
:expect
{:exit 1,
:stdout-json-paths
{[:status] "error",
[:error :code] "graph-already-exists",
[:error :data :graph-name] "{{graph}}"},
:stdout-contains ["delete it before uploading again"]},
:id "sync-upload-rejects-duplicate-remote-graph",
:graph "sync-e2e-upload-rejects-duplicate-remote-graph"}
{:tags [:happy-path :steady-state :status],
:extends :sync/common,
:setup
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncSteadyStateHome >/dev/null"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncSteadyStateHome --content '{{marker-content}}' >/dev/null"],
:cmds
["python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{marker-content}}\")] ]' --require-result"
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 30 --interval-s 1"
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync status --graph {{graph-arg}}"],
:id "sync-status-steady-state",
:graph "sync-e2e-status-steady-state",
:vars {:marker-content "sync-happy-steady-state-marker"}}
{:tags [:stress :bidirectional :random :block-ops],
:extends :sync/common,
:setup
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page '{{random-page}}' >/dev/null"],
:cmds
["python3 '{{repo-root}}/cli-e2e/scripts/random_bidirectional_block_ops.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --page '{{random-page}}' --profile default --rounds-per-client {{rounds-per-client}} --seed {{random-seed}}"
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{root-dir}}' --config '{{config-path}}' --graph '{{graph}}' --timeout-s 180 --interval-s 1 --min-tx-delta 1 --baseline-tx 0"
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 180 --interval-s 1 --min-tx-delta 1 --baseline-tx 0"
"python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find (pull ?b [:block/uuid :block/title :block/order {:block/parent [:block/uuid]}]) :where [?p :block/title \"{{random-page}}\"] [?b :block/page ?p] [?b :block/uuid]]' --require-result"
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync status --graph {{graph-arg}}"],
:id "sync-random-bidirectional-block-ops",
:graph "sync-e2e-random-bidirectional-block-ops",
:vars
{:random-seed "424242",
:random-page "SyncRandomOpsHome",
:rounds-per-client "40"}}
{:tags [:happy-path :asset-download :a-to-b],
:extends :sync/common,
:setup
["printf '{{asset-payload}}' > '{{tmp-dir}}/sync-asset-download.txt'"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert asset --graph {{graph-arg}} --path '{{tmp-dir}}/sync-asset-download.txt' --content '{{asset-title}}' --target-page Home >/dev/null"],
:cmds
["HOME='{{tmp-dir}}/home' python3 '{{repo-root}}/cli-e2e/scripts/sync_asset_download.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --asset-title '{{asset-title}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b'"],
:id "sync-asset-download-a-to-b",
:graph "sync-e2e-asset-download-a-to-b",
:vars
{:asset-title "Sync Asset Download Asset"
:asset-payload "sync-asset-download-payload"},
:covers
{:commands ["upsert asset" "query" "sync asset download"],
:options {:sync ["--id" "--uuid"]}}}
{:tags [:stress :offline :bidirectional :random :block-ops],
:extends :sync/common,
:setup
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page '{{random-page}}' >/dev/null"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page '{{random-page}}' --content '{{seed-marker}}' >/dev/null"],
:cmds
["python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{seed-marker}}\")] ]' --require-result"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync stop --graph {{graph-arg}}"
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync stop --graph {{graph-arg}}"
"python3 '{{repo-root}}/cli-e2e/scripts/random_bidirectional_block_ops.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --page '{{random-page}}' --profile high-stress --rounds-per-client {{rounds-per-client}} --seed {{random-seed}}"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync start --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync start --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{root-dir}}' --config '{{config-path}}' --graph '{{graph}}' --timeout-s 240 --interval-s 1 --min-tx-delta 1 --baseline-tx 0"
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 240 --interval-s 1 --min-tx-delta 1 --baseline-tx 0"
"python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find (pull ?b [:block/uuid :block/title :block/order {:block/parent [:block/uuid]}]) :where [?p :block/title \"{{random-page}}\"] [?b :block/page ?p] [?b :block/uuid]]' --require-result"
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync status --graph {{graph-arg}}"],
:id "sync-offline-random-bidirectional-block-ops",
:graph "sync-e2e-offline-random-bidirectional-block-ops",
:vars
{:seed-marker "sync-offline-random-seed-marker",
:random-seed "989898",
:random-page "SyncOfflineRandomOpsHome",
:rounds-per-client "40"}}]}

View File

@@ -0,0 +1,14 @@
{:excluded-command-prefixes ["login" "logout"]
:scopes
{:global
{:options ["--config"
"--graph"
"--root-dir"
"--output"]}
:sync
{:commands ["sync upload"
"sync download"
"sync asset download"
"sync status"]
:options []}}}

View File

@@ -0,0 +1,184 @@
(ns logseq.cli.e2e.cleanup
(:require [babashka.fs :as fs]
[clojure.java.shell :as java-shell]
[clojure.string :as string]))
(def ^:private cli-e2e-temp-prefix "logseq-cli-e2e-")
(def ^:private db-sync-default-port 18080)
(defn- parse-ps-line
[line]
(when-let [[_ pid-str command] (re-matches #"\s*(\d+)\s+(.*)" (or line ""))]
{:pid (Long/parseLong pid-str)
:command command}))
(defn- cli-e2e-db-worker-command?
[command]
(and (re-find #"db-worker-node(?:\.js)?\b" command)
(re-find #"logseq-cli-e2e-" command)))
(defn list-cli-e2e-db-worker-pids
([]
(list-cli-e2e-db-worker-pids {}))
([{:keys [shell-fn]
:or {shell-fn java-shell/sh}}]
(let [{:keys [exit out err]} (shell-fn "ps" "-ax" "-o" "pid=" "-o" "command=")]
(when-not (zero? exit)
(throw (ex-info "Unable to scan processes"
{:exit exit
:err err})))
(->> (string/split-lines (or out ""))
(keep parse-ps-line)
(filter (fn [{:keys [command]}]
(cli-e2e-db-worker-command? command)))
(mapv :pid)))))
(defn- pid-alive?
[pid]
(zero? (:exit (java-shell/sh "kill" "-0" (str pid)))))
(defn- kill-pid!
[pid]
(try
(if-not (pid-alive? pid)
:not-found
(do
(java-shell/sh "kill" "-TERM" (str pid))
(Thread/sleep 100)
(if-not (pid-alive? pid)
:killed
(do
(java-shell/sh "kill" "-KILL" (str pid))
(Thread/sleep 50)
(if (pid-alive? pid)
:failed
:killed)))))
(catch Exception _
:failed)))
(defn cleanup-db-worker-processes!
([]
(cleanup-db-worker-processes! {}))
([{:keys [dry-run list-pids-fn kill-pid-fn]}]
(let [list-pids-fn (or list-pids-fn list-cli-e2e-db-worker-pids)
kill-pid-fn (or kill-pid-fn kill-pid!)
found-pids (vec (list-pids-fn))]
(if dry-run
{:dry-run? true
:found-pids found-pids
:would-kill-pids found-pids
:killed-pids []
:failed-pids []}
(let [{:keys [killed-pids failed-pids]}
(reduce (fn [acc pid]
(if (= :failed (kill-pid-fn pid))
(update acc :failed-pids conj pid)
(update acc :killed-pids conj pid)))
{:killed-pids []
:failed-pids []}
found-pids)]
{:dry-run? false
:found-pids found-pids
:would-kill-pids []
:killed-pids killed-pids
:failed-pids failed-pids})))))
(defn- parse-long-safe
[value]
(try
(Long/parseLong value)
(catch Exception _
nil)))
(defn list-cli-e2e-db-sync-port-pids
([]
(list-cli-e2e-db-sync-port-pids {}))
([{:keys [shell-fn port]
:or {shell-fn java-shell/sh
port db-sync-default-port}}]
(let [{:keys [exit out err]} (shell-fn "lsof" "-nP" (str "-iTCP:" port) "-sTCP:LISTEN")
out-lines (->> (string/split-lines (or out ""))
(map string/trim)
(remove string/blank?)
vec)]
(when (and (not (zero? exit))
(or (not (string/blank? err))
(seq out-lines)))
(throw (ex-info "Unable to scan db-sync server port listeners"
{:exit exit
:err err
:port port})))
(->> out-lines
(filter #(re-find (re-pattern (str ":" port "\\b")) %))
(keep (fn [line]
(some-> (string/split line #"\s+")
(nth 1 nil)
parse-long-safe)))
distinct
vec))))
(defn cleanup-db-sync-port-processes!
([]
(cleanup-db-sync-port-processes! {}))
([{:keys [dry-run list-pids-fn kill-pid-fn]
:as _opts}]
(let [list-pids-fn (or list-pids-fn list-cli-e2e-db-sync-port-pids)
kill-pid-fn (or kill-pid-fn kill-pid!)
found-pids (vec (list-pids-fn))]
(if dry-run
{:dry-run? true
:found-pids found-pids
:would-kill-pids found-pids
:killed-pids []
:failed-pids []}
(let [{:keys [killed-pids failed-pids]}
(reduce (fn [acc pid]
(if (= :failed (kill-pid-fn pid))
(update acc :failed-pids conj pid)
(update acc :killed-pids conj pid)))
{:killed-pids []
:failed-pids []}
found-pids)]
{:dry-run? false
:found-pids found-pids
:would-kill-pids []
:killed-pids killed-pids
:failed-pids failed-pids})))))
(defn- list-cli-e2e-temp-roots
[tmp-root]
(->> (fs/list-dir tmp-root)
(filter fs/directory?)
(filter #(string/starts-with? (fs/file-name %) cli-e2e-temp-prefix))
(mapv str)))
(defn cleanup-temp-roots!
([]
(cleanup-temp-roots! {}))
([{:keys [dry-run tmp-root list-dirs-fn delete-dir-fn]
:or {tmp-root (System/getProperty "java.io.tmpdir")
delete-dir-fn fs/delete-tree}}]
(let [list-dirs-fn (or list-dirs-fn
#(list-cli-e2e-temp-roots tmp-root))
found-dirs (vec (list-dirs-fn))]
(if dry-run
{:dry-run? true
:found-dirs found-dirs
:would-remove-dirs found-dirs
:removed-dirs []
:failed-dirs []}
(let [{:keys [removed-dirs failed-dirs]}
(reduce (fn [acc dir]
(try
(delete-dir-fn dir)
(update acc :removed-dirs conj dir)
(catch Exception _
(update acc :failed-dirs conj dir))))
{:removed-dirs []
:failed-dirs []}
found-dirs)]
{:dry-run? false
:found-dirs found-dirs
:would-remove-dirs []
:removed-dirs removed-dirs
:failed-dirs failed-dirs})))))

View File

@@ -0,0 +1,120 @@
(ns logseq.cli.e2e.coverage
(:require [clojure.set :as set]
[clojure.string :as string]))
(defn- normalize-commands
[commands]
(->> commands
(map str)
(remove string/blank?)
distinct
sort
vec))
(defn- normalize-options
[options]
(->> options
(map str)
(remove string/blank?)
distinct
sort
vec))
(defn- excluded-command?
[excluded-prefixes command]
(let [head (first (string/split (str command) #" "))]
(contains? (set excluded-prefixes) head)))
(defn validate-inventory!
[{:keys [excluded-command-prefixes scopes] :as inventory}]
(let [invalid-commands (->> scopes
vals
(mapcat :commands)
(filter #(excluded-command? excluded-command-prefixes %))
normalize-commands)
invalid-scopes (->> scopes
(keep (fn [[scope {:keys [commands]}]]
(when (some #(excluded-command? excluded-command-prefixes %) commands)
scope)))
sort
vec)]
(when (seq invalid-commands)
(throw (ex-info "Excluded commands cannot appear in cli-e2e inventory"
{:invalid-commands invalid-commands
:invalid-scopes invalid-scopes
:inventory inventory})))
inventory))
(defn validate-cases!
[{:keys [excluded-command-prefixes] :as inventory} cases]
(let [invalid-cases (->> cases
(filter (fn [{:keys [covers]}]
(some #(excluded-command? excluded-command-prefixes %)
(get covers :commands []))))
(mapv :id))]
(when (seq invalid-cases)
(throw (ex-info "Excluded commands cannot be covered by cli-e2e cases"
{:invalid-case-ids invalid-cases
:inventory inventory})))
cases))
(defn command->scope
[{:keys [scopes]}]
(into {}
(mapcat (fn [[scope {:keys [commands]}]]
(map (fn [command]
[command scope])
commands)))
scopes))
(defn required-commands
[{:keys [scopes]}]
(->> scopes
vals
(mapcat :commands)
normalize-commands))
(defn covered-commands
[cases]
(->> cases
(mapcat #(get-in % [:covers :commands]))
normalize-commands))
(defn required-options-by-scope
[{:keys [scopes]}]
(into {}
(map (fn [[scope {:keys [options]}]]
[scope (normalize-options options)]))
scopes))
(defn covered-options-by-scope
[cases]
(reduce (fn [acc {:keys [covers]}]
(reduce-kv (fn [acc' scope options]
(update acc' scope (fnil into #{}) options))
acc
(get covers :options {})))
{}
cases))
(defn coverage-report
[inventory cases]
(validate-inventory! inventory)
(validate-cases! inventory cases)
(let [required-commands* (set (required-commands inventory))
covered-commands* (set (covered-commands cases))
covered-options* (covered-options-by-scope cases)
required-options* (required-options-by-scope inventory)]
{:missing-commands (->> (set/difference required-commands* covered-commands*)
normalize-commands)
:missing-options (into {}
(map (fn [[scope options]]
[scope (->> (set/difference (set options)
(get covered-options* scope #{}))
normalize-options)]))
required-options*)}))
(defn complete?
[{:keys [missing-commands missing-options]}]
(and (empty? missing-commands)
(every? empty? (vals missing-options))))

View File

@@ -0,0 +1,468 @@
(ns logseq.cli.e2e.main
(:require [clojure.set :as set]
[logseq.cli.e2e.cleanup :as cleanup]
[logseq.cli.e2e.coverage :as coverage]
[logseq.cli.e2e.manifests :as manifests]
[logseq.cli.e2e.preflight :as preflight]
[logseq.cli.e2e.report :as report]
[logseq.cli.e2e.runner :as runner]
[logseq.cli.e2e.shell :as shell]
[logseq.cli.e2e.sync-fixture :as sync-fixture])
(:import (java.util.concurrent Executors LinkedBlockingQueue TimeUnit)))
(defn select-cases
[cases {:keys [case include]}]
(let [cases (remove :skip cases)]
(cond
case
(filterv #(= case (:id %)) cases)
(seq include)
(let [include-tags (set (map keyword include))]
(filterv (fn [{:keys [tags]}]
(not-empty (set/intersection include-tags (set tags))))
cases))
:else
(vec cases))))
(def default-suite :non-sync)
(def default-jobs 1)
(def default-cli-jobs 4)
(defn- suite-from-opts
[opts]
(or (:suite opts) default-suite))
(defn- elapsed-ms
[started-at]
(long (/ (- (System/nanoTime) started-at) 1000000)))
(defn- format-duration
[started-at]
(format "%.2fs" (/ (double (- (System/nanoTime) started-at)) 1000000000.0)))
(defn- positive-jobs
[jobs]
(let [jobs (or jobs default-jobs)]
(when-not (and (integer? jobs) (pos? jobs))
(throw (ex-info "--jobs must be a positive integer"
{:jobs jobs})))
jobs))
(defn run-selected-cases!
[selected-cases run-case run-command {:keys [on-case-start on-case-success on-case-failure detailed-log? timings? jobs]}]
(let [total (count selected-cases)
_ (positive-jobs jobs)]
(reduce (fn [acc [idx case]]
(let [index (inc idx)
started-at (System/nanoTime)]
(when on-case-start
(on-case-start {:index index
:total total
:case case}))
(try
(let [result (run-case case {:run-command run-command
:detailed-log? detailed-log?
:timings? timings?})
payload {:index index
:total total
:case case
:result result
:elapsed-ms (elapsed-ms started-at)}]
(when on-case-success
(on-case-success payload))
(conj acc result))
(catch Exception error
(when on-case-failure
(on-case-failure {:index index
:total total
:case case
:error error
:elapsed-ms (elapsed-ms started-at)}))
(throw error)))))
[]
(map-indexed vector selected-cases))))
(defn run-selected-cases-in-parallel!
[selected-cases run-case run-command {:keys [on-case-start on-case-success on-case-failure detailed-log? timings? jobs]}]
(let [total (count selected-cases)
jobs (positive-jobs jobs)
executor (Executors/newFixedThreadPool jobs)
completions (LinkedBlockingQueue.)]
(try
(doseq [[idx case] (map-indexed vector selected-cases)]
(let [index (inc idx)]
(when on-case-start
(on-case-start {:index index
:total total
:case case}))
(.submit executor
^Runnable
(fn []
(let [started-at (System/nanoTime)]
(.put completions
(try
(let [result (run-case case {:run-command run-command
:detailed-log? detailed-log?
:timings? timings?})]
{:index index
:total total
:case case
:result result
:elapsed-ms (elapsed-ms started-at)})
(catch Exception error
{:index index
:total total
:case case
:error error
:elapsed-ms (elapsed-ms started-at)}))))))))
(loop [remaining total
results []
failure nil]
(if (zero? remaining)
(do
(when failure
(throw failure))
(->> results
(sort-by :index)
(mapv :result)))
(let [payload (.take completions)]
(if-let [error (:error payload)]
(do
(when on-case-failure
(on-case-failure payload))
(recur (dec remaining) results (or failure error)))
(do
(when on-case-success
(on-case-success payload))
(recur (dec remaining) (conj results payload) failure))))))
(finally
(.shutdown executor)
(.awaitTermination executor 1 TimeUnit/MINUTES)))))
(defn run!
[{:keys [inventory cases skip-build run-command jobs]
:as opts}]
(let [run-command (or run-command shell/run!)
run-case (or (:run-case opts) runner/run-case!)
suite (suite-from-opts opts)
sync-suite? (= suite :sync)
jobs (positive-jobs jobs)
targeted-run? (or (:case opts) (seq (:include opts)))
on-preflight-start (:on-preflight-start opts)
on-preflight-complete (:on-preflight-complete opts)
on-cases-ready (:on-cases-ready opts)]
(when on-preflight-start
(on-preflight-start {:skip-build skip-build}))
(let [preflight-result (preflight/run! {:skip-build skip-build
:run-command run-command})
inventory (or inventory (manifests/load-inventory suite))
cases (or cases (manifests/load-cases suite))]
(when on-preflight-complete
(on-preflight-complete preflight-result))
(let [selected-cases (select-cases cases opts)
coverage-result (when-not targeted-run?
(coverage/coverage-report inventory selected-cases))]
(when (and coverage-result
(not (coverage/complete? coverage-result)))
(throw (ex-info "Missing coverage"
{:coverage coverage-result
:message (report/format-missing-coverage coverage-result)})))
(when on-cases-ready
(on-cases-ready {:total (count selected-cases)
:targeted-run? targeted-run?}))
(let [suite-context (when sync-suite?
(sync-fixture/before-suite! {:run-command run-command}))
sync-context (if suite-context
(assoc suite-context :e2ee-password (:e2ee-password opts))
suite-context)
run-case* (if sync-suite?
(fn [case case-opts]
(run-case (sync-fixture/prepare-case case sync-context)
case-opts))
run-case)]
(try
{:status :ok
:cases selected-cases
:coverage coverage-result
:results ((if (> jobs 1)
run-selected-cases-in-parallel!
run-selected-cases!)
selected-cases
run-case*
run-command
(assoc opts :jobs jobs))}
(finally
(when suite-context
(sync-fixture/after-suite! suite-context {:run-command run-command})))))))))
(defn build!
[opts]
(preflight/run! opts))
(defn list-cases!
[opts]
(doseq [{:keys [id]} (manifests/load-cases (suite-from-opts opts))]
(println id)))
(defn list-sync-cases!
[opts]
(list-cases! (assoc opts :suite :sync)))
(defn- print-cleanup-help!
[]
(println "Usage: bb -f cli-e2e/bb.edn cleanup [options]")
(println)
(println "Options:")
(println " -h, --help Show this help and exit")
(println " --dry-run Scan and report only; do not kill/delete")
(println)
(println "Cleanups performed:")
(println " - Terminate cli-e2e db-worker-node processes")
(println " - Terminate db-sync server listeners on port 18080")
(println " - Remove cli-e2e temp roots")
(flush))
(defn cleanup!
[opts]
(if (:help opts)
(do
(print-cleanup-help!)
{:status :help})
(let [dry-run? (boolean (:dry-run opts))
cleanup-opts (cond-> {}
dry-run? (assoc :dry-run true))
processes (cleanup/cleanup-db-worker-processes! cleanup-opts)
db-sync-port-processes (cleanup/cleanup-db-sync-port-processes! cleanup-opts)
temp-roots (cleanup/cleanup-temp-roots! cleanup-opts)]
(println "==> Running cli-e2e cleanup")
(if dry-run?
(do
(println (format "[dry-run] db-worker-node processes: found %d, would kill %d"
(count (:found-pids processes))
(count (:would-kill-pids processes))))
(println (format "[dry-run] db-sync server processes (port 18080): found %d, would kill %d"
(count (:found-pids db-sync-port-processes))
(count (:would-kill-pids db-sync-port-processes))))
(println (format "[dry-run] temp roots: found %d, would remove %d"
(count (:found-dirs temp-roots))
(count (:would-remove-dirs temp-roots)))))
(do
(println (format "db-worker-node processes: found %d, killed %d, failed %d"
(count (:found-pids processes))
(count (:killed-pids processes))
(count (:failed-pids processes))))
(println (format "db-sync server processes (port 18080): found %d, killed %d, failed %d"
(count (:found-pids db-sync-port-processes))
(count (:killed-pids db-sync-port-processes))
(count (:failed-pids db-sync-port-processes))))
(println (format "temp roots: found %d, removed %d, failed %d"
(count (:found-dirs temp-roots))
(count (:removed-dirs temp-roots))
(count (:failed-dirs temp-roots))))))
(flush)
{:status :ok
:dry-run? dry-run?
:processes processes
:db-sync-port-processes db-sync-port-processes
:temp-roots temp-roots})))
(defn- print-failure-details!
[error]
(let [data (ex-data error)]
(println (str " reason: " (.getMessage error)))
(when-let [cmd (:cmd data)]
(println (str " cmd: " cmd)))
(when-let [stream (:stream data)]
(println (str " stream: " stream)))
(when-let [snippet (:snippet data)]
(println (str " snippet: " snippet)))
(flush)))
(defn- format-step-label
[{:keys [phase step-index step-total]}]
(let [phase-name (name (or phase :command))]
(format "%s %d/%d" phase-name (or step-index 1) (or step-total 1))))
(defn- print-case-timings!
[timings]
(when (seq timings)
(println " step timings:")
(doseq [{:keys [elapsed-ms status cmd] :as timing} timings]
(println (format " - [%-12s] %6dms (%s) $ %s"
(format-step-label timing)
elapsed-ms
(name (or status :ok))
cmd)))
(flush)))
(defn- print-slow-steps!
[all-step-timings]
(when (seq all-step-timings)
(println "Slow steps (top 10):")
(doseq [{:keys [case-id elapsed-ms cmd] :as timing}
(->> all-step-timings
(sort-by :elapsed-ms >)
(take 10))]
(println (format " - %-45s [%-12s] %6dms $ %s"
case-id
(format-step-label timing)
elapsed-ms
cmd)))
(flush)))
(defn- progress-prefix
[{:keys [parallel? index total]} symbol]
(if parallel?
(str symbol " ")
(format "[%d/%d] %s " index total symbol)))
(defn- print-test-help!
[command-name suite]
(let [sync-suite? (= suite :sync)]
(println (str "Usage: bb -f cli-e2e/bb.edn " command-name " [options]"))
(println)
(println "Options:")
(println " -h, --help Show this help and exit")
(println " --skip-build Skip build preflight steps")
(println " -i, --include TAG Run only cases with matching tag (repeatable)")
(println " --case ID Run a single case by id")
(println (format " --jobs N %s (Default: %d)"
(if sync-suite?
"Run up to N sync cases in parallel"
"Run up to N non-sync cases in parallel")
default-cli-jobs))
(when sync-suite?
(println " --e2ee-password VALUE E2EE password for sync commands (Default: 11111)"))
(println " --verbose Enable verbose output")
(println " --timings Print per-step timings and slow-step summary")
(println)
(println "Examples:")
(if sync-suite?
(println (str " bb -f cli-e2e/bb.edn " command-name))
(println (str " bb -f cli-e2e/bb.edn " command-name " --skip-build")))
(println (str " bb -f cli-e2e/bb.edn " command-name
(if sync-suite?
" --jobs 4"
" --skip-build --jobs 4")))
(println (str " bb -f cli-e2e/bb.edn " command-name " -i smoke"))
(if sync-suite?
(println (str " bb -f cli-e2e/bb.edn " command-name " --case sync-upload-download-mvp"))
(println (str " bb -f cli-e2e/bb.edn " command-name " --skip-build --case global-help")))
(when sync-suite?
(println (str " bb -f cli-e2e/bb.edn " command-name " --e2ee-password 'my-secret'")))
(flush)))
(defn- test-suite!
[opts {:keys [suite command-name]
:or {suite default-suite
command-name "test"}}]
(let [suite (or suite default-suite)
opts (assoc opts :suite suite)]
(if (:help opts)
(do
(print-test-help! command-name suite)
{:status :help})
(let [started-at (System/nanoTime)
passed (atom 0)
failed (atom 0)
total-count (atom 0)
timings? (boolean (:timings opts))
all-step-timings (atom [])
detailed-case-log? (some? (:case opts))
parallel? (> (positive-jobs (:jobs opts)) 1)
base-run-command (or (:run-command opts) shell/run!)
run-command (if detailed-case-log?
(fn [{:keys [cmd phase step-index step-total] :as command-opts}]
(let [prefix (case phase
:setup (format " [setup %d/%d]" step-index step-total)
:main (format " [main %d/%d]" (or step-index 1) (or step-total 1))
:cleanup (format " [cleanup %d/%d]" step-index step-total)
" [command]")]
(println (str prefix " $ " cmd))
(flush)
(base-run-command (assoc command-opts :stream-output? true))))
base-run-command)]
(println "==> Running cli-e2e cases")
(when detailed-case-log?
(println (format "==> Detailed case logging enabled (--case %s)" (:case opts))))
(when timings?
(println "==> Step timing enabled (--timings)"))
(flush)
(try
(run! (assoc opts
:run-command run-command
:detailed-log? detailed-case-log?
:timings? timings?
:on-preflight-start (fn [_]
(println "==> Build preflight: running...")
(flush))
:on-preflight-complete (fn [{:keys [status]}]
(println (case status
:skipped "==> Build preflight: skipped (--skip-build)"
"==> Build preflight: completed"))
(flush))
:on-cases-ready (fn [{:keys [total]}]
(reset! total-count total)
(println (format "==> Prepared %d case(s), starting execution" total))
(flush))
:on-case-start (fn [{:keys [index total case]}]
(println (str (progress-prefix {:parallel? parallel?
:index index
:total total}
"▶")
(:id case)))
(flush))
:on-case-success (fn [{:keys [index total result elapsed-ms]}]
(swap! passed inc)
(println (format "%s%s (%dms)"
(progress-prefix {:parallel? parallel?
:index index
:total total}
"✓")
(:id result)
elapsed-ms))
(when timings?
(let [case-timings (vec (:timings result))]
(swap! all-step-timings into
(map #(assoc % :case-id (:id result)) case-timings))
(print-case-timings! case-timings)))
(flush))
:on-case-failure (fn [{:keys [index total case error elapsed-ms]}]
(swap! failed inc)
(println (format "%s%s (%dms)"
(progress-prefix {:parallel? parallel?
:index index
:total total}
"✗")
(:id case)
elapsed-ms))
(print-failure-details! error)
(when timings?
(let [case-timings (vec (:timings (ex-data error)))]
(swap! all-step-timings into
(map #(assoc % :case-id (:id case)) case-timings))
(print-case-timings! case-timings))))))
(println (format "Summary: %d passed, %d failed" @passed @failed))
(println (str "Selected cases: " @total-count))
(println (str "Duration: " (format-duration started-at)))
(when timings?
(print-slow-steps! @all-step-timings))
(catch Exception error
(let [failed-count (max 1 @failed)]
(println (format "Summary: %d passed, %d failed" @passed failed-count))
(println (str "Selected cases: " (max @total-count failed-count)))
(println (str "Duration: " (format-duration started-at)))
(when timings?
(print-slow-steps! @all-step-timings)))
(throw error)))))))
(defn test!
[opts]
(test-suite! opts {:suite :non-sync
:command-name "test"}))
(defn test-sync!
[opts]
(test-suite! opts {:suite :sync
:command-name "test-sync"}))

View File

@@ -0,0 +1,237 @@
(ns logseq.cli.e2e.manifests
(:require [clojure.edn :as edn]
[logseq.cli.e2e.paths :as paths]))
(def suite->manifest-files
{:non-sync {:inventory "non_sync_inventory.edn"
:cases "non_sync_cases.edn"}
:sync {:inventory "sync_inventory.edn"
:cases "sync_cases.edn"}})
(def default-suite :non-sync)
(def ^:private append-merge-keys
#{:setup :cmds :cleanup :tags})
(def ^:private deep-merge-keys
#{:vars :covers :expect})
(defn read-edn-file
[path]
(edn/read-string (slurp path)))
(defn- normalize-suite
[suite]
(let [suite' (cond
(nil? suite) default-suite
(keyword? suite) suite
(string? suite) (keyword suite)
:else suite)]
(when-not (contains? suite->manifest-files suite')
(throw (ex-info "Unknown cli-e2e suite"
{:suite suite
:known-suites (sort (keys suite->manifest-files))})))
suite'))
(defn- manifest-file
[suite kind]
(get-in suite->manifest-files [(normalize-suite suite) kind]))
(defn load-inventory
([]
(load-inventory nil))
([suite]
(read-edn-file (paths/spec-path (manifest-file suite :inventory)))))
(defn- normalize-extends
[extends]
(let [extends' (cond
(nil? extends) []
(keyword? extends) [extends]
(vector? extends) extends
:else
(throw (ex-info "Invalid :extends value in cli-e2e manifest"
{:extends extends
:expected "keyword | vector<keyword> | nil"})))
invalid-entries (remove keyword? extends')]
(when (seq invalid-entries)
(throw (ex-info "Invalid :extends entries in cli-e2e manifest"
{:extends extends
:invalid-entries (vec invalid-entries)
:expected "keyword | vector<keyword> | nil"})))
extends'))
(defn- parse-manifest
[manifest-data]
(if-not (map? manifest-data)
(throw (ex-info "Invalid cli-e2e manifest format"
{:manifest-type (type manifest-data)
:expected "{:templates {...} :cases [...]}"}))
(let [templates (or (:templates manifest-data) {})
cases (:cases manifest-data)]
(when-not (map? templates)
(throw (ex-info "Invalid cli-e2e manifest :templates format"
{:templates templates
:expected "map"})))
(when-not (vector? cases)
(throw (ex-info "Invalid cli-e2e manifest :cases format"
{:cases cases
:expected "vector"})))
(doseq [case cases]
(when-not (map? case)
(throw (ex-info "Invalid cli-e2e case format"
{:case case
:expected "map"}))))
{:templates templates
:cases cases})))
(defn- reachable-template-ids
[templates roots]
(loop [stack (vec roots)
visited #{}]
(if-let [template-id (peek stack)]
(if (contains? visited template-id)
(recur (pop stack) visited)
(let [template (get templates template-id)
parent-ids (if template
(normalize-extends (:extends template))
[])]
(recur (into (pop stack) parent-ids)
(conj visited template-id))))
visited)))
(defn- lint-manifest!
[{:keys [templates cases]}]
(let [template-ids (set (keys templates))
template-refs (mapcat (fn [[template-id template]]
(map (fn [target]
{:type :invalid-extends
:source-type :template
:source template-id
:target target})
(normalize-extends (:extends template))))
templates)
case-refs (mapcat (fn [[index case]]
(map (fn [target]
{:type :invalid-extends
:source-type :case
:source (or (:id case)
(str "case#" (inc index)))
:target target})
(normalize-extends (:extends case))))
(map-indexed vector cases))
invalid-extends-issues (->> (concat template-refs case-refs)
(filter (fn [{:keys [target]}]
(not (contains? template-ids target)))))
duplicate-id-issues (->> cases
(keep :id)
frequencies
(filter (fn [[_ count]] (> count 1)))
(sort-by first)
(map (fn [[case-id count]]
{:type :duplicate-case-id
:id case-id
:count count})))
roots (->> cases
(mapcat #(normalize-extends (:extends %)))
distinct)
used-template-ids (reachable-template-ids templates roots)
unused-template-issues (->> (keys templates)
(remove used-template-ids)
sort
(map (fn [template-id]
{:type :unused-template
:template template-id})))
issues (vec (concat invalid-extends-issues
duplicate-id-issues
unused-template-issues))]
(when (seq issues)
(throw (ex-info "cli-e2e manifest lint failed"
{:issues issues})))))
(defn- as-seq
[value]
(cond
(nil? value) []
(sequential? value) value
:else [value]))
(defn- deep-merge-maps
[left right]
(merge-with (fn [left-val right-val]
(if (and (map? left-val)
(map? right-val))
(deep-merge-maps left-val right-val)
right-val))
(or left {})
(or right {})))
(defn- merge-entry
[parent child]
(let [all-keys (set (concat (keys parent) (keys child)))]
(reduce (fn [acc key]
(let [parent-val (get parent key)
child-val (get child key)]
(assoc acc
key
(cond
(contains? append-merge-keys key)
(if (contains? child key)
(vec (concat (as-seq parent-val)
(as-seq child-val)))
(vec (as-seq parent-val)))
(contains? deep-merge-keys key)
(if (contains? child key)
(deep-merge-maps parent-val child-val)
parent-val)
(contains? child key)
child-val
:else
parent-val))))
{}
all-keys)))
(defn- resolve-template
[templates template-id stack]
(when (some #{template-id} stack)
(throw (ex-info "Circular template inheritance detected in cli-e2e manifest"
{:template template-id
:cycle (conj (vec stack) template-id)})))
(let [template (get templates template-id)]
(when-not template
(throw (ex-info "Unknown template in cli-e2e manifest"
{:template template-id
:known-templates (sort (keys templates))})))
(let [parent-ids (normalize-extends (:extends template))
parent-values (map #(resolve-template templates % (conj stack template-id))
parent-ids)]
(reduce merge-entry
{}
(concat parent-values
[(dissoc template :extends)])))))
(defn- expand-manifest-cases
[{:keys [templates cases]}]
(mapv (fn [case]
(let [parent-ids (normalize-extends (:extends case))
parent-values (map #(resolve-template templates % [])
parent-ids)]
(reduce merge-entry
{}
(concat parent-values
[(dissoc case :extends)]))))
cases))
(defn load-cases
([]
(load-cases nil))
([suite]
(let [manifest-data (-> (manifest-file suite :cases)
paths/spec-path
read-edn-file
parse-manifest)]
(lint-manifest! manifest-data)
(expand-manifest-cases manifest-data))))

View File

@@ -0,0 +1,49 @@
(ns logseq.cli.e2e.paths
(:require [babashka.fs :as fs]))
(defn- ancestors
[path]
(take-while some? (iterate fs/parent path)))
(defn- find-parent-named
[path name]
(some (fn [candidate]
(when (= name (fs/file-name candidate))
candidate))
(ancestors (fs/canonicalize path))))
(def ^:private cli-e2e-root-path
(or (some-> *file*
(find-parent-named "cli-e2e")
str)
(throw (ex-info "Unable to locate cli-e2e root"
{:file *file*}))))
(defn cli-e2e-root
[]
cli-e2e-root-path)
(defn repo-root
[]
(str (fs/parent cli-e2e-root-path)))
(defn repo-path
[& segments]
(str (apply fs/path (repo-root) segments)))
(defn cli-e2e-path
[& segments]
(str (apply fs/path (cli-e2e-root) segments)))
(defn spec-path
[filename]
(cli-e2e-path "spec" filename))
(defn required-artifacts
[]
[(repo-path "cli" "_build" "default" "dist" "logseq-cli.js")
(repo-path "static" "logseq-cli.js")
(repo-path "static" "db-worker-node.js")
(repo-path "dist" "db-worker-node.js")
(repo-path "dist" "db-worker-node-assets.json")
(repo-path "deps" "db-sync" "worker" "dist" "node-adapter.js")])

View File

@@ -0,0 +1,37 @@
(ns logseq.cli.e2e.preflight
(:require [babashka.fs :as fs]
[logseq.cli.e2e.paths :as paths]
[logseq.cli.e2e.shell :as shell]))
(def build-plan
[{:cmd "pnpm cli:release"}
{:cmd "pnpm db-worker-node:compile:bundle"}
{:cmd "pnpm --dir deps/db-sync build:node-adapter"}])
(defn missing-artifacts
([]
(missing-artifacts (paths/required-artifacts) fs/exists?))
([artifacts file-exists?]
(->> artifacts
(remove file-exists?)
vec)))
(defn run!
[{:keys [skip-build run-command file-exists?]
:or {run-command shell/run!
file-exists? fs/exists?}}]
(if skip-build
{:status :skipped
:commands []
:missing-artifacts []}
(let [artifacts (paths/required-artifacts)]
(doseq [{:keys [cmd]} build-plan]
(run-command {:cmd cmd
:dir (paths/repo-root)}))
(let [missing-after-build (missing-artifacts artifacts file-exists?)]
(when (seq missing-after-build)
(throw (ex-info "Build preflight completed but required artifacts are missing"
{:missing-artifacts missing-after-build})))
{:status :ok
:commands build-plan
:missing-artifacts []}))))

View File

@@ -0,0 +1,18 @@
(ns logseq.cli.e2e.report
(:require [clojure.string :as string]))
(defn format-missing-coverage
[{:keys [missing-commands missing-options]}]
(str
"Missing coverage\n"
(when (seq missing-commands)
(str "Commands:\n"
(string/join "\n" (map #(str "- " %) missing-commands))
"\n"))
(string/join
"\n"
(keep (fn [[scope options]]
(when (seq options)
(str (name scope) " options:\n"
(string/join "\n" (map #(str "- " %) options)))))
missing-options))))

View File

@@ -0,0 +1,280 @@
(ns logseq.cli.e2e.runner
(:require [babashka.fs :as fs]
[cheshire.core :as json]
[clojure.edn :as edn]
[clojure.string :as string]
[logseq.cli.e2e.paths :as paths]
[logseq.cli.e2e.shell :as shell]))
(defn shell-escape
[value]
(let [text (str value)]
(if (re-find #"[^\w@%+=:,./-]" text)
(str "'" (string/replace text #"'" "'\"'\"'") "'")
text)))
(def template-pattern #"\{\{([^}]+)\}\}")
(def ^:private e2e-env {"CLI_E2E_TEST" "1"
"NO_COLOR" "1"})
(defn- render-string
[template context]
(string/replace template
template-pattern
(fn [[_ key]]
(let [lookup-key (keyword (string/trim key))]
(when-not (contains? context lookup-key)
(throw (ex-info "Missing case template value"
{:template template
:key lookup-key
:context context})))
(str (get context lookup-key))))))
(defn- render-value
[value context]
(cond
(string? value) (render-string value context)
(map? value) (into {}
(map (fn [[k v]]
[k (render-value v context)]))
value)
(vector? value) (mapv #(render-value % context) value)
(seq? value) (doall (map #(render-value % context) value))
:else value))
(defn render-case
[case context]
(render-value case context))
(defn- ensure-contains!
[id cmd output snippets stream]
(doseq [snippet (or snippets [])]
(let [normalize-pathish-text (fn [text]
(-> (str text)
(string/replace "\\\\" "\\")
(string/replace "\\" "/")
(string/replace "\r" "")))
matches? (or (string/includes? output snippet)
(string/includes? (normalize-pathish-text output)
(normalize-pathish-text snippet)))]
(when-not matches?
(throw (ex-info (str stream " did not contain expected text")
{:id id
:cmd cmd
:snippet snippet
:output output
:stream stream}))))))
(defn- ensure-not-contains!
[id cmd output snippets stream]
(doseq [snippet (or snippets [])]
(when (string/includes? output snippet)
(throw (ex-info (str stream " contained forbidden text")
{:id id
:cmd cmd
:snippet snippet
:output output
:stream stream})))))
(defn- ensure-equals!
[id cmd actual expected label]
(when (and (some? expected)
(not= expected actual))
(throw (ex-info (str label " did not match")
{:id id
:cmd cmd
:expected expected
:actual actual}))))
(defn- ensure-json-paths!
[id cmd output expected-paths]
(when expected-paths
(let [payload (json/parse-string output true)]
(doseq [[path expected] expected-paths]
(let [actual (get-in payload path)]
(when-not (= expected actual)
(throw (ex-info "stdout JSON path did not match"
{:id id
:cmd cmd
:path path
:expected expected
:actual actual
:payload payload}))))))))
(defn- ensure-edn-paths!
[id cmd output expected-paths]
(when expected-paths
(let [payload (edn/read-string output)]
(doseq [[path expected] expected-paths]
(let [actual (get-in payload path)]
(when-not (= expected actual)
(throw (ex-info "stdout EDN path did not match"
{:id id
:cmd cmd
:path path
:expected expected
:actual actual
:payload payload}))))))))
(defn assert-result!
[{:keys [id expect]} {:keys [cmd exit out err] :as result}]
(when-let [expected-exit (:exit expect)]
(when-not (= expected-exit exit)
(throw (ex-info "Unexpected exit code"
{:id id
:cmd cmd
:expected expected-exit
:actual exit
:result result}))))
(ensure-equals! id cmd out (:stdout-equals expect) "stdout")
(ensure-equals! id cmd err (:stderr-equals expect) "stderr")
(ensure-contains! id cmd out (:stdout-contains expect) "stdout")
(ensure-contains! id cmd err (:stderr-contains expect) "stderr")
(ensure-not-contains! id cmd out (:stdout-not-contains expect) "stdout")
(ensure-not-contains! id cmd err (:stderr-not-contains expect) "stderr")
(ensure-json-paths! id cmd out (:stdout-json-paths expect))
(ensure-edn-paths! id cmd out (:stdout-edn-paths expect))
nil)
(defn- default-context
[case context]
(let [tmp-dir (or (:tmp-dir context)
(str (fs/create-temp-dir {:prefix (str "logseq-cli-e2e-" (:id case) "-")})))
root-dir (or (:root-dir context)
tmp-dir)
config-path (or (:config-path context)
(str (fs/path root-dir "cli.edn")))
export-path (or (:export-path context)
(str (fs/path tmp-dir "graph-export.edn")))
graph (or (:graph context)
(:graph case)
(str "cli-e2e-" (:id case)))]
(fs/create-dirs (fs/path root-dir "graphs"))
(spit config-path "{:output-format :json}\n")
{:tmp-dir tmp-dir
:tmp-dir-arg (shell-escape tmp-dir)
:root-dir root-dir
:root-dir-arg (shell-escape root-dir)
:config-path config-path
:config-path-arg (shell-escape config-path)
:export-path export-path
:export-path-arg (shell-escape export-path)
:repo-root (paths/repo-root)
:repo-root-arg (shell-escape (paths/repo-root))
:cli (str "node " (shell-escape (paths/repo-path "static" "logseq-cli.js")))
:graph graph
:graph-arg (shell-escape graph)}))
(defn- case-context
[case {:keys [context]}]
(merge (default-context case context)
context
(:vars case)))
(defn- run-command!
[command context {:keys [run-command stdin allow-failure phase step-index step-total case-id]}]
(run-command {:cmd (render-string command context)
:dir (paths/repo-root)
:env e2e-env
:stdin (some-> stdin (render-string context))
:phase phase
:step-index step-index
:step-total step-total
:case-id case-id
:throw? (not allow-failure)}))
(defn- elapsed-ms
[started-at]
(long (/ (- (System/nanoTime) started-at) 1000000)))
(defn- run-step!
[timings command context {:keys [timings? phase step-index step-total]
:as run-opts}]
(if-not timings?
(run-command! command context run-opts)
(let [started-at (System/nanoTime)]
(try
(let [result (run-command! command context run-opts)]
(swap! timings conj {:phase phase
:step-index step-index
:step-total step-total
:cmd (:cmd result)
:elapsed-ms (elapsed-ms started-at)
:status :ok})
result)
(catch Exception error
(swap! timings conj {:phase phase
:step-index step-index
:step-total step-total
:cmd (or (:cmd (ex-data error))
(render-string command context))
:elapsed-ms (elapsed-ms started-at)
:status :failed})
(throw error))))))
(defn run-case!
[case {:keys [run-command detailed-log? timings?]
:or {run-command shell/run!}
:as opts}]
(let [context (case-context case opts)
rendered (render-case case context)
case-id (:id rendered)
timings? (boolean timings?)
timings (atom [])
cleanup-commands (vec (:cleanup rendered))
setup-commands (vec (:setup rendered))
main-commands (vec (:cmds rendered))
cleanup! (fn []
(doseq [[idx command] (map-indexed vector cleanup-commands)]
(try
(run-step! timings command context {:run-command run-command
:timings? timings?
:allow-failure true
:phase (when (or detailed-log? timings?) :cleanup)
:step-index (inc idx)
:step-total (count cleanup-commands)
:case-id case-id})
(catch Exception _
nil))))]
(try
(doseq [[idx command] (map-indexed vector setup-commands)]
(run-step! timings command context {:run-command run-command
:timings? timings?
:phase (when (or detailed-log? timings?) :setup)
:step-index (inc idx)
:step-total (count setup-commands)
:case-id case-id}))
(let [main-total (count main-commands)
_ (when (zero? main-total)
(throw (ex-info "Missing case commands"
{:id case-id
:case rendered})))
result (reduce (fn [_ [idx command]]
(let [last-step? (= idx (dec main-total))]
(run-step! timings command context {:run-command run-command
:timings? timings?
:stdin (when last-step? (:stdin rendered))
:allow-failure last-step?
:phase (when (or detailed-log? timings?) :main)
:step-index (inc idx)
:step-total main-total
:case-id case-id})))
nil
(map-indexed vector main-commands))]
(assert-result! rendered result)
(cleanup!)
(cond-> {:id case-id
:status :ok
:cmd (:cmd result)
:result result
:context context}
timings? (assoc :timings @timings)))
(catch Exception error
(cleanup!)
(if timings?
(throw (ex-info (.getMessage error)
(assoc (or (ex-data error) {})
:timings @timings
:case-id case-id)
error))
(throw error))))))

View File

@@ -0,0 +1,67 @@
(ns logseq.cli.e2e.shell
(:require [babashka.process :as process])
(:import [java.io ByteArrayOutputStream OutputStream]))
(defn- utf8-string
[^ByteArrayOutputStream payload]
(.toString payload "UTF-8"))
(defn- capture-stream
[target stream-output?]
(let [payload (ByteArrayOutputStream.)]
{:payload payload
:stream (if stream-output?
(proxy [OutputStream] []
(write
([byte]
(.write payload byte)
(.write target byte)
(.flush target))
([bytes offset length]
(.write payload bytes offset length)
(.write target bytes offset length)
(.flush target)))
(flush []
(.flush target))
(close []
(.flush target)))
payload)}))
(defn- default-executor
[cmd {:keys [dir extra-env in stream-output?]}]
(let [{out-payload :payload
out-stream :stream} (capture-stream System/out stream-output?)
{err-payload :payload
err-stream :stream} (capture-stream System/err stream-output?)
result @(process/process {:continue true
:dir dir
:extra-env extra-env
:in in
:out out-stream
:err err-stream}
"bash"
"-c"
cmd)]
{:exit (:exit result 0)
:out (utf8-string out-payload)
:err (utf8-string err-payload)}))
(defn run!
[{:keys [cmd dir env stdin executor throw? stream-output?]
:or {throw? true}}]
(let [runner (or executor default-executor)
result (runner cmd {:dir dir
:extra-env env
:in stdin
:stream-output? stream-output?})
exit (:exit result 0)
payload {:cmd cmd
:dir dir
:exit exit
:out (:out result "")
:err (:err result "")}]
(when (and throw?
(not (zero? exit)))
(throw (ex-info "Shell command failed"
payload)))
payload))

View File

@@ -0,0 +1,140 @@
(ns logseq.cli.e2e.sync-fixture
(:require [babashka.fs :as fs]
[clojure.string :as string]
[logseq.cli.e2e.paths :as paths]
[logseq.cli.e2e.runner :as runner]
[logseq.cli.e2e.shell :as shell]))
(def default-sync-port "18080")
(def default-e2ee-password "11111")
(def default-user-keys-graph "sync-e2e-user-keys-bootstrap")
(def ^:private heavy-setup-patterns
[#"^mkdir -p '\{\{tmp-dir\}\}/home/logseq'$"
#"^cp ~/logseq/auth\.json\b"
#"prepare_sync_config\.py"
#"db_sync_server\.py'? start"])
(def ^:private heavy-cleanup-patterns
[#"db_sync_server\.py'? stop"])
(defn- shell-quote
[value]
(runner/shell-escape value))
(defn- heavy-command?
[command patterns]
(boolean (some #(re-find % command) patterns)))
(defn- case-local-setup-prefix
[]
["mkdir -p '{{tmp-dir}}/home/logseq'"
"cp ~/logseq/auth.json '{{tmp-dir}}/home/logseq/auth.json'"
"python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{config-path}}' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'"
"python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{tmp-dir}}/cli-b.edn' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'"])
(def ^:private case-local-resource-markers
["{{cli-home}}"
"{{config-path}}"
"{{config-path-arg}}"
"{{tmp-dir}}/cli-b.edn"
"{{auth-path}}"
"{{home-dir}}"])
(defn- requires-case-local-resources?
[command]
(boolean (some #(string/includes? command %) case-local-resource-markers)))
(defn- suite-user-keys-bootstrap-command
[suite-tmp-dir]
(let [lock-dir (shell-quote (str (fs/path suite-tmp-dir "user-rsa-keys.lock")))
done-file (shell-quote (str (fs/path suite-tmp-dir "user-rsa-keys.ready")))]
(format (str "LOCK_DIR=%s; DONE_FILE=%s; "
"if [ -f \"$DONE_FILE\" ]; then exit 0; fi; "
"while ! mkdir \"$LOCK_DIR\" 2>/dev/null; do [ -f \"$DONE_FILE\" ] && exit 0; sleep 0.1; done; "
"trap 'rmdir \"$LOCK_DIR\" 2>/dev/null || true' EXIT; "
"if [ ! -f \"$DONE_FILE\" ]; then "
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{user-keys-graph-arg}} >/dev/null 2>/dev/null || true; "
"if {{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync ensure-keys --graph {{user-keys-graph-arg}} --e2ee-password {{e2ee-password-arg}} --upload-keys >/dev/null; then "
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{user-keys-graph-arg}} >/dev/null 2>/dev/null || true; "
"touch \"$DONE_FILE\"; "
"else exit 1; fi; "
"fi")
lock-dir
done-file)))
(defn before-suite!
[{:keys [run-command sync-port]
:or {run-command shell/run!
sync-port default-sync-port}}]
(let [sync-port (str sync-port)
suite-tmp-dir (str (fs/create-temp-dir {:prefix "logseq-cli-e2e-sync-suite-"}))
db-sync-pid-file (str (fs/path suite-tmp-dir "db-sync-server.pid"))
db-sync-log-file (str (fs/path suite-tmp-dir "db-sync-server.log"))
db-sync-root-dir (str (fs/path suite-tmp-dir "db-sync-server-data"))
sync-http-base (str "http://127.0.0.1:" sync-port)
sync-ws-url (str "ws://127.0.0.1:" sync-port "/sync/%s")
auth-path (str (fs/path (System/getProperty "user.home") "logseq" "auth.json"))
start-db-sync-cmd (format "python3 %s start --repo-root %s --pid-file %s --log-file %s --data-dir %s --port %s --startup-timeout-s 60 --auth-path %s"
(shell-quote (paths/repo-path "cli-e2e" "scripts" "db_sync_server.py"))
(shell-quote (paths/repo-root))
(shell-quote db-sync-pid-file)
(shell-quote db-sync-log-file)
(shell-quote db-sync-root-dir)
sync-port
(shell-quote auth-path))]
(run-command {:cmd start-db-sync-cmd
:dir (paths/repo-root)})
{:suite-tmp-dir suite-tmp-dir
:db-sync-pid-file db-sync-pid-file
:db-sync-log-file db-sync-log-file
:db-sync-root-dir db-sync-root-dir
:sync-port sync-port
:sync-http-base sync-http-base
:sync-ws-url sync-ws-url}))
(defn prepare-case
[case {:keys [suite-tmp-dir sync-port sync-http-base sync-ws-url e2ee-password]}]
(let [e2ee-password (or e2ee-password default-e2ee-password)
user-keys-graph (or (get-in case [:vars :user-keys-graph])
default-user-keys-graph)
setup-commands (vec (:setup case))
insertion-point? (fn [command]
(or (heavy-command? command heavy-setup-patterns)
(requires-case-local-resources? command)))
leading-setup (->> setup-commands
(take-while #(not (insertion-point? %)))
vec)
trailing-setup (->> setup-commands
(drop (count leading-setup))
(remove #(heavy-command? % heavy-setup-patterns))
vec)
bootstrap-setup (when suite-tmp-dir
[(suite-user-keys-bootstrap-command suite-tmp-dir)])
cleanup' (->> (:cleanup case)
(remove #(heavy-command? % heavy-cleanup-patterns))
vec)]
(-> case
(update :vars merge {:sync-port sync-port
:sync-http-base sync-http-base
:sync-ws-url sync-ws-url
:e2ee-password e2ee-password
:e2ee-password-arg (shell-quote e2ee-password)
:user-keys-graph-arg (shell-quote user-keys-graph)})
(assoc :setup (vec (concat leading-setup
(case-local-setup-prefix)
bootstrap-setup
trailing-setup)))
(assoc :cleanup cleanup'))))
(defn after-suite!
[{:keys [db-sync-pid-file]}
{:keys [run-command]
:or {run-command shell/run!}}]
(when (and (string? db-sync-pid-file)
(not (string/blank? db-sync-pid-file)))
(run-command {:cmd (format "python3 %s stop --pid-file %s"
(shell-quote (paths/repo-path "cli-e2e" "scripts" "db_sync_server.py"))
(shell-quote db-sync-pid-file))
:dir (paths/repo-root)
:throw? false})))

View File

@@ -0,0 +1,23 @@
(ns logseq.cli.e2e.test-runner
(:require [clojure.test :as test]))
(def test-namespaces
'[logseq.cli.e2e.coverage-test
logseq.cli.e2e.preflight-test
logseq.cli.e2e.paths-test
logseq.cli.e2e.shell-test
logseq.cli.e2e.runner-test
logseq.cli.e2e.cleanup-test
logseq.cli.e2e.manifests-test
logseq.cli.e2e.sync-fixture-test
logseq.cli.e2e.main-test])
(defn run!
[_opts]
(doseq [ns-sym test-namespaces]
(require ns-sym))
(let [{:keys [fail error]} (apply test/run-tests test-namespaces)]
(when (pos? (+ fail error))
(throw (ex-info "cli-e2e unit tests failed"
{:fail fail
:error error})))))

View File

@@ -0,0 +1,149 @@
(ns logseq.cli.e2e.cleanup-test
(:require [babashka.fs :as fs]
[clojure.test :refer [deftest is testing]]
[logseq.cli.e2e.cleanup :as cleanup]))
(deftest list-cli-e2e-db-worker-pids-filters-processes
(let [shell-fn (fn [& _]
{:exit 0
:out (str " 101 node /repo/dist/db-worker-node.js --repo graph-a --root-dir /tmp/logseq-cli-e2e-graph-a-123/graphs --owner-source cli\n"
" 202 node /repo/dist/db-worker-node.js --repo production --root-dir /tmp/production-graphs --owner-source cli\n"
" 303 node /repo/static/logseq-cli.js graph list\n"
" 404 /usr/bin/python3 background.py\n"
" 505 node /repo/static/db-worker-node.js --repo graph-b --root-dir /private/tmp/logseq-cli-e2e-graph-b-999/graphs --owner-source cli\n")
:err ""})]
(is (= [101 505]
(cleanup/list-cli-e2e-db-worker-pids {:shell-fn shell-fn}))))
(testing "throws when ps invocation fails"
(is (thrown-with-msg?
clojure.lang.ExceptionInfo
#"Unable to scan processes"
(cleanup/list-cli-e2e-db-worker-pids {:shell-fn (fn [& _]
{:exit 1
:out ""
:err "permission denied"})})))))
(deftest cleanup-db-worker-processes-separates-killed-and-failed
(let [killed (atom [])
result (cleanup/cleanup-db-worker-processes! {:list-pids-fn (fn [] [11 22 33])
:kill-pid-fn (fn [pid]
(swap! killed conj pid)
(if (= pid 22)
:failed
:killed))})]
(is (= [11 22 33] (:found-pids result)))
(is (= [11 33] (:killed-pids result)))
(is (= [22] (:failed-pids result)))
(is (= [11 22 33] @killed))))
(deftest cleanup-db-worker-processes-dry-run-does-not-kill
(let [killed (atom [])
result (cleanup/cleanup-db-worker-processes! {:dry-run true
:list-pids-fn (fn [] [11 22])
:kill-pid-fn (fn [pid]
(swap! killed conj pid)
:killed)})]
(is (= [11 22] (:found-pids result)))
(is (true? (:dry-run? result)))
(is (= [11 22] (:would-kill-pids result)))
(is (= [] (:killed-pids result)))
(is (= [] (:failed-pids result)))
(is (= [] @killed))))
(deftest list-cli-e2e-db-sync-port-pids-filters-port-18080
(let [shell-fn (fn [& _]
{:exit 0
:out (str "COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\n"
"node 111 me 13u IPv6 0x0 0t0 TCP *:18080 (LISTEN)\n"
"python 222 me 13u IPv4 0x0 0t0 TCP 127.0.0.1:18080 (LISTEN)\n"
"node 333 me 13u IPv6 0x0 0t0 TCP *:18081 (LISTEN)\n")
:err ""})]
(is (= [111 222]
(cleanup/list-cli-e2e-db-sync-port-pids {:shell-fn shell-fn}))))
(testing "returns empty when no listener exists"
(is (= []
(cleanup/list-cli-e2e-db-sync-port-pids {:shell-fn (fn [& _]
{:exit 1
:out ""
:err ""})}))))
(testing "throws when lsof invocation fails"
(is (thrown-with-msg?
clojure.lang.ExceptionInfo
#"Unable to scan db-sync server port listeners"
(cleanup/list-cli-e2e-db-sync-port-pids {:shell-fn (fn [& _]
{:exit 1
:out ""
:err "permission denied"})})))))
(deftest cleanup-db-sync-port-processes-separates-killed-and-failed
(let [killed (atom [])
result (cleanup/cleanup-db-sync-port-processes! {:list-pids-fn (fn [] [44 55])
:kill-pid-fn (fn [pid]
(swap! killed conj pid)
(if (= pid 55)
:failed
:killed))})]
(is (= [44 55] (:found-pids result)))
(is (= [44] (:killed-pids result)))
(is (= [55] (:failed-pids result)))
(is (= [44 55] @killed))))
(deftest cleanup-db-sync-port-processes-dry-run-does-not-kill
(let [killed (atom [])
result (cleanup/cleanup-db-sync-port-processes! {:dry-run true
:list-pids-fn (fn [] [44])
:kill-pid-fn (fn [pid]
(swap! killed conj pid)
:killed)})]
(is (= [44] (:found-pids result)))
(is (true? (:dry-run? result)))
(is (= [44] (:would-kill-pids result)))
(is (= [] (:killed-pids result)))
(is (= [] (:failed-pids result)))
(is (= [] @killed))))
(deftest cleanup-temp-roots-removes-only-cli-e2e-temp-roots
(let [tmp-root (fs/create-temp-dir {:prefix "cleanup-e2e-test-"})
matching-root (fs/path tmp-root "logseq-cli-e2e-case-a-123")
another-matching-root (fs/path tmp-root "logseq-cli-e2e-case-b-456")
non-matching-root (fs/path tmp-root "other-case")]
(try
(fs/create-dirs (fs/path matching-root "graphs"))
(fs/create-dirs (fs/path matching-root "home" "logseq"))
(fs/create-dirs (fs/path another-matching-root "graphs"))
(fs/create-dirs (fs/path non-matching-root "graphs"))
(let [result (cleanup/cleanup-temp-roots! {:tmp-root (str tmp-root)})
expected-dirs (sort [(str matching-root)
(str another-matching-root)])]
(is (= expected-dirs
(sort (:found-dirs result))))
(is (= expected-dirs
(sort (:removed-dirs result))))
(is (empty? (:failed-dirs result)))
(is (false? (fs/exists? matching-root)))
(is (false? (fs/exists? another-matching-root)))
(is (true? (fs/exists? non-matching-root))))
(finally
(fs/delete-tree tmp-root)))))
(deftest cleanup-temp-roots-dry-run-does-not-delete
(let [deleted (atom [])
result (cleanup/cleanup-temp-roots! {:dry-run true
:list-dirs-fn (fn [] ["/tmp/logseq-cli-e2e-a"
"/tmp/logseq-cli-e2e-b"])
:delete-dir-fn (fn [dir]
(swap! deleted conj dir))})]
(is (= ["/tmp/logseq-cli-e2e-a"
"/tmp/logseq-cli-e2e-b"]
(:found-dirs result)))
(is (true? (:dry-run? result)))
(is (= ["/tmp/logseq-cli-e2e-a"
"/tmp/logseq-cli-e2e-b"]
(:would-remove-dirs result)))
(is (= [] (:removed-dirs result)))
(is (= [] (:failed-dirs result)))
(is (= [] @deleted))))

View File

@@ -0,0 +1,97 @@
from __future__ import annotations
import importlib.util
import json
from pathlib import Path
from types import SimpleNamespace
MODULE_PATH = Path(__file__).resolve().parents[4] / "scripts" / "compare_graph_queries.py"
spec = importlib.util.spec_from_file_location("compare_graph_queries", MODULE_PATH)
compare_graph_queries = importlib.util.module_from_spec(spec)
assert spec.loader is not None
spec.loader.exec_module(compare_graph_queries)
def test_parse_args_supports_repeated_queries(monkeypatch) -> None:
monkeypatch.setattr(
compare_graph_queries.sys,
"argv",
[
"compare_graph_queries.py",
"--cli",
"/tmp/logseq-cli.js",
"--graph",
"demo",
"--config-a",
"/tmp/a.edn",
"--root-dir-a",
"/tmp/a",
"--config-b",
"/tmp/b.edn",
"--root-dir-b",
"/tmp/b",
"--query",
"[:find ?x]",
"--query",
"[:find ?y]",
],
)
args = compare_graph_queries.parse_args()
assert args.root_dir_a == "/tmp/a"
assert args.root_dir_b == "/tmp/b"
assert args.query == ["[:find ?x]", "[:find ?y]"]
def test_main_batches_multiple_queries_in_one_process(tmp_path: Path) -> None:
left_queries = []
right_queries = []
printed = []
cli_path = tmp_path / "logseq-cli.js"
cli_path.write_text("// mock cli\n")
def fake_run_query(cli_path, config_path, root_dir, graph, query):
record = {
"cli_path": str(cli_path),
"config_path": str(config_path),
"root_dir": str(root_dir),
"graph": graph,
"query": query,
}
if str(config_path).endswith("a.edn"):
left_queries.append(record)
else:
right_queries.append(record)
return {
"payload": {"status": "ok"},
"result": [{"title": query}],
}
compare_graph_queries.run_query = fake_run_query
compare_graph_queries.parse_args = lambda: SimpleNamespace(
cli=str(cli_path),
graph="demo",
query=["[:find ?x]", "[:find ?y]"],
config_a="/tmp/a.edn",
root_dir_a="/tmp/a",
config_b="/tmp/b.edn",
root_dir_b="/tmp/b",
require_result=False,
)
compare_graph_queries.print = lambda value, **kwargs: printed.append(json.loads(value))
compare_graph_queries.main()
assert [item["query"] for item in left_queries] == ["[:find ?x]", "[:find ?y]"]
assert [item["query"] for item in right_queries] == ["[:find ?x]", "[:find ?y]"]
assert printed == [
{
"status": "ok",
"results": {
"[:find ?x]": [{"title": "[:find ?x]"}],
"[:find ?y]": [{"title": "[:find ?y]"}],
},
}
]

View File

@@ -0,0 +1,58 @@
(ns logseq.cli.e2e.coverage-test
(:require [clojure.test :refer [deftest is testing]]
[logseq.cli.e2e.coverage :as coverage]
[logseq.cli.e2e.manifests :as manifests]))
(def sample-inventory
{:excluded-command-prefixes ["sync" "login" "logout"]
:scopes {:global {:options ["--help" "--version"]}
:graph {:commands ["graph list" "graph create"]
:options ["--file" "--type"]}}})
(deftest reports-missing-commands-and-options
(let [cases [{:id "global-help"
:covers {:options {:global ["--help"]}}}
{:id "graph-list"
:covers {:commands ["graph list"]
:options {:graph ["--file"]}}}]
report (coverage/coverage-report sample-inventory cases)]
(is (= ["graph create"] (:missing-commands report)))
(is (= ["--version"] (get-in report [:missing-options :global])))
(is (= ["--type"] (get-in report [:missing-options :graph])))))
(deftest rejects-excluded-commands-in-inventory
(let [inventory (assoc-in sample-inventory [:scopes :bad :commands] ["login"])]
(is (thrown-with-msg?
clojure.lang.ExceptionInfo
#"Excluded commands"
(coverage/validate-inventory! inventory)))))
(deftest rejects-excluded-commands-in-case-covers
(let [cases [{:id "bad-login"
:covers {:commands ["logout"]}}]]
(is (thrown-with-msg?
clojure.lang.ExceptionInfo
#"Excluded commands"
(coverage/validate-cases! sample-inventory cases)))))
(deftest complete-coverage-is-recognized
(let [cases [{:id "global"
:covers {:options {:global ["--help" "--version"]}}}
{:id "graph-create"
:covers {:commands ["graph create"]
:options {:graph ["--type"]}}}
{:id "graph-list"
:covers {:commands ["graph list"]
:options {:graph ["--file"]}}}]
report (coverage/coverage-report sample-inventory cases)]
(is (coverage/complete? report))))
(deftest sync-suite-manifests-cover-mvp-commands
(let [inventory (manifests/load-inventory :sync)
cases (manifests/load-cases :sync)
covered-commands (set (mapcat #(get-in % [:covers :commands]) cases))
report (coverage/coverage-report inventory cases)]
(is (contains? covered-commands "sync upload"))
(is (contains? covered-commands "sync download"))
(is (contains? covered-commands "sync status"))
(is (coverage/complete? report))))

View File

@@ -0,0 +1,798 @@
(ns logseq.cli.e2e.main-test
(:require [babashka.cli :as cli]
[clojure.string :as string]
[clojure.test :refer [deftest is]]
[logseq.cli.e2e.cleanup :as cleanup]
[logseq.cli.e2e.main :as main]
[logseq.cli.e2e.manifests :as manifests]
[logseq.cli.e2e.sync-fixture :as sync-fixture]))
(def sample-cases
[{:id "global-help"
:cmds ["node static/logseq-cli.js --help"]
:covers {:options {:global ["--help"]}}
:tags [:global :smoke]}
{:id "graph-create"
:cmds ["node static/logseq-cli.js graph create --graph demo"]
:covers {:commands ["graph create"]
:options {:graph ["--type"]}}
:tags [:graph]}
{:id "graph-list"
:cmds ["node static/logseq-cli.js graph list"]
:covers {:commands ["graph list"]
:options {:graph ["--file"]}}
:tags [:graph :smoke]}])
(def complete-inventory
{:excluded-command-prefixes ["sync" "login" "logout"]
:scopes {:global {:options ["--help"]}
:graph {:commands ["graph create" "graph list"]
:options ["--type" "--file"]}}})
(def cli-parse-config
{:alias {:i :include
:h :help}
:spec {:jobs {:default 4}}
:coerce {:include []
:help :boolean
:dry-run :boolean
:skip-build :boolean
:verbose :boolean
:timings :boolean
:jobs :long}})
(deftest cli-opts-parses-jobs-as-integer
(is (= 3
(:jobs (cli/parse-opts ["--jobs" "3"] cli-parse-config)))))
(deftest cli-opts-defaults-jobs-to-four
(is (= 4
(:jobs (cli/parse-opts [] cli-parse-config)))))
(deftest cli-opts-rejects-non-integer-jobs
(is (thrown-with-msg?
clojure.lang.ExceptionInfo
#"Coerce failure"
(cli/parse-opts ["--jobs" "nope"] cli-parse-config))))
(deftest run-rejects-jobs-less-than-one
(is (thrown-with-msg?
clojure.lang.ExceptionInfo
#"--jobs must be a positive integer"
(main/run! {:inventory complete-inventory
:cases sample-cases
:skip-build true
:jobs 0
:run-command (fn [_]
{:exit 0
:out ""
:err ""})
:run-case (fn [case _opts]
{:id (:id case)
:status :ok})}))))
(deftest run-non-sync-uses-parallel-runner-when-jobs-greater-than-one
(let [parallel-call (atom nil)
serial-called? (atom false)]
(with-redefs [main/run-selected-cases-in-parallel! (fn [selected-cases run-case run-command opts]
(reset! parallel-call {:case-ids (mapv :id selected-cases)
:run-case run-case
:run-command run-command
:jobs (:jobs opts)})
[{:id "global-help" :status :ok}
{:id "graph-list" :status :ok}])
main/run-selected-cases! (fn [& _]
(reset! serial-called? true)
(throw (ex-info "serial runner should not be used" {})))]
(let [result (main/run! {:inventory complete-inventory
:cases sample-cases
:include ["smoke"]
:skip-build true
:jobs 2
:run-command (fn [_]
{:exit 0
:out ""
:err ""})
:run-case (fn [case _opts]
{:id (:id case)
:status :ok})})]
(is (= :ok (:status result)))
(is (= ["global-help" "graph-list"] (:case-ids @parallel-call)))
(is (= 2 (:jobs @parallel-call)))
(is (false? @serial-called?))))))
(deftest run-sync-suite-uses-parallel-runner-when-jobs-greater-than-one
(let [parallel-call (atom nil)
serial-called? (atom false)
sync-inventory {:excluded-command-prefixes ["login" "logout"]
:scopes {:sync {:commands ["sync upload" "sync status"]
:options []}}}
sync-cases [{:id "sync-upload"
:cmds ["node static/logseq-cli.js sync upload"]
:covers {:commands ["sync upload"]}}
{:id "sync-status"
:cmds ["node static/logseq-cli.js sync status"]
:covers {:commands ["sync status"]}}]]
(with-redefs [main/run-selected-cases-in-parallel! (fn [selected-cases run-case run-command opts]
(reset! parallel-call {:case-ids (mapv :id selected-cases)
:run-case run-case
:run-command run-command
:jobs (:jobs opts)})
(mapv (fn [case]
{:id (:id case)
:status :ok})
selected-cases))
main/run-selected-cases! (fn [& _]
(reset! serial-called? true)
(throw (ex-info "serial runner should not be used" {})))]
(let [result (main/run! {:suite :sync
:inventory sync-inventory
:cases sync-cases
:skip-build true
:jobs 4
:run-command (fn [_]
{:exit 0
:out ""
:err ""})
:run-case (fn [case _opts]
{:id (:id case)
:status :ok})})]
(is (= :ok (:status result)))
(is (= ["sync-upload" "sync-status"] (:case-ids @parallel-call)))
(is (= 4 (:jobs @parallel-call)))
(is (false? @serial-called?))))))
(deftest run-jobs-one-keeps-serial-runner
(let [serial-call (atom nil)
parallel-called? (atom false)]
(with-redefs [main/run-selected-cases! (fn [selected-cases run-case run-command opts]
(reset! serial-call {:case-ids (mapv :id selected-cases)
:run-case run-case
:run-command run-command
:jobs (:jobs opts)})
(mapv (fn [case]
{:id (:id case)
:status :ok})
selected-cases))
main/run-selected-cases-in-parallel! (fn [& _]
(reset! parallel-called? true)
(throw (ex-info "parallel runner should not be used" {})))]
(let [result (main/run! {:inventory complete-inventory
:cases sample-cases
:include ["smoke"]
:skip-build true
:jobs 1
:run-command (fn [_]
{:exit 0
:out ""
:err ""})
:run-case (fn [case _opts]
{:id (:id case)
:status :ok})})]
(is (= :ok (:status result)))
(is (= ["global-help" "graph-list"] (:case-ids @serial-call)))
(is (= 1 (:jobs @serial-call)))
(is (false? @parallel-called?))))))
(deftest parallel-runner-collects-completions-before-rethrowing-failure
(let [started (atom [])
finished (atom [])
started-latch (java.util.concurrent.CountDownLatch. 2)
release-success (promise)
cases [{:id "global-help"}
{:id "graph-list"}]
error (try
(main/run-selected-cases-in-parallel!
cases
(fn [case _opts]
(swap! started conj (:id case))
(.countDown started-latch)
(.await started-latch)
(if (= "graph-list" (:id case))
(do
(swap! finished conj [:failed (:id case)])
(deliver release-success true)
(throw (ex-info "boom" {:id (:id case)})))
(do
@release-success
(swap! finished conj [:ok (:id case)])
{:id (:id case)
:status :ok})))
(fn [_]
{:exit 0
:out ""
:err ""})
{:jobs 2})
nil
(catch Exception ex
ex))]
(is (instance? Exception error))
(is (= #{"global-help" "graph-list"}
(set @started)))
(is (= #{[:ok "global-help"]
[:failed "graph-list"]}
(set @finished)))))
(deftest parallel-runner-elapsed-ms-starts-when-case-begins-running
(let [events (atom [])]
(main/run-selected-cases-in-parallel!
[{:id "slow-1"}
{:id "slow-2"}
{:id "fast-after-queue"}]
(fn [test-case _opts]
(clojure.core/case (:id test-case)
"slow-1" (Thread/sleep 180)
"slow-2" (Thread/sleep 180)
"fast-after-queue" (Thread/sleep 10))
{:id (:id test-case)
:status :ok})
(fn [_]
{:exit 0
:out ""
:err ""})
{:jobs 2
:on-case-success (fn [payload]
(swap! events conj [(:id (:result payload)) (:elapsed-ms payload)]))})
(let [elapsed-map (into {} @events)]
(is (< (get elapsed-map "fast-after-queue" 1000) 120)
"queued case should measure only its own execution time, not time spent waiting in the pool")
(is (>= (get elapsed-map "slow-1" 0) 150))
(is (>= (get elapsed-map "slow-2" 0) 150)))))
(deftest select-cases-supports-case-id
(is (= ["graph-create"]
(mapv :id (main/select-cases sample-cases {:case "graph-create"})))))
(deftest select-cases-supports-include-tags
(is (= ["global-help" "graph-list"]
(mapv :id (main/select-cases sample-cases {:include ["smoke"]})))))
(deftest select-cases-omits-skipped-cases
(let [cases (conj sample-cases
{:id "stale-fallback"
:cmds ["node static/logseq-cli.js debug pull --id 1"]
:skip "Requires a removed current graph fallback"
:covers {:commands ["debug pull"]}
:tags [:debug :smoke]})]
(is (= ["global-help" "graph-create" "graph-list"]
(mapv :id (main/select-cases cases {}))))
(is (= ["global-help" "graph-list"]
(mapv :id (main/select-cases cases {:include ["smoke"]}))))
(is (= []
(mapv :id (main/select-cases cases {:case "stale-fallback"}))))))
(deftest run-fails-on-missing-coverage-before-case-execution
(let [ran? (atom false)]
(is (thrown-with-msg?
clojure.lang.ExceptionInfo
#"Missing coverage"
(main/run! {:inventory complete-inventory
:cases [{:id "global-help"
:cmds ["node static/logseq-cli.js --help"]
:covers {:options {:global ["--help"]}}}]
:skip-build true
:run-command (fn [_]
(reset! ran? true)
{:exit 0
:out ""
:err ""})})))
(is (false? @ran?))))
(deftest run-succeeds-when-coverage-is-complete
(let [result (main/run! {:inventory complete-inventory
:cases [{:id "global-help"
:cmds ["node static/logseq-cli.js --help"]
:covers {:options {:global ["--help"]}}}
{:id "graph-create"
:cmds ["node static/logseq-cli.js graph create --type markdown"]
:covers {:commands ["graph create"]
:options {:graph ["--type"]}}}
{:id "graph-list"
:cmds ["node static/logseq-cli.js graph list --file demo.edn"]
:covers {:commands ["graph list"]
:options {:graph ["--file"]}}}]
:skip-build true
:run-command (fn [_]
{:exit 0
:out ""
:err ""})})]
(is (= :ok (:status result)))
(is (= 3 (count (:cases result))))))
(deftest targeted-run-skips-full-coverage-check
(let [executed (atom [])]
(let [result (main/run! {:inventory complete-inventory
:cases [{:id "global-help"
:cmds ["node static/logseq-cli.js --help"]
:covers {:options {:global ["--help"]}}}
{:id "graph-create"
:cmds ["node static/logseq-cli.js graph create --graph demo"]
:covers {:commands ["graph create"]
:options {:graph ["--type"]}}}]
:case "global-help"
:skip-build true
:run-command (fn [_]
{:exit 0
:out ""
:err ""})
:run-case (fn [case _opts]
(swap! executed conj (:id case))
{:id (:id case)
:status :ok})})]
(is (= :ok (:status result)))
(is (= ["global-help"] @executed)))))
(deftest run-invokes-case-progress-hooks
(let [events (atom [])]
(main/run! {:inventory complete-inventory
:cases sample-cases
:include ["smoke"]
:skip-build true
:run-command (fn [_]
{:exit 0
:out ""
:err ""})
:run-case (fn [case _opts]
{:id (:id case)
:status :ok})
:on-case-start (fn [{:keys [index total case]}]
(swap! events conj [:start index total (:id case)]))
:on-case-success (fn [{:keys [index total result]}]
(swap! events conj [:ok index total (:id result)]))})
(is (= [[:start 1 2 "global-help"]
[:ok 1 2 "global-help"]
[:start 2 2 "graph-list"]
[:ok 2 2 "graph-list"]]
@events))))
(deftest run-loads-non-sync-suite-by-default
(let [suite-calls (atom [])]
(with-redefs [manifests/load-inventory (fn [suite]
(swap! suite-calls conj [:inventory suite])
complete-inventory)
manifests/load-cases (fn [suite]
(swap! suite-calls conj [:cases suite])
sample-cases)]
(let [result (main/run! {:skip-build true
:run-command (fn [_]
{:exit 0
:out ""
:err ""})
:run-case (fn [case _opts]
{:id (:id case)
:status :ok})})]
(is (= :ok (:status result))))
(is (= [[:inventory :non-sync]
[:cases :non-sync]]
@suite-calls)))))
(deftest run-loads-sync-suite-when-explicit
(let [suite-calls (atom [])
sync-inventory {:excluded-command-prefixes ["login" "logout"]
:scopes {:sync {:commands ["sync upload"]
:options []}}}
sync-cases [{:id "sync-upload"
:cmds ["node static/logseq-cli.js sync upload"]
:covers {:commands ["sync upload"]}}]]
(with-redefs [manifests/load-inventory (fn [suite]
(swap! suite-calls conj [:inventory suite])
sync-inventory)
manifests/load-cases (fn [suite]
(swap! suite-calls conj [:cases suite])
sync-cases)]
(let [result (main/run! {:suite :sync
:skip-build true
:run-command (fn [_]
{:exit 0
:out ""
:err ""})
:run-case (fn [case _opts]
{:id (:id case)
:status :ok})})]
(is (= :ok (:status result))))
(is (= [[:inventory :sync]
[:cases :sync]]
@suite-calls)))))
(deftest run-sync-suite-uses-suite-fixture-once-with-parallel-runner
(let [before-called (atom 0)
after-called (atom 0)
prepared-case-ids (atom [])
run-case-seen (atom [])
parallel-call (atom nil)
sync-inventory {:excluded-command-prefixes ["login" "logout"]
:scopes {:sync {:commands ["sync upload" "sync status"]
:options []}}}
sync-cases [{:id "sync-upload"
:cmds ["node static/logseq-cli.js sync upload"]
:covers {:commands ["sync upload"]}}
{:id "sync-status"
:cmds ["node static/logseq-cli.js sync status"]
:covers {:commands ["sync status"]}}]]
(with-redefs [sync-fixture/before-suite! (fn [_]
(swap! before-called inc)
{:suite :sync})
sync-fixture/prepare-case (fn [case _suite-context]
(swap! prepared-case-ids conj (:id case))
(assoc case :prepared? true))
sync-fixture/after-suite! (fn [_ _]
(swap! after-called inc))
main/run-selected-cases-in-parallel! (fn [selected-cases run-case run-command opts]
(reset! parallel-call {:case-ids (mapv :id selected-cases)
:jobs (:jobs opts)})
(mapv (fn [case]
(run-case case {:run-command run-command}))
selected-cases))]
(let [result (main/run! {:suite :sync
:inventory sync-inventory
:cases sync-cases
:skip-build true
:jobs 2
:run-command (fn [_]
{:exit 0
:out ""
:err ""})
:run-case (fn [case _opts]
(swap! run-case-seen conj [(:id case) (:prepared? case)])
{:id (:id case)
:status :ok})})]
(is (= :ok (:status result)))))
(is (= 1 @before-called))
(is (= 1 @after-called))
(is (= ["sync-upload" "sync-status"] (:case-ids @parallel-call)))
(is (= 2 (:jobs @parallel-call)))
(is (= ["sync-upload" "sync-status"] @prepared-case-ids))
(is (= [["sync-upload" true]
["sync-status" true]]
@run-case-seen))))
(deftest list-cases-defaults-to-non-sync
(let [selected-suite (atom nil)
output (with-out-str
(with-redefs [manifests/load-cases (fn [suite]
(reset! selected-suite suite)
[{:id "non-sync-case"}])]
(main/list-cases! {})))]
(is (= :non-sync @selected-suite))
(is (string/includes? output "non-sync-case"))))
(deftest list-sync-cases-uses-sync-suite
(let [selected-suite (atom nil)
output (with-out-str
(with-redefs [manifests/load-cases (fn [suite]
(reset! selected-suite suite)
[{:id "sync-case"}])]
(main/list-sync-cases! {})))]
(is (= :sync @selected-suite))
(is (string/includes? output "sync-case"))))
(deftest test-prints-progress-and-summary
(let [output (with-out-str
(main/test! {:inventory complete-inventory
:cases sample-cases
:include ["smoke"]
:skip-build true
:run-command (fn [_]
{:exit 0
:out ""
:err ""})
:run-case (fn [case _opts]
{:id (:id case)
:status :ok
:cmd (last (:cmds case))})}))]
(is (string/includes? output "==> Running cli-e2e cases"))
(is (string/includes? output "==> Build preflight: running..."))
(is (string/includes? output "==> Build preflight: skipped (--skip-build)"))
(is (string/includes? output "==> Prepared 2 case(s), starting execution"))
(is (string/includes? output "[1/2] ▶ global-help"))
(is (string/includes? output "[1/2] ✓ global-help"))
(is (string/includes? output "[2/2] ▶ graph-list"))
(is (string/includes? output "[2/2] ✓ graph-list"))
(is (string/includes? output "Summary: 2 passed, 0 failed"))))
(deftest test-parallel-output-omits-meaningless-index-prefixes
(let [output (with-out-str
(main/test! {:inventory complete-inventory
:cases sample-cases
:include ["smoke"]
:skip-build true
:jobs 2
:run-command (fn [_]
{:exit 0
:out ""
:err ""})
:run-case (fn [case _opts]
(Thread/sleep (if (= "global-help" (:id case)) 25 5))
{:id (:id case)
:status :ok})}))]
(is (string/includes? output "▶ global-help"))
(is (string/includes? output "✓ global-help"))
(is (string/includes? output "✓ graph-list"))
(is (not (string/includes? output "[1/2]")))
(is (not (string/includes? output "[2/2]")))))
(deftest test-timings-prints-step-details-and-slow-summary
(let [output (with-out-str
(main/test! {:inventory complete-inventory
:cases sample-cases
:include ["smoke"]
:skip-build true
:timings true
:run-command (fn [_]
{:exit 0
:out ""
:err ""})
:run-case (fn [case _opts]
(if (= "global-help" (:id case))
{:id "global-help"
:status :ok
:timings [{:phase :setup
:step-index 1
:step-total 1
:elapsed-ms 12
:status :ok
:cmd "setup-global"}
{:phase :main
:step-index 1
:step-total 1
:elapsed-ms 55
:status :ok
:cmd "main-global"}]}
{:id "graph-list"
:status :ok
:timings [{:phase :main
:step-index 1
:step-total 1
:elapsed-ms 210
:status :ok
:cmd "main-graph-list"}]}))}))]
(is (string/includes? output "==> Step timing enabled (--timings)"))
(is (string/includes? output "step timings:"))
(is (string/includes? output "Slow steps (top 10):"))
(is (string/includes? output "main-graph-list"))))
(deftest test-without-timings-keeps-output-concise
(let [output (with-out-str
(main/test! {:inventory complete-inventory
:cases sample-cases
:include ["smoke"]
:skip-build true
:run-command (fn [_]
{:exit 0
:out ""
:err ""})
:run-case (fn [case _opts]
{:id (:id case)
:status :ok
:timings [{:phase :main
:step-index 1
:step-total 1
:elapsed-ms 88
:status :ok
:cmd "hidden-step"}]})}))]
(is (not (string/includes? output "==> Step timing enabled (--timings)")))
(is (not (string/includes? output "step timings:")))
(is (not (string/includes? output "Slow steps (top 10):")))))
(deftest test-sync-timings-prints-step-details-and-slow-summary
(let [sync-inventory {:excluded-command-prefixes ["login" "logout"]
:scopes {:sync {:commands ["sync status"]
:options []}}}
sync-cases [{:id "sync-status-case"
:cmds ["node static/logseq-cli.js sync status"]
:covers {:commands ["sync status"]}}]
output (with-out-str
(main/test-sync! {:inventory sync-inventory
:cases sync-cases
:skip-build true
:timings true
:run-command (fn [_]
{:exit 0
:out ""
:err ""})
:run-case (fn [case _opts]
{:id (:id case)
:status :ok
:timings [{:phase :setup
:step-index 1
:step-total 1
:elapsed-ms 10
:status :ok
:cmd "sync-setup"}
{:phase :main
:step-index 1
:step-total 1
:elapsed-ms 150
:status :ok
:cmd "sync-main"}
{:phase :cleanup
:step-index 1
:step-total 1
:elapsed-ms 20
:status :ok
:cmd "sync-cleanup"}]})}))]
(is (string/includes? output "==> Running cli-e2e cases"))
(is (string/includes? output "==> Step timing enabled (--timings)"))
(is (string/includes? output "step timings:"))
(is (string/includes? output "Slow steps (top 10):"))
(is (string/includes? output "sync-main"))))
(deftest test-help-prints-usage-and-skips-execution
(let [ran? (atom false)
result (atom nil)
output (with-out-str
(reset! result
(main/test! {:help true
:run-command (fn [_]
(reset! ran? true)
{:exit 0
:out ""
:err ""})
:run-case (fn [_ _]
(reset! ran? true)
{:id "unexpected"
:status :ok})}))) ]
(is (= :help (:status @result)))
(is (false? @ran?))
(is (string/includes? output "Usage: bb -f cli-e2e/bb.edn test [options]"))
(is (string/includes? output "--skip-build"))
(is (not (string/includes? output "--force-build")))
(is (string/includes? output "--include TAG"))
(is (string/includes? output "--case ID"))
(is (string/includes? output "--jobs N"))
(is (string/includes? output "Run up to N non-sync cases in parallel"))
(is (string/includes? output "--skip-build --jobs 4"))
(is (string/includes? output "Default: 4"))
(is (string/includes? output "--timings"))
(is (not (string/includes? output "--e2ee-password")))))
(deftest test-sync-help-prints-usage-and-skips-execution
(let [ran? (atom false)
result (atom nil)
output (with-out-str
(reset! result
(main/test-sync! {:help true
:run-command (fn [_]
(reset! ran? true)
{:exit 0
:out ""
:err ""})
:run-case (fn [_ _]
(reset! ran? true)
{:id "unexpected"
:status :ok})})))]
(is (= :help (:status @result)))
(is (false? @ran?))
(is (string/includes? output "Usage: bb -f cli-e2e/bb.edn test-sync [options]"))
(is (string/includes? output "--skip-build"))
(is (not (string/includes? output "--force-build")))
(is (string/includes? output "--include TAG"))
(is (string/includes? output "--case ID"))
(is (string/includes? output "--jobs N"))
(is (string/includes? output "Run up to N sync cases in parallel"))
(is (string/includes? output "--jobs 4"))
(is (string/includes? output "bb -f cli-e2e/bb.edn test-sync"))
(is (string/includes? output "--case sync-upload-download-mvp"))
(is (not (string/includes? output "--skip-build --case sync-upload-download-mvp")))
(is (string/includes? output "Default: 4"))
(is (string/includes? output "--timings"))
(is (string/includes? output "--e2ee-password VALUE"))
(is (string/includes? output "Default: 11111"))))
(deftest run-does-not-pass-force-build-to-preflight
(let [preflight-call (atom nil)]
(with-redefs [logseq.cli.e2e.preflight/run! (fn [opts]
(reset! preflight-call opts)
{:status :ok
:commands []
:missing-artifacts []})]
(let [result (main/run! {:inventory complete-inventory
:cases sample-cases
:include ["smoke"]
:skip-build false
:force-build true
:run-command (fn [_]
{:exit 0
:out ""
:err ""})
:run-case (fn [case _opts]
{:id (:id case)
:status :ok})})]
(is (= :ok (:status result)))
(is (not (contains? @preflight-call :force-build)))))))
(deftest test-single-case-enables-detailed-command-logging
(let [command-opts (atom nil)
output (with-out-str
(main/test! {:inventory complete-inventory
:cases [{:id "global-help"
:cmds ["node static/logseq-cli.js --help"]
:covers {:options {:global ["--help"]}}}]
:case "global-help"
:skip-build true
:run-command (fn [{:keys [cmd] :as opts}]
(reset! command-opts opts)
{:cmd cmd
:exit 0
:out "ok"
:err ""})
:run-case (fn [case {:keys [run-command]}]
(let [result (run-command {:cmd (first (:cmds case))
:phase :main
:step-index 1
:step-total 1})]
{:id (:id case)
:status :ok
:cmd (:cmd result)}) )}))]
(is (string/includes? output "==> Detailed case logging enabled (--case global-help)"))
(is (string/includes? output " [main 1/1] $ node static/logseq-cli.js --help"))
(is (true? (:stream-output? @command-opts)))))
(deftest cleanup-help-prints-usage
(let [output (with-out-str (main/cleanup! {:help true}))]
(is (string/includes? output "Usage: bb -f cli-e2e/bb.edn cleanup"))
(is (string/includes? output "Terminate cli-e2e db-worker-node processes"))
(is (string/includes? output "Terminate db-sync server listeners on port 18080"))
(is (string/includes? output "Remove cli-e2e temp roots"))
(is (string/includes? output "--dry-run"))))
(deftest cleanup-prints-summary-and-returns-status
(with-redefs [cleanup/cleanup-db-worker-processes! (fn [_] {:found-pids [101 202]
:killed-pids [101]
:failed-pids [202]})
cleanup/cleanup-db-sync-port-processes! (fn [_] {:found-pids [303]
:killed-pids [303]
:failed-pids []})
cleanup/cleanup-temp-roots! (fn [_] {:found-dirs ["/tmp/logseq-cli-e2e-a"
"/tmp/logseq-cli-e2e-b"]
:removed-dirs ["/tmp/logseq-cli-e2e-a"]
:failed-dirs ["/tmp/logseq-cli-e2e-b"]})]
(let [result (atom nil)
output (with-out-str
(reset! result (main/cleanup! {})))]
(is (= :ok (:status @result)))
(is (= [101] (get-in @result [:processes :killed-pids])))
(is (= [303] (get-in @result [:db-sync-port-processes :killed-pids])))
(is (= ["/tmp/logseq-cli-e2e-a"] (get-in @result [:temp-roots :removed-dirs])))
(is (string/includes? output "db-worker-node processes: found 2, killed 1, failed 1"))
(is (string/includes? output "db-sync server processes (port 18080): found 1, killed 1, failed 0"))
(is (string/includes? output "temp roots: found 2, removed 1, failed 1")))))
(deftest cleanup-dry-run-prints-summary-and-passes-option
(let [process-opts (atom nil)
db-sync-opts (atom nil)
dir-opts (atom nil)]
(with-redefs [cleanup/cleanup-db-worker-processes! (fn [opts]
(reset! process-opts opts)
{:dry-run? true
:found-pids [101 202]
:would-kill-pids [101 202]
:killed-pids []
:failed-pids []})
cleanup/cleanup-db-sync-port-processes! (fn [opts]
(reset! db-sync-opts opts)
{:dry-run? true
:found-pids [303]
:would-kill-pids [303]
:killed-pids []
:failed-pids []})
cleanup/cleanup-temp-roots! (fn [opts]
(reset! dir-opts opts)
{:dry-run? true
:found-dirs ["/tmp/logseq-cli-e2e-a"]
:would-remove-dirs ["/tmp/logseq-cli-e2e-a"]
:removed-dirs []
:failed-dirs []})]
(let [result (atom nil)
output (with-out-str
(reset! result (main/cleanup! {:dry-run true})))]
(is (= {:dry-run true} @process-opts))
(is (= {:dry-run true} @db-sync-opts))
(is (= {:dry-run true} @dir-opts))
(is (= :ok (:status @result)))
(is (true? (get-in @result [:processes :dry-run?])))
(is (true? (get-in @result [:db-sync-port-processes :dry-run?])))
(is (true? (get-in @result [:temp-roots :dry-run?])))
(is (string/includes? output "[dry-run] db-worker-node processes: found 2, would kill 2"))
(is (string/includes? output "[dry-run] db-sync server processes (port 18080): found 1, would kill 1"))
(is (string/includes? output "[dry-run] temp roots: found 1, would remove 1"))))))

View File

@@ -0,0 +1,225 @@
(ns logseq.cli.e2e.manifests-test
(:require [clojure.test :refer [deftest is testing]]
[logseq.cli.e2e.manifests :as manifests]))
(deftest load-cases-requires-new-map-format
(with-redefs [manifests/read-edn-file (fn [_]
[{:id "legacy-a"}
{:id "legacy-b"}])]
(let [error (try
(manifests/load-cases :non-sync)
nil
(catch clojure.lang.ExceptionInfo error
error))]
(is (some? error))
(is (re-find #"Invalid cli-e2e manifest format" (.getMessage error)))
(is (= "{:templates {...} :cases [...]}"
(:expected (ex-data error)))))))
(deftest load-cases-merges-templates-and-cases-by-rules
(with-redefs [manifests/read-edn-file
(fn [_]
{:templates
{:base {:setup ["setup-a"]
:cmds ["cmd-a"]
:cleanup ["cleanup-a"]
:tags [:base]
:vars {:nested {:left 1}
:only-base true}
:covers {:commands ["base-command"]
:options {:global ["--base"]}}
:expect {:stdout-json-paths {[:status] "ok"
[:data :base] 1}}
:graph "base-graph"}
:addon {:setup ["setup-b"]
:cmds ["cmd-b"]
:cleanup ["cleanup-b"]
:tags [:addon]
:vars {:nested {:right 2}}
:covers {:options {:graph ["--addon"]}}
:expect {:stdout-json-paths {[:data :addon] 2}}
:graph "addon-graph"}
:shared {:setup ["setup-shared"]
:cmds ["cmd-shared"]
:cleanup ["cleanup-shared"]
:tags [:shared]
:vars {:nested {:shared 9}}
:graph "shared-graph"}}
:cases
[{:id "single-parent"
:extends :base
:cmds ["cmd-case"]
:expect {:stdout-json-paths {[:data :case] 3}}
:graph "single-parent-graph"}
{:id "multi-parent"
:extends [:base :addon]
:setup ["setup-case"]
:cmds ["cmd-case"]
:cleanup ["cleanup-case"]
:tags [:case]
:vars {:nested {:leaf 3}
:only-case true}
:covers {:commands ["case-command"]
:options {:graph ["--case"]}}
:expect {:stdout-json-paths {[:data :case] 3}}
:graph "case-graph"}
{:id "negative-case"
:extends :shared
:expect {:exit 1
:stdout-json-paths {[:error :code] "boom"}}
:graph "negative-graph"}]})]
(let [[single-parent multi-parent negative-case] (manifests/load-cases :sync)]
(testing "supports :extends keyword"
(is (= "single-parent" (:id single-parent)))
(is (= ["cmd-a" "cmd-case"] (:cmds single-parent)))
(is (= "single-parent-graph" (:graph single-parent))))
(testing "append merge keys"
(is (= ["setup-a" "setup-b" "setup-case"] (:setup multi-parent)))
(is (= ["cmd-a" "cmd-b" "cmd-case"] (:cmds multi-parent)))
(is (= ["cleanup-a" "cleanup-b" "cleanup-case"] (:cleanup multi-parent)))
(is (= [:base :addon :case] (:tags multi-parent))))
(testing "deep merge keys"
(is (= {:nested {:left 1 :right 2 :leaf 3}
:only-base true
:only-case true}
(:vars multi-parent)))
(is (= {:commands ["case-command"]
:options {:global ["--base"]
:graph ["--case"]}}
(:covers multi-parent)))
(is (= {[:status] "ok"
[:data :base] 1
[:data :addon] 2
[:data :case] 3}
(get-in multi-parent [:expect :stdout-json-paths]))))
(testing "child expect overrides inherited maps when parent omits them"
(is (= {:exit 1
:stdout-json-paths {[:error :code] "boom"}}
(:expect negative-case))))
(testing "base templates can omit happy-path defaults"
(is (= ["setup-shared"] (:setup negative-case)))
(is (= ["cmd-shared"] (:cmds negative-case)))
(is (= ["cleanup-shared"] (:cleanup negative-case))))
(testing "scalar keys are overridden by child"
(is (= "case-graph" (:graph multi-parent)))))))
(deftest load-cases-detects-circular-template-inheritance-with-cycle-path
(with-redefs [manifests/read-edn-file (fn [_]
{:templates
{:a {:extends :b
:setup ["a"]}
:b {:extends :a
:setup ["b"]}}
:cases [{:id "cycle" :extends :a}]})]
(let [error (try
(manifests/load-cases :sync)
nil
(catch clojure.lang.ExceptionInfo error
error))]
(is (some? error))
(is (re-find #"Circular template inheritance" (.getMessage error)))
(is (= [:a :b :a] (:cycle (ex-data error)))))))
(deftest load-cases-validates-extends-entries
(with-redefs [manifests/read-edn-file (fn [_]
{:templates {:base {:cmds ["base"]}}
:cases [{:id "invalid-extends"
:extends [:base "bad"]
:cmds ["case"]}]})]
(let [error (try
(manifests/load-cases :non-sync)
nil
(catch clojure.lang.ExceptionInfo error
error))]
(is (some? error))
(is (re-find #"Invalid :extends entries" (.getMessage error)))
(is (= ["bad"] (:invalid-entries (ex-data error)))))))
(deftest load-cases-lint-detects-invalid-extends-references
(with-redefs [manifests/read-edn-file (fn [_]
{:templates {:base {:cmds ["base"]}
:unused {:extends :missing-template
:cmds ["unused"]}}
:cases [{:id "valid" :extends :base :cmds ["case"]}
{:id "invalid" :extends :also-missing :cmds ["case"]}]})]
(let [error (try
(manifests/load-cases :sync)
nil
(catch clojure.lang.ExceptionInfo error
error))
issues (:issues (ex-data error))]
(is (some? error))
(is (re-find #"manifest lint failed" (.getMessage error)))
(is (= #{:also-missing :missing-template}
(set (map :target (filter #(= :invalid-extends (:type %)) issues))))))))
(deftest load-cases-lint-detects-duplicate-case-ids
(with-redefs [manifests/read-edn-file (fn [_]
{:templates {:base {:cmds ["base"]}}
:cases [{:id "dup" :extends :base :cmds ["case-a"]}
{:id "dup" :extends :base :cmds ["case-b"]}]})]
(let [error (try
(manifests/load-cases :non-sync)
nil
(catch clojure.lang.ExceptionInfo error
error))
duplicate-issues (filter #(= :duplicate-case-id (:type %))
(:issues (ex-data error)))]
(is (some? error))
(is (= [{:type :duplicate-case-id :id "dup" :count 2}]
(vec duplicate-issues))))))
(deftest load-cases-lint-detects-unused-templates
(with-redefs [manifests/read-edn-file (fn [_]
{:templates {:base {:cmds ["base"]}
:unused {:cmds ["unused"]}}
:cases [{:id "only" :extends :base :cmds ["case"]}]})]
(let [error (try
(manifests/load-cases :sync)
nil
(catch clojure.lang.ExceptionInfo error
error))
unused-issues (filter #(= :unused-template (:type %))
(:issues (ex-data error)))]
(is (some? error))
(is (= [{:type :unused-template :template :unused}]
(vec unused-issues))))))
(deftest sync-multi-batch-operations-uses-state-driven-waits-instead-of-fixed-sleeps
(let [cases (manifests/load-cases :sync)
multi-batch (some #(when (= "sync-multi-batch-operations" (:id %)) %) cases)
commands (:cmds multi-batch)
upload-commands (filter #(re-find #"sync upload --graph" %) commands)
wait-commands (filter #(re-find #"wait_sync_status\.py" %) commands)]
(is (some? multi-batch))
(is (not-any? #(= "sleep 1" %) commands))
(is (= 1 (count upload-commands)))
(is (= 5 (count wait-commands)))
(is (= 3 (count (filter #(re-find #"--root-dir '\{\{tmp-dir\}\}/graphs-b'.+--timeout-s 30 --interval-s 1" %) wait-commands))))))
(deftest sync-manifest-includes-duplicate-upload-negative-case
(let [cases (manifests/load-cases :sync)
duplicate-upload (some #(when (= "sync-upload-rejects-duplicate-remote-graph" (:id %)) %) cases)
json-paths (get-in duplicate-upload [:expect :stdout-json-paths])
setup-commands (:setup duplicate-upload)
main-commands (:cmds duplicate-upload)]
(is (some? duplicate-upload))
(is (= 1 (get-in duplicate-upload [:expect :exit])))
(is (= "graph-already-exists"
(get-in duplicate-upload [:expect :stdout-json-paths [:error :code]])))
(is (= ["delete it before uploading again"]
(get-in duplicate-upload [:expect :stdout-contains])))
(is (= 1 (count (filter #(re-find #"sync upload --graph" %) setup-commands))))
(is (= 1 (count (filter #(re-find #"sync upload --graph" %) main-commands))))
(is (false? (contains? json-paths [:data :pending-local])))
(is (false? (contains? json-paths [:data :pending-asset])))
(is (false? (contains? json-paths [:data :pending-server])))
(is (false? (contains? json-paths [:data :last-error])))))
(deftest sync-status-steady-state-does-not-repeat-identical-b-side-steady-state-waits
(let [cases (manifests/load-cases :sync)
steady-state (some #(when (= "sync-status-steady-state" (:id %)) %) cases)
steady-waits (filter #(re-find #"wait_sync_status\.py.+--root-dir '\{\{tmp-dir\}\}/graphs-b'.+--timeout-s 30 --interval-s 1" %) (:cmds steady-state))]
(is (some? steady-state))
(is (= 1 (count steady-waits)))
(is (= 1 (count (filter #(re-find #"sync status --graph" %) (:cmds steady-state)))))))

View File

@@ -0,0 +1,18 @@
(ns logseq.cli.e2e.paths-test
(:require [clojure.test :refer [deftest is]]
[logseq.cli.e2e.paths :as paths]))
(deftest repo-root-does-not-depend-on-runtime-file-binding
(let [expected (paths/repo-root)]
(binding [*file* nil]
(is (= expected
(paths/repo-root))))))
(deftest required-artifacts-include-cli-runtime-contract
(is (= [(paths/repo-path "cli" "_build" "default" "dist" "logseq-cli.js")
(paths/repo-path "static" "logseq-cli.js")
(paths/repo-path "static" "db-worker-node.js")
(paths/repo-path "dist" "db-worker-node.js")
(paths/repo-path "dist" "db-worker-node-assets.json")
(paths/repo-path "deps" "db-sync" "worker" "dist" "node-adapter.js")]
(paths/required-artifacts))))

View File

@@ -0,0 +1,98 @@
(ns logseq.cli.e2e.preflight-test
(:require [clojure.test :refer [deftest is testing]]
[logseq.cli.e2e.preflight :as preflight]))
(def required-artifacts
["/repo/cli/_build/default/dist/logseq-cli.js"
"/repo/static/logseq-cli.js"
"/repo/static/db-worker-node.js"
"/repo/dist/db-worker-node.js"
"/repo/dist/db-worker-node-assets.json"
"/repo/deps/db-sync/worker/dist/node-adapter.js"])
(defn- with-required-artifacts
[f]
(with-redefs [logseq.cli.e2e.paths/repo-root (constantly "/repo")
logseq.cli.e2e.paths/required-artifacts (fn [] required-artifacts)]
(f)))
(deftest build-plan-matches-required-commands
(is (= ["pnpm cli:release"
"pnpm db-worker-node:compile:bundle"
"pnpm --dir deps/db-sync build:node-adapter"]
(mapv :cmd preflight/build-plan)))
(is (not-any? #(re-find #"clojure -M:cljs compile logseq-cli" (:cmd %))
preflight/build-plan)))
(deftest missing-artifacts-returns-unreadable-paths
(let [artifacts ["/repo/static/logseq-cli.js"
"/repo/static/db-worker-node.js"
"/repo/dist/db-worker-node.js"]
present? #{"/repo/static/logseq-cli.js"}]
(is (= ["/repo/static/db-worker-node.js"
"/repo/dist/db-worker-node.js"]
(preflight/missing-artifacts artifacts present?)))))
(deftest skip-build-avoids-running-shell-commands
(let [called? (atom false)
result (preflight/run! {:skip-build true
:run-command (fn [_]
(reset! called? true))
:file-exists? (constantly false)})]
(is (= :skipped (:status result)))
(is (false? @called?))))
(deftest build-runs-commands-even-when-artifacts-are-ready
(let [calls (atom [])]
(with-required-artifacts
#(let [result (preflight/run! {:run-command (fn [{:keys [cmd]}]
(swap! calls conj cmd)
{:cmd cmd
:exit 0
:out ""
:err ""})
:file-exists? (set required-artifacts)})]
(is (= :ok (:status result)))
(is (= ["pnpm cli:release"
"pnpm db-worker-node:compile:bundle"
"pnpm --dir deps/db-sync build:node-adapter"]
@calls))))))
(deftest build-runs-commands-when-artifacts-are-partially-present
(let [calls (atom [])
existing (atom (disj (set required-artifacts)
"/repo/dist/db-worker-node-assets.json"))]
(with-required-artifacts
#(let [result (preflight/run! {:run-command (fn [{:keys [cmd]}]
(swap! calls conj cmd)
(reset! existing (set required-artifacts))
{:cmd cmd
:exit 0
:out ""
:err ""})
:file-exists? (fn [path]
(contains? @existing path))})]
(is (= :ok (:status result)))
(is (= ["pnpm cli:release"
"pnpm db-worker-node:compile:bundle"
"pnpm --dir deps/db-sync build:node-adapter"]
@calls))))))
(deftest build-runs-commands-when-artifacts-are-absent
(let [calls (atom [])
existing (atom #{})]
(with-required-artifacts
#(let [result (preflight/run! {:run-command (fn [{:keys [cmd]}]
(swap! calls conj cmd)
(reset! existing (set required-artifacts))
{:cmd cmd
:exit 0
:out ""
:err ""})
:file-exists? (fn [path]
(contains? @existing path))})]
(is (= :ok (:status result)))
(is (= ["pnpm cli:release"
"pnpm db-worker-node:compile:bundle"
"pnpm --dir deps/db-sync build:node-adapter"]
@calls))))))

View File

@@ -0,0 +1,102 @@
from __future__ import annotations
import importlib.util
from pathlib import Path
import sys
MODULE_PATH = Path(__file__).resolve().parents[4] / "scripts" / "random_bidirectional_block_ops.py"
spec = importlib.util.spec_from_file_location("random_bidirectional_block_ops", MODULE_PATH)
random_bidirectional_block_ops = importlib.util.module_from_spec(spec)
assert spec.loader is not None
sys.modules[spec.name] = random_bidirectional_block_ops
spec.loader.exec_module(random_bidirectional_block_ops)
def test_default_profile_is_faster_than_high_stress(monkeypatch) -> None:
monkeypatch.setattr(
random_bidirectional_block_ops.sys,
"argv",
[
"random_bidirectional_block_ops.py",
"--cli",
"/tmp/logseq-cli.js",
"--graph",
"demo",
"--config-a",
"/tmp/a.edn",
"--root-dir-a",
"/tmp/a",
"--config-b",
"/tmp/b.edn",
"--root-dir-b",
"/tmp/b",
"--page",
"Home",
],
)
default_args = random_bidirectional_block_ops.parse_args()
monkeypatch.setattr(
random_bidirectional_block_ops.sys,
"argv",
[
"random_bidirectional_block_ops.py",
"--cli",
"/tmp/logseq-cli.js",
"--graph",
"demo",
"--config-a",
"/tmp/a.edn",
"--root-dir-a",
"/tmp/a",
"--config-b",
"/tmp/b.edn",
"--root-dir-b",
"/tmp/b",
"--page",
"Home",
"--profile",
"high-stress",
],
)
high_stress_args = random_bidirectional_block_ops.parse_args()
assert default_args.root_dir_a == "/tmp/a"
assert default_args.root_dir_b == "/tmp/b"
assert default_args.profile == "default"
assert high_stress_args.profile == "high-stress"
assert default_args.rounds_per_client < high_stress_args.rounds_per_client
def test_explicit_rounds_override_profile_default(monkeypatch) -> None:
monkeypatch.setattr(
random_bidirectional_block_ops.sys,
"argv",
[
"random_bidirectional_block_ops.py",
"--cli",
"/tmp/logseq-cli.js",
"--graph",
"demo",
"--config-a",
"/tmp/a.edn",
"--root-dir-a",
"/tmp/a",
"--config-b",
"/tmp/b.edn",
"--root-dir-b",
"/tmp/b",
"--page",
"Home",
"--profile",
"high-stress",
"--rounds-per-client",
"12",
],
)
args = random_bidirectional_block_ops.parse_args()
assert args.profile == "high-stress"
assert args.rounds_per_client == 12

View File

@@ -0,0 +1,152 @@
(ns logseq.cli.e2e.runner-test
(:require [clojure.test :refer [deftest is]]
[logseq.cli.e2e.runner :as runner]))
(deftest render-case-expands-template-values-recursively
(let [rendered (runner/render-case
{:id "graph-create"
:setup ["{{cli}} --graph {{graph-arg}}"]
:cmds ["{{cli}} graph info --graph {{graph-arg}}"]
:expect {:stdout-json-paths {[:data :graph] "{{graph}}"
[:status] "ok"}}}
{:cli "node /tmp/logseq-cli.js"
:graph "demo"
:graph-arg "'demo'"})]
(is (= ["node /tmp/logseq-cli.js --graph 'demo'"] (:setup rendered)))
(is (= ["node /tmp/logseq-cli.js graph info --graph 'demo'"] (:cmds rendered)))
(is (= "demo" (get-in rendered [:expect :stdout-json-paths [:data :graph]])))))
(deftest run-case-executes-setup-before-main-command
(let [calls (atom [])
result (runner/run-case!
{:id "graph-info"
:setup ["setup one" "setup two"]
:cmds ["main command one" "main command two"]
:expect {:exit 0}}
{:context {}
:run-command (fn [{:keys [cmd]}]
(swap! calls conj cmd)
{:cmd cmd
:exit 0
:out ""
:err ""})})]
(is (= ["setup one" "setup two" "main command one" "main command two"] @calls))
(is (= "graph-info" (:id result)))
(is (= "main command two" (get-in result [:result :cmd])))))
(deftest run-case-includes-command-phase-metadata-when-detailed
(let [calls (atom [])]
(runner/run-case!
{:id "graph-info"
:setup ["setup one" "setup two"]
:cmds ["main command one" "main command two"]
:cleanup ["cleanup one"]
:expect {:exit 0}}
{:context {}
:detailed-log? true
:run-command (fn [{:keys [cmd phase step-index step-total case-id throw?] :as opts}]
(swap! calls conj (select-keys opts [:cmd :phase :step-index :step-total :case-id :throw?]))
{:cmd cmd
:exit 0
:out ""
:err ""})})
(is (= [{:cmd "setup one" :phase :setup :step-index 1 :step-total 2 :case-id "graph-info" :throw? true}
{:cmd "setup two" :phase :setup :step-index 2 :step-total 2 :case-id "graph-info" :throw? true}
{:cmd "main command one" :phase :main :step-index 1 :step-total 2 :case-id "graph-info" :throw? true}
{:cmd "main command two" :phase :main :step-index 2 :step-total 2 :case-id "graph-info" :throw? false}
{:cmd "cleanup one" :phase :cleanup :step-index 1 :step-total 1 :case-id "graph-info" :throw? false}]
@calls))))
(deftest run-case-collects-step-timings-when-enabled
(let [result (runner/run-case!
{:id "graph-info"
:setup ["setup one"]
:cmds ["main command"]
:cleanup ["cleanup one"]
:expect {:exit 0}}
{:context {}
:timings? true
:run-command (fn [{:keys [cmd]}]
{:cmd cmd
:exit 0
:out ""
:err ""})})]
(is (= 3 (count (:timings result))))
(is (= [:setup :main :cleanup]
(mapv :phase (:timings result))))))
(deftest run-case-attaches-timings-to-error-when-enabled
(let [error (try
(runner/run-case!
{:id "graph-info"
:setup ["setup one"]
:cmds ["main command"]
:cleanup ["cleanup one"]
:expect {:exit 0}}
{:context {}
:timings? true
:run-command (fn [{:keys [cmd]}]
(when (= cmd "main command")
(throw (ex-info "boom" {:cmd cmd})))
{:cmd cmd
:exit 0
:out ""
:err ""})})
nil
(catch clojure.lang.ExceptionInfo error
error))
timings (:timings (ex-data error))]
(is (= "graph-info" (:case-id (ex-data error))))
(is (= [:setup :main :cleanup]
(mapv :phase timings)))
(is (= :failed (:status (second timings))))))
(deftest run-case-validates-json-paths-and-nonzero-exit
(let [result (runner/run-case!
{:id "invalid-shell"
:cmds ["node static/logseq-cli.js completion fish"]
:expect {:exit 1
:stdout-json-paths {[:status] "error"
[:error :code] "invalid-options"}}}
{:context {}
:run-command (fn [{:keys [cmd]}]
{:cmd cmd
:exit 1
:out "{\"status\":\"error\",\"error\":{\"code\":\"invalid-options\"}}"
:err ""})})]
(is (= 1 (get-in result [:result :exit])))))
(deftest assert-result-validates-edn-paths
(is (nil? (runner/assert-result!
{:id "graph-list-edn"
:expect {:exit 0
:stdout-edn-paths {[:status] :ok
[:data :graphs] ["demo"]}}}
{:cmd "node cli.js graph list"
:exit 0
:out "{:status :ok, :data {:graphs [\"demo\"]}}"
:err ""}))))
(deftest assert-result-validates-stdout-not-contains
(is (thrown-with-msg?
clojure.lang.ExceptionInfo
#"stdout contained forbidden text"
(runner/assert-result!
{:id "remove-block"
:expect {:exit 0
:stdout-not-contains ["Alpha block"]}}
{:cmd "node cli.js show --page Home"
:exit 0
:out "Home\n- Alpha block"
:err ""}))))
(deftest assert-result-normalizes-json-escaped-windows-paths-for-contains
(is (nil? (runner/assert-result!
{:id "doctor-dev-script-json"
:expect {:exit 0
:stdout-contains ["static/db-worker-node.js"
"C:\\Users\\demo\\tmp\\graph-export.edn"]}}
{:cmd "node cli.js doctor --dev-script"
:exit 0
:out "{\"status\":\"ok\",\"data\":{\"path\":\"C:\\\\Users\\\\demo\\\\tmp\\\\graph-export.edn\",\"message\":\"Found readable file: C:\\\\Users\\\\demo\\\\project\\\\static\\\\db-worker-node.js\"}}"
:err ""}))))

View File

@@ -0,0 +1,54 @@
(ns logseq.cli.e2e.shell-test
(:require [babashka.process :as process]
[clojure.test :refer [deftest is testing]]
[logseq.cli.e2e.shell :as shell]))
(deftest preserves-the-exact-command-string
(let [captured (atom nil)
command "node static/logseq-cli.js --graph 'Graph With Space' --help"
result (shell/run! {:cmd command
:dir "/repo"
:executor (fn [cmd opts]
(reset! captured {:cmd cmd
:opts opts})
{:exit 0
:out "ok"
:err ""})})]
(is (= command (:cmd result)))
(is (= command (:cmd @captured)))
(is (= "/repo" (get-in @captured [:opts :dir])))))
(deftest shell-failures-include-command-context
(let [command "node static/logseq-cli.js completion fish"]
(try
(shell/run! {:cmd command
:executor (fn [_ _]
{:exit 2
:out ""
:err "unsupported shell"})})
(is false "Expected shell/run! to throw")
(catch clojure.lang.ExceptionInfo ex
(is (= command (:cmd (ex-data ex))))
(is (= 2 (:exit (ex-data ex))))
(is (= "unsupported shell" (:err (ex-data ex))))))))
(deftest allow-failure-returns-result-without-throwing
(let [command "node static/logseq-cli.js graph info --graph missing"
result (shell/run! {:cmd command
:throw? false
:executor (fn [_ _]
{:exit 1
:out "failed"
:err "missing"})})]
(is (= command (:cmd result)))
(is (= 1 (:exit result)))
(is (= "failed" (:out result)))
(is (= "missing" (:err result)))))
(deftest default-executor-uses-resolvable-shell-command
(let [captured (atom nil)]
(with-redefs [process/process (fn [_opts & args]
(reset! captured args)
(future {:exit 0}))]
(shell/run! {:cmd "echo ok"}))
(is (= ["bash" "-c" "echo ok"] @captured))))

View File

@@ -0,0 +1,184 @@
(ns logseq.cli.e2e.sync-fixture-test
(:require [babashka.fs :as fs]
[clojure.java.shell :as shell]
[clojure.string :as string]
[clojure.test :refer [deftest is testing]]
[logseq.cli.e2e.sync-fixture :as sync-fixture]))
(defn- free-port []
(with-open [socket (java.net.ServerSocket. 0)]
(.getLocalPort socket)))
(deftest prepare-sync-config-writes-oauth-token-endpoint
(let [tmp-dir (fs/create-temp-dir {:prefix "logseq-cli-sync-config-test-"})
auth-path (fs/path tmp-dir "auth.json")
config-path (fs/path tmp-dir "cli.edn")
script (fs/path "cli-e2e" "scripts" "prepare_sync_config.py")
result (do
(spit (str auth-path) "{\"refresh-token\":\"refresh-token\"}\n")
(shell/sh "python3" (str script)
"--output" (str config-path)
"--auth-path" (str auth-path)
"--http-base" "http://127.0.0.1:18080"
"--ws-url" "ws://127.0.0.1:18080/sync/%s"))]
(try
(is (= 0 (:exit result)) (:err result))
(let [config (slurp (str config-path))]
(is (string/includes? config ":http-base \"http://127.0.0.1:18080\""))
(is (string/includes? config ":ws-url \"ws://127.0.0.1:18080/sync/%s\""))
(is (string/includes? config ":oauth-token-endpoint "))
(is (string/includes? config "https://logseq-prod.auth.us-east-1.amazoncognito.com/oauth2/token")))
(finally
(fs/delete-tree tmp-dir)))))
(deftest prepare-case-injects-case-local-sync-resources
(let [suite-context {:sync-port "18080"
:sync-http-base "http://127.0.0.1:18080"
:sync-ws-url "ws://127.0.0.1:18080/sync/%s"}
input-case {:id "sync-case"
:vars {:existing true}
:setup ["mkdir -p '{{tmp-dir}}/graphs-b'"
"mkdir -p '{{tmp-dir}}/home/logseq'"
"cp ~/logseq/auth.json '{{tmp-dir}}/home/logseq/auth.json'"
"python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{config-path}}' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'"
"python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{tmp-dir}}/cli-b.edn' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'"
"python3 '{{repo-root}}/cli-e2e/scripts/db_sync_server.py' start --port 18080"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null"]
:cleanup ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{graph-arg}}"
"python3 '{{repo-root}}/cli-e2e/scripts/db_sync_server.py' stop --pid-file '{{tmp-dir}}/db-sync-server.pid'"]}
prepared (sync-fixture/prepare-case input-case suite-context)]
(is (= "sync-case" (:id prepared)))
(is (= ["mkdir -p '{{tmp-dir}}/graphs-b'"
"mkdir -p '{{tmp-dir}}/home/logseq'"
"cp ~/logseq/auth.json '{{tmp-dir}}/home/logseq/auth.json'"
"python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{config-path}}' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'"
"python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{tmp-dir}}/cli-b.edn' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null"]
(:setup prepared)))
(is (= ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{graph-arg}}"]
(:cleanup prepared)))
(is (= "http://127.0.0.1:18080" (get-in prepared [:vars :sync-http-base])))
(is (= "ws://127.0.0.1:18080/sync/%s" (get-in prepared [:vars :sync-ws-url])))
(is (= "18080" (get-in prepared [:vars :sync-port])))
(is (= "11111" (get-in prepared [:vars :e2ee-password])))
(is (= "11111" (get-in prepared [:vars :e2ee-password-arg])))
(is (not (contains? (:vars prepared) :suite-auth-path)))
(is (not (contains? (:vars prepared) :suite-config-path)))
(is (not-any? #(string/includes? % "suite-auth-path") (:setup prepared)))
(is (not-any? #(string/includes? % "suite-config-path") (:setup prepared)))
(is (not-any? #(string/includes? % "db_sync_server.py' start") (:setup prepared)))
(is (not-any? #(string/includes? % "db_sync_server.py' stop") (:cleanup prepared)))))
(deftest prepare-case-places-case-local-auth-before-cli-sync-commands
(let [suite-context {:suite-tmp-dir "/tmp/sync-suite"
:sync-port "18080"
:sync-http-base "http://127.0.0.1:18080"
:sync-ws-url "ws://127.0.0.1:18080/sync/%s"}
prepared (sync-fixture/prepare-case
{:id "sync-case"
:setup ["mkdir -p '{{tmp-dir}}/graphs-b'"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null"
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync ensure-keys --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"]
:cleanup []}
suite-context)
setup (:setup prepared)
bootstrap-cmd (nth setup 5)]
(is (= "mkdir -p '{{tmp-dir}}/graphs-b'" (first setup)))
(is (= "mkdir -p '{{tmp-dir}}/home/logseq'" (second setup)))
(is (= "cp ~/logseq/auth.json '{{tmp-dir}}/home/logseq/auth.json'" (nth setup 2)))
(is (= "python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{config-path}}' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'"
(nth setup 3)))
(is (= "python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{tmp-dir}}/cli-b.edn' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'"
(nth setup 4)))
(is (string/includes? bootstrap-cmd "sync ensure-keys"))
(is (string/includes? bootstrap-cmd "--upload-keys"))
(is (string/includes? bootstrap-cmd "/tmp/sync-suite/user-rsa-keys.lock"))
(is (string/includes? bootstrap-cmd "/tmp/sync-suite/user-rsa-keys.ready"))
(is (= "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null"
(nth setup 6)))
(is (= "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync ensure-keys --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"
(nth setup 7)))
(is (not-any? #(and (string/includes? % "sync ensure-keys")
(not= % bootstrap-cmd)
(string/includes? % "--upload-keys"))
setup))))
(deftest prepare-case-accepts-custom-e2ee-password
(let [suite-context {:sync-port "18080"
:sync-http-base "http://127.0.0.1:18080"
:sync-ws-url "ws://127.0.0.1:18080/sync/%s"
:e2ee-password "pass word"}
prepared (sync-fixture/prepare-case {:id "sync-case"
:setup []
:cleanup []}
suite-context)]
(is (= "pass word" (get-in prepared [:vars :e2ee-password])))
(is (= "'pass word'" (get-in prepared [:vars :e2ee-password-arg])))))
(deftest before-and-after-suite-only-manage-shared-sync-server-lifecycle
(let [calls (atom [])
run-command (fn [opts]
(swap! calls conj opts)
{:exit 0
:out ""
:err ""})
suite-context (sync-fixture/before-suite! {:run-command run-command})]
(testing "before-suite only starts the shared server"
(is (= 1 (count @calls)))
(is (string/includes? (:cmd (first @calls)) "db_sync_server.py"))
(is (string/includes? (:cmd (first @calls)) " start "))
(is (string/includes? (:cmd (first @calls)) "--data-dir"))
(is (not (string/includes? (:cmd (first @calls)) "--root-dir")))
(is (string/includes? (:cmd (first @calls)) "--port 18080"))
(is (string/includes? (:cmd (first @calls)) "--startup-timeout-s 60"))
(is (not (string/includes? (:cmd (first @calls)) "prepare_sync_config.py")))
(is (not (contains? suite-context :suite-auth-path)))
(is (not (contains? suite-context :suite-config-path))))
(sync-fixture/after-suite! suite-context {:run-command run-command})
(testing "after-suite only stops the shared server once"
(is (= 2 (count @calls)))
(is (string/includes? (:cmd (last @calls)) "db_sync_server.py"))
(is (string/includes? (:cmd (last @calls)) " stop "))
(is (false? (:throw? (last @calls)))))))
(deftest db-sync-server-start-adds-db-sync-node-modules-to-node-path
(let [tmp-dir (fs/create-temp-dir {:prefix "logseq-cli-db-sync-server-test-"})
repo-root (fs/path tmp-dir "repo")
entry (fs/path repo-root "deps" "db-sync" "worker" "dist" "node-adapter.js")
node-modules (fs/path repo-root "deps" "db-sync" "node_modules")
pid-file (fs/path tmp-dir "server.pid")
log-file (fs/path tmp-dir "server.log")
data-dir (fs/path tmp-dir "server-data")
node-path-file (fs/path tmp-dir "node-path.txt")
script (fs/path "cli-e2e" "scripts" "db_sync_server.py")
port (free-port)]
(try
(fs/create-dirs (fs/parent entry))
(fs/create-dirs node-modules)
(spit (str entry)
(str "const fs = require('node:fs');\n"
"const http = require('node:http');\n"
"fs.writeFileSync(" (pr-str (str node-path-file)) ", process.env.NODE_PATH || '', 'utf8');\n"
"http.createServer((req, res) => {\n"
" if (req.url === '/health') { res.writeHead(200); res.end('ok'); return; }\n"
" res.writeHead(404); res.end('not found');\n"
"}).listen(process.env.DB_SYNC_PORT, '127.0.0.1');\n"))
(let [start (shell/sh "python3" (str script)
"start"
"--repo-root" (str repo-root)
"--pid-file" (str pid-file)
"--log-file" (str log-file)
"--data-dir" (str data-dir)
"--port" (str port)
"--startup-timeout-s" "5"
"--auth-path" "")]
(try
(is (= 0 (:exit start)) (:err start))
(is (string/includes? (slurp (str node-path-file)) (str node-modules)))
(finally
(shell/sh "python3" (str script)
"stop"
"--pid-file" (str pid-file)
"--shutdown-timeout-s" "2"))))
(finally
(fs/delete-tree tmp-dir)))))

View File

@@ -0,0 +1,55 @@
from __future__ import annotations
import importlib.util
from pathlib import Path
from types import SimpleNamespace
MODULE_PATH = Path(__file__).resolve().parents[4] / "scripts" / "wait_sync_status.py"
spec = importlib.util.spec_from_file_location("wait_sync_status", MODULE_PATH)
wait_sync_status = importlib.util.module_from_spec(spec)
assert spec.loader is not None
spec.loader.exec_module(wait_sync_status)
def test_resolved_status_command_is_built_once_and_reused() -> None:
args = SimpleNamespace(
cli="~/repo/static/logseq-cli.js",
root_dir="~/tmp/root",
config="~/tmp/cli.edn",
graph="demo",
)
command = wait_sync_status.status_command(args)
assert command[0] == "node"
assert command[2] == "--root-dir"
assert command[-2:] == ["--graph", "demo"]
assert Path(command[1]).is_absolute()
assert Path(command[3]).is_absolute()
assert Path(command[5]).is_absolute()
assert wait_sync_status.status_command(args) is command
def test_run_status_uses_precomputed_command() -> None:
command = ["node", "/abs/cli.js", "--graph", "demo"]
args = SimpleNamespace(status_command=command)
calls = []
def fake_run(cmd, capture_output, text):
calls.append(cmd)
return SimpleNamespace(
returncode=0,
stdout='{"status":"ok","data":{"pending-local":0,"pending-asset":0,"pending-server":0}}',
stderr="",
)
original_run = wait_sync_status.subprocess.run
wait_sync_status.subprocess.run = fake_run
try:
payload = wait_sync_status.run_status(args)
finally:
wait_sync_status.subprocess.run = original_run
assert payload["status"] == "ok"
assert calls == [command]