remote-mcp-manager/LESSONS.md

Lessons

Recurring mistakes and gotchas found while building and operating this project, and how to avoid or work around them.

"Rebuild" must re-clone, not reuse the original checkout

buildInstallation used to run nixpacks build directly against the repo directory left over from createDraftInstallation's one-time clone — it never re-pulled. Every "rebuild" from the panel silently rebuilt the exact same old commit forever, even after pushing new commits to the repo; it looked like the rebuild worked (new image, new timestamp) but the code never changed. Found repeatedly while iterating on nixpacks.toml fixes against a live repo. Fixed by having buildInstallation call cloneOrUpdateRepo again (same as the initial clone) before every build.

A pinned external image's contents can still change

The Dockerfile pinned ghcr.io/railwayapp/nixpacks:1.41.0 and copied /usr/bin/nixpacks out of it. That path existed when the plan was written but was gone by the time of the first real production deploy — the image got repurposed upstream (now ships a Nix daemon, not the nixpacks CLI) even though the tag 1.41.0 still resolves.

Avoid it: pinning a tag is not enough to guarantee what's inside an externally-published image stays the same — only pinning by digest guarantees bytes. For a single binary like this, prefer downloading the pinned release asset directly from the project's GitHub releases (a specific vX.Y.Z tag + exact filename) over extracting it from someone else's Docker image.

git clone ssh://... needs an actual ssh binary, and a key

node:20-slim ships git but not openssh-client — cloning an https:// repo works, but ssh:// fails with "cannot run ssh: No such file or directory". Fixed by installing openssh-client in the final image. Separately, even with ssh present, cloning a private repo needs a key with access mounted into the container (SSH_DIR in docker-compose.yml, reusing the host's own deploy key) — confirmed both failures back to back while deploying.

docker build needs the buildx plugin present to use BuildKit

Nixpacks shells out to docker build, which defaults to BuildKit. The manager's image only copied the bare docker CLI binary from docker:27-cli, not its buildx plugin, so every build failed with "BuildKit is enabled but the buildx component is missing or broken" — confirmed on the first real build attempt. Fixed by also copying /usr/local/libexec/docker/cli-plugins/docker-buildx from the same docker:27-cli image.

Per-container memory/CPU cgroup limits need the host to delegate controllers

docker run --memory=... fails outright ("error setting cgroup config... memory.max: no such file or directory") on a host where cgroup.subtree_control isn't populated down to where the container's cgroup lives. Confirmed on an unprivileged, OpenRC-based nested LXC (Alpine, no systemd) with nesting=1: systemd normally does this delegation automatically at boot; OpenRC doesn't, so cgroup.subtree_control stays empty at the LXC's own root and no controller is ever available to nested Docker containers. CONTAINER_MEMORY_MB/ CONTAINER_CPUS can both be set to 0 to skip asking for a cap at all — on a host like this the outer LXC's own memory limit (if Proxmox/systemd-nspawn/etc. sets one) is the only enforced ceiling either way. Properly fixing the delegation means moving every process already in the LXC's root cgroup into a leaf scope and then writing the wanted controllers to the root's cgroup.subtree_control — not done here, since it's a live production LXC and restructuring its cgroup tree wasn't something to risk without asking first.

A global body parser breaks a reverse proxy mounted on the same app

app.use(express.json()) with no path restriction runs for every request, including the /mcp/:slug proxy route. By the time http-proxy tries to forward the request, body-parser has already fully drained the incoming stream — so the proxied request reaches the backend with the right Content-Length header but zero actual body bytes, and the backend just hangs waiting for a body that will never arrive. Every proxied POST (i.e. every real MCP call) hung with no response — found while testing the very first container through the proxy, not during earlier plain curl-to-container testing (which never went through Express's full middleware stack). Fixed by scoping the JSON parser to /api only (app.use("/api", express.json())), since the proxy route needs the raw stream untouched.

Nixpacks silently picks its own default runtime version

Nixpacks does not fail when a repo's engines/version requirement isn't met — it silently falls back to its own default (e.g. picked Node 18 for a repo requiring Node >=20 in Social-MCP). The build goes green, then the container crashes at runtime (crypto is not defined, missing globals, etc.).

Avoid it: don't trust a green build as proof the runtime is correct. The manager now detects a Node version hint from the repo itself (.nvmrc, else package.json engines.node) and passes it to Nixpacks (apps/server/src/build/nixpacks.ts, detectNodeVersion), so the build uses the version the repo actually declares instead of Nixpacks' own default. A Node repo (has package.json) with no hint of its own gets DEFAULT_NODE_VERSION (currently 20) instead of falling through to Nixpacks' own default — confirmed necessary against a real repo (ZenvairoSocial-MCP, no .nvmrc/engines.node at all): build green, container crashed with "crypto is not defined" because Nixpacks picked Node 18, which predates stable globalThis.crypto.

Gotcha inside the gotcha: nixpacks does not read NIXPACKS_* config from the process environment at all — only from its own -e/--env CLI flag. Setting NIXPACKS_NODE_VERSION as a plain env var on the spawned process is silently ignored (nixpacks plan still showed nodejs_18); the fix above only actually took effect once passed as nixpacks build ... -e NIXPACKS_NODE_VERSION=20. Caught because the very first container built after adding the "fix" still crashed the same way.

Nixpacks needs a dependency manifest, and fails cleanly without one

No requirements.txt/pyproject.toml (Python) or package.json (Node) in the installed repo → Nixpacks fails with "Nixpacks was unable to generate a build plan" (exit 1). This is expected and not something the manager works around generically — several of the user's own Python MCPs lack a manifest entirely.

Avoid it: always surface the full build log in the UI so this failure is immediately diagnosable; don't try to add generic manifest-inference logic.

Piped exit codes mask the real command result

Piping a subprocess's output through another command (e.g. | tee) loses the original exit code in $?. This hid a real nixpacks build failure during manual testing.

Avoid it: redirect output straight to a file and check $? immediately after the command, don't pipe when the exit code matters.

Rootless Podman isolates the host network namespace from the container bridge

The host cannot reach a container's bridge-network IP directly under rootless Podman (netavark backend, "isolate": "true"). Only another container on the same network can reach it.

Avoid it: for local dev, run the manager itself as a container attached to mcp-net — this also matches the production topology, where the manager runs as a container too.

SELinux can deny access to a bind-mounted socket even with matching UID/GID

Got an EACCES on the Podman socket despite --userns=keep-id lining up UID/GID. Root cause was an SELinux AVC denial in enforcing mode, not a permissions issue.

Avoid it: when socket access fails despite correct ownership, check audit.log/ausearch for AVC denials before chasing permissions further. --security-opt label=disable confirms the diagnosis but is not a production fix — use a proper SELinux policy/context instead.

Podman images must be referenced fully-qualified

A short image name wasn't enough to remove an image via the Docker-compatible socket under Podman — needed the fully-qualified localhost/... name.

Avoid it: always use the fully-qualified name dockerode/the Docker API returns, don't assume short names round-trip.

npm workspaces hoist node_modules to the repo root

Mounting only apps/server into a test container produced node_modules not found — the hoisted root node_modules wasn't mounted.

Avoid it: mount the whole repo root when testing a workspace package, not just its subdirectory.