Files
remote-mcp-manager/LESSONS.md
Dasemu 306043e5b9
All checks were successful
CI / test (push) Successful in 44s
docs: add lesson about pinned-tag images changing contents upstream
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-08 04:22:18 +02:00

58 lines
4.1 KiB
Markdown

# Lessons
Recurring mistakes and gotchas found while building and operating this project, and how to avoid or work around them.
## A pinned external image's *contents* can still change
The Dockerfile pinned `ghcr.io/railwayapp/nixpacks:1.41.0` and copied `/usr/bin/nixpacks` out
of it. That path existed when the plan was written but was gone by the time of the first
real production deploy — the image got repurposed upstream (now ships a Nix daemon, not the
nixpacks CLI) even though the tag `1.41.0` still resolves.
**Avoid it:** pinning a tag is not enough to guarantee what's inside an externally-published
image stays the same — only pinning by digest guarantees bytes. For a single binary like this,
prefer downloading the pinned release asset directly from the project's GitHub releases (a
specific `vX.Y.Z` tag + exact filename) over extracting it from someone else's Docker image.
## Nixpacks silently picks its own default runtime version
Nixpacks does not fail when a repo's `engines`/version requirement isn't met — it silently falls back to its own default (e.g. picked Node 18 for a repo requiring Node >=20 in `Social-MCP`). The build goes green, then the container crashes at runtime (`crypto is not defined`, missing globals, etc.).
**Avoid it:** don't trust a green build as proof the runtime is correct. The manager now detects a Node version hint from the repo itself (`.nvmrc`, else `package.json` `engines.node`) and passes it to Nixpacks as `NIXPACKS_NODE_VERSION` (`apps/server/src/build/nixpacks.ts`, `detectNodeVersion`), so the build uses the version the repo actually declares instead of Nixpacks' own default. Repos with no version hint at all still get Nixpacks' default, unchanged.
## Nixpacks needs a dependency manifest, and fails cleanly without one
No `requirements.txt`/`pyproject.toml` (Python) or `package.json` (Node) in the installed repo → Nixpacks fails with "Nixpacks was unable to generate a build plan" (exit 1). This is expected and not something the manager works around generically — several of the user's own Python MCPs lack a manifest entirely.
**Avoid it:** always surface the full build log in the UI so this failure is immediately diagnosable; don't try to add generic manifest-inference logic.
## Piped exit codes mask the real command result
Piping a subprocess's output through another command (e.g. `| tee`) loses the original exit code in `$?`. This hid a real nixpacks build failure during manual testing.
**Avoid it:** redirect output straight to a file and check `$?` immediately after the command, don't pipe when the exit code matters.
## Rootless Podman isolates the host network namespace from the container bridge
The host cannot reach a container's bridge-network IP directly under rootless Podman (netavark backend, `"isolate": "true"`). Only another container on the same network can reach it.
**Avoid it:** for local dev, run the manager itself as a container attached to `mcp-net` — this also matches the production topology, where the manager runs as a container too.
## SELinux can deny access to a bind-mounted socket even with matching UID/GID
Got an `EACCES` on the Podman socket despite `--userns=keep-id` lining up UID/GID. Root cause was an SELinux AVC denial in enforcing mode, not a permissions issue.
**Avoid it:** when socket access fails despite correct ownership, check `audit.log`/`ausearch` for AVC denials before chasing permissions further. `--security-opt label=disable` confirms the diagnosis but is not a production fix — use a proper SELinux policy/context instead.
## Podman images must be referenced fully-qualified
A short image name wasn't enough to remove an image via the Docker-compatible socket under Podman — needed the fully-qualified `localhost/...` name.
**Avoid it:** always use the fully-qualified name dockerode/the Docker API returns, don't assume short names round-trip.
## npm workspaces hoist node_modules to the repo root
Mounting only `apps/server` into a test container produced `node_modules not found` — the hoisted root `node_modules` wasn't mounted.
**Avoid it:** mount the whole repo root when testing a workspace package, not just its subdirectory.