AUTHENTICATION_ERROR on self-hosted n8n behind k8s ingress, despite identical curl returning 200

[klfjsljkd]

AUTHENTICATION_ERROR on self-hosted n8n behind k8s ingress, despite identical curl returning 200

Summary

Every authenticated tool call (n8n_list_workflows, n8n_get_workflow,
etc.) returns AUTHENTICATION_ERROR against a self-hosted n8n instance
behind a Kubernetes ingress, while a raw curl with the identical
X-N8N-API-KEY header from the same shell returns 200.

n8n_health_check reports connected: true because it probes /healthz
(unauthenticated), so the diagnostic is misleadingly green.

Reproduced on both 2.54.0 and 2.56.0 (latest at time of report).

Environment

• n8n-mcp: tested 2.54.0 and 2.56.0 — same failure on both

• n8n: self-hosted (recent), behind a k8s ingress

• Topology:

  • Internal DNS resolves to an RFC1918 address
  • HTTP (not HTTPS) on port 80, ingress terminates and routes by Host
    header

• API key shape: JWT-style (eyJhbGci…), ~267 chars, generated via
  the n8n UI → Settings → n8n API → Create an API key

• Host OS: macOS, Node v22.17.0, npx-launched MCP via the host
  agent

• MCP configuration for the n8n server (sanitized):

  "n8n": {
    "command": "npx",
    "args": ["-y", "n8n-mcp"],
    "env": {
      "N8N_API_URL": "http://<internal-n8n-host>",
      "N8N_API_KEY": "${N8N_API_KEY}",
      "MCP_MODE": "stdio",
      "WEBHOOK_SECURITY_MODE": "permissive"
    }
  }

WEBHOOK_SECURITY_MODE=permissive also exported in the launching
shell (verified reaching the subprocess).
N8N_API_KEY exported in the launching shell (verified: same value
as in the n8n UI; non-empty inside the subprocess).
Repro
npx -y n8n-mcp from a shell with the four env vars above set
From an MCP client, call n8n_list_workflows
Expected: list of workflows
Actual:

{
"success": false,
"error": "Failed to authenticate with n8n. Please check your API key.",
"code": "AUTHENTICATION_ERROR"
}
Smoking gun
From the same shell at the same time, the identical header succeeds:

curl -sS -o /dev/null -w "HTTP %{http_code}\n"
-H "X-N8N-API-KEY: $N8N_API_KEY"
"http:///api/v1/workflows?limit=1"

→ HTTP 200

Re-tested with axios-style UA + accept headers, with empty UA, with
explicit --http1.1 — all return 200 to curl. Only the n8n-mcp axios
client gets 401.

What we ruled out
❌ Not the API key — same key works in curl from the same env
(verified JWT prefix + length).
❌ Not SSRF blocking — WEBHOOK_SECURITY_MODE=permissive is set
AND a SSRF-blocked request would map to REQUEST_ERROR per
utils/n8n-errors.js, not AUTHENTICATION_ERROR. So the request IS
reaching the n8n server.
❌ Not header construction — services/n8n-api-client.js L72
correctly sends X-N8N-API-KEY: apiKey (not Bearer).
❌ Not a redirect — maxRedirects: 0 + /api/v1/workflows
returns 200 directly with no redirect_url.
❌ Not a User-Agent filter on the ingress — curl with axios-style
UA returns 200.
❌ Not an HTTP-version mismatch — curl with --http1.1 returns
200.
❌ Not a Host-header issue from SSRF pinning — pinned-DNS connect
to the resolved IP with the correct Host header returns 200 via curl.
❌ Not a recent regression — same failure on 2.54.0 and
2.56.0.
Conjecture
The bug appears isolated to how the n8n-mcp axios client interacts with
this n8n+ingress combination. Possible investigation directions:

The keepAlive: false pinned agent — subtle connection-reuse or
sequencing issue when the SSRF-pinned lookup callback is in play?
Some interceptor stripping or rewriting the X-N8N-API-KEY header
before the wire? (We did NOT verify with tcpdump on the local
machine; this would be the next concrete diagnostic step.)
An axios HTTP/HTTPS upgrade that breaks the n8n ingress contract on
HTTP-only deployments?
Suggested next steps
A tcpdump capture of the actual on-wire request from the n8n-mcp
subprocess (happy to produce this on request — let us know the
format).
A debug flag that logs the literal request.headers axios emits at
request-interception time would close the gap quickly without
requiring users to tcpdump.
Optional: an N8N_API_DEBUG=1 env var that prints the response
data body for non-2xx responses — currently the error path
swallows the n8n-side message and only surfaces the generic
Failed to authenticate.
Happy to provide additional diagnostics on request — fast turnaround
since we're actively trying to adopt this MCP.

[czlonkowski]

Thanks for the unusually thorough report, @klfjsljkd — the ruled-out list and the curl-vs-axios comparison made this much faster to reason about.

What we found in the current code

We walked the authenticated request path and your diagnosis is essentially correct on every point we can verify from this side:

• AUTHENTICATION_ERROR is only ever produced from a genuine HTTP 401 — handleN8nApiError maps status === 401 → N8nAuthenticationError (src/utils/n8n-errors.ts), and nothing else in the client synthesizes that code. SSRF rejection throws before the request and surfaces as REQUEST_ERROR/an SSRF message, not AUTHENTICATION_ERROR. So your "the request reached n8n and n8n answered 401" reading is right.
• The header is sent verbatim as X-N8N-API-KEY: <apiKey> (src/services/n8n-api-client.ts), no Bearer, no rewrite.
• maxRedirects: 0, so no redirect is silently dropping the header.
• The SSRF-pinned agents (createPinnedAgents in src/utils/ssrf-protection.ts) only override DNS lookup to pin the connection to the validated IP. They do not touch request headers and do not set/override the Host header — axios derives Host from the configured N8N_API_URL, the same host your curl uses. keepAlive: false means each request is a fresh connection; there's no cross-request connection reuse that could carry stale state.

So from the client's perspective the request that goes on the wire should be header-identical to your curl. Given that, a 401 reaching the client strongly implies something between the n8n-mcp process and the n8n app server is treating these two requests differently — most likely the k8s ingress stripping/normalizing the X-N8N-API-KEY header (some ingress controllers drop or rewrite headers with underscores/specific casing, or apply different rules per route), or routing the axios request to a different upstream/n8n than your curl hits. The fact that curl from the same shell with axios-style UA, --http1.1, and pinned DNS all return 200 points outward, not at the client.

What would let us confirm (or refute) that

Two captures would close this quickly:

1. The actual outbound request headers as seen at the wire from the n8n-mcp process — a tcpdump/packet capture (or mitmproxy in front of the pinned IP) of the real request the subprocess emits, so we can compare it byte-for-byte against your working curl. The header name/casing and presence of X-N8N-API-KEY on that specific request is the key thing.
2. n8n's own access/auth logs for the 401 — the request line n8n logged for the rejected call (path, and whether it saw the API-key header at all). If n8n logs the request without the key header while curl's reached it with the key, that pins it on the ingress.

If you can run the capture against the pinned RFC1918 IP at the same time as the failing tool call, that pairing (what we send vs. what n8n received) should be conclusive.

On the debug-flag suggestions

You're right that the current error path swallows the n8n-side response body and only surfaces the generic message, and that there's no flag to dump the literal emitted request headers. LOG_LEVEL=debug today logs the request method+URL and the response status, but not request headers or the non-2xx response body. The two improvements you suggested — logging emitted request headers at interception time, and an opt-in flag to surface the non-2xx response body — are reasonable diagnostics and we're tracking them as a follow-up here. (We'd redact the API key value when logging headers.)

Keeping this open. The packet capture + n8n access log for the 401 are the next concrete steps — happy to look as soon as you can attach them.