Linkerd Mesh Expansion

[gangadharkasturi]

Mesh Expansion with Linkerd for Non-Kubernetes Workloads (Java JAR / Docker) – Guidance Needed

Description

I have a Kubernetes cluster set up with Linkerd installed, including linkerd-viz and linkerd-proxy. I am trying to use Linkerd mesh expansion to bring non-Kubernetes workloads into the service mesh.

Goal

I want to:

• Run my main application as a plain Java process (java -jar app.jar) outside Kubernetes
• Run additional workloads in Docker on other servers (also outside Kubernetes)
• Connect both types of workloads to the Linkerd service mesh using mesh expansion
• Avoid deploying these applications into Kubernetes entirely
• Still leverage service mesh features such as:
  • mTLS
  • Observability
  • Traffic policies

Current Setup

1. Installed:

   • Linkerd control plane
   • linkerd-viz
   • linkerd-proxy

2. Installed SPIRE components:

   • SPIRE Server
   • SPIRE Agent

3. Configured workloads for mesh expansion

Issue

After configuring workloads, I proceeded with network configuration changes (including iptables rules). However, this caused unexpected IP routing issues and broke connectivity on the host.

At this point, I am stuck and unable to move forward safely.

What I am looking for

I need clarification and guidance on:

• The correct end-to-end steps to implement Linkerd mesh expansion for:
  • Java applications running directly on VMs/bare metal (java -jar)
  • Docker-based workloads on non-Kubernetes hosts
• Whether SPIRE is required for this setup, or if there is a simpler recommended approach
• How to properly configure networking (especially iptables) without breaking host routing
• Any reference architecture or working example for this exact use case

Notes

I have already gone through the official Linkerd documentation, but I find it unclear for this specific scenario (non-Kubernetes workloads + mesh expansion). I am looking for a practical, working example or clearer step-by-step guidance.

[cookesan]

Mesh expansion is the right feature, but it is not just "run linkerd-proxy next to any process".

The current Linkerd mesh-expansion guide has four moving parts:

1. install the Linkerd proxy on the VM/bare-metal host,
2. configure network rules so traffic is routed through the proxy,
3. register the external workload in Kubernetes,
4. use SPIRE/SPIFFE identity for the off-cluster workload.

SPIRE is part of the documented path because off-cluster processes do not have Kubernetes ServiceAccount tokens for Linkerd identity. The guide says Linkerd's normal in-cluster identity service uses those ServiceAccount tokens, while external workloads use SPIRE-issued identities compatible with Linkerd mTLS.

The networking prerequisites are also strict: the host needs privileges to change iptables, IP connectivity to every pod in the mesh, and DNS that can resolve in-cluster Kubernetes workloads. If iptables broke host routing, I would stop there and reproduce the guide on a disposable VM first, before applying it to production hosts.

For Java vs Docker, the shape is the same: the workload is represented by an ExternalWorkload resource, and the local machine routes traffic through the proxy. Docker does not remove the need for the host/proxy/networking and identity setup.

I do not see a simpler supported path in the docs that avoids SPIRE for arbitrary non-Kubernetes workloads. If you already have SPIFFE infrastructure, you may not need the tutorial's minimal SPIRE setup, but you still need workload identity, proxy setup, routing, and ExternalWorkload registration.

[gkspyder777]

Thanks for the detailed comment.

I would really appreciate if you could please help with below details in depth.

How to properly configure networking (especially iptables) without breaking host routing.
Any reference architecture or working example or document for this exact use case.
The documentation mentioned here : https://linkerd.io/2-edge/tasks/adding-non-kubernetes-workloads/ is very minimal and it lacks the understanding.

Thank you.

[cookesan]

I would not apply the guide's sample rules unchanged on a general-purpose host. They are the same basic shape as Linkerd's pod iptables rules, but in Kubernetes they are scoped to the pod network namespace; on a VM they affect the host namespace unless you isolate them. The guide's core requirements are privileges to modify iptables, pod reachability, and DNS for in-cluster names. Guide: https://linkerd.io/2-edge/tasks/adding-non-kubernetes-workloads/ and iptables reference: https://linkerd.io/2-edge/reference/iptables/

The safest reference architecture is a dedicated VM/container/network namespace per external workload: one app, one linkerd2-proxy, one isolated set of nat rules. Run the proxy under its own Unix UID and the app under a different UID, because the outbound --uid-owner bypass must match only the proxy. Then narrow the redirects instead of redirecting every host TCP flow:

PROXY_UID=$(id -u linkerd-proxy)
APP_PORTS=8080
SERVICE_CIDR=10.96.0.0/12
POD_CIDR=10.244.0.0/16

iptables -t nat -N PROXY_INIT_REDIRECT 2>/dev/null || iptables -t nat -F PROXY_INIT_REDIRECT
iptables -t nat -A PROXY_INIT_REDIRECT -p tcp -m multiport --dports 22,4190,4191 -j RETURN
iptables -t nat -A PROXY_INIT_REDIRECT -p tcp -m multiport --dports "$APP_PORTS" -j REDIRECT --to-port 4143
iptables -t nat -C PREROUTING -j PROXY_INIT_REDIRECT 2>/dev/null || iptables -t nat -A PREROUTING -j PROXY_INIT_REDIRECT

iptables -t nat -N PROXY_INIT_OUTPUT 2>/dev/null || iptables -t nat -F PROXY_INIT_OUTPUT
iptables -t nat -A PROXY_INIT_OUTPUT -m owner --uid-owner "$PROXY_UID" -j RETURN
iptables -t nat -A PROXY_INIT_OUTPUT -o lo -j RETURN
iptables -t nat -A PROXY_INIT_OUTPUT -p tcp -m multiport --dports 22,4190,4191 -j RETURN
iptables -t nat -A PROXY_INIT_OUTPUT -p tcp -d "$SERVICE_CIDR" -j REDIRECT --to-port 4140
iptables -t nat -A PROXY_INIT_OUTPUT -p tcp -d "$POD_CIDR" -j REDIRECT --to-port 4140
iptables -t nat -C OUTPUT -j PROXY_INIT_OUTPUT 2>/dev/null || iptables -t nat -A OUTPUT -j PROXY_INIT_OUTPUT

Adjust APP_PORTS, skipped admin ports, and CIDRs for your cluster. Use iptables-nft consistently if your distro uses nftables. Before enabling it on a real VM, take iptables-save -t nat backups, apply the rules idempotently, and verify with iptables-save -t nat, ss -lntp, curl from the VM to a meshed service, and Linkerd metrics/tap from the in-cluster side. If you need this at scale, Buoyant Enterprise's mesh-expansion automation is the productized path; the OSS working example is the Linkerd guide plus a carefully scoped wrapper around those rules.

[gkspyder777]

Thanks for your detailed inputs.