Best Recommendation for level where to terminate the TLS.

[chandreshreddyg17]

Hello All,
I have deployed APISIX gateway in decouple mode. I have one control plane to manage the admin api and two data plane to mange the client traffic and a Load Balancer to distribute client traffic to data planes. ETCD cluster to store the data. I need the requests in https so the data must be stayed in encrypted. I want to terminate the tls where I place the certs and keys.
So I am thinking to terminate tls at the Load Balancer level or data plane (gateway) level (storing certs and keys in etcd so all the data plane shares the same).

@dosu

[dosubot[bot]]

Both options are valid — here's a breakdown to help you decide:

Option A: Terminate TLS at the Load Balancer

• Offloads the computationally expensive TLS handshakes from your APISIX data planes [1]
• Simpler certificate management — you only manage certs on the LB
• Traffic between the LB and APISIX data planes would be unencrypted (unless you add a second layer of TLS)

Option B: Terminate TLS at the APISIX Data Plane (recommended for your use case)

Since you explicitly need data to stay encrypted end-to-end and you're already running in decoupled mode with etcd, this is the stronger fit:

• SSL certificates are stored in etcd under /ssls and automatically synchronized to all data plane nodes in real-time via etcd's watch mechanism [2]. This means both your data planes will share the same certs without manual distribution.
• Certificate updates propagate to all DPs without dropping active connections or requiring a reload — this is called "hitless rotation" [3].
• APISIX uses SNI-based matching to select the correct certificate during the TLS handshake [4], so you can manage multiple domains from a single deployment.
• You get visibility into TLS connection details at the gateway level, which enables fine-grained security policies and mTLS if needed.
• You can also reference secrets from external managers using $secret:// or $env:// prefixes instead of storing raw keys in etcd [5].

Hybrid option (also common): Use a TCP/L4 load balancer with TLS passthrough to distribute traffic to your APISIX data planes, which then perform the actual TLS termination [6]. This gives you both HA load distribution and end-to-end encryption. One caveat: you may need Proxy Protocol v2 on the LB to preserve client IPs, and there's a known quirk where X-Forwarded-Port may reflect the internal APISIX port rather than 443 [6].

My recommendation: Given your requirement that data must stay encrypted and you already have etcd in place, terminate TLS at the APISIX data plane level with a TCP passthrough load balancer in front. This ensures encryption all the way to the gateway while still leveraging your LB for traffic distribution and health checks. Store certs via the Admin API (control plane) and etcd will handle synchronization to both data planes automatically.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[pick7]

The deciding question is what you mean by “encrypted”.

If TLS terminates at the load balancer, then client-to-LB traffic is encrypted, but LB-to-APISIX is plaintext unless you add a second TLS hop. That is fine for many internal networks, but it is not end-to-end encryption to the gateway.

If you need traffic to remain encrypted all the way to APISIX data planes, use this layout:

client --TLS--> L4/TCP load balancer --TLS passthrough--> APISIX data plane

Then terminate TLS on APISIX. Manage certificates through the APISIX SSL resource on the control plane; the data planes will receive the SSL config from etcd/config sync. This gives you:

• one shared certificate config for both data planes,
• SNI-based certificate selection at APISIX,
• the ability to do mTLS/client-cert checks at APISIX if needed,
• no plaintext HTTP segment between the LB and gateway.

The trade-off is that a pure TCP/L4 load balancer cannot inspect HTTP paths/headers. If you need WAF/routing/HTTP logic at the LB, then use L7 TLS termination at the LB and re-encrypt to APISIX:

client --TLS--> L7 load balancer --TLS again--> APISIX data plane

In that case, configure APISIX to listen with TLS too, and make sure X-Forwarded-* / proxy protocol handling matches how you preserve client IP and original scheme.

So my recommendation would be:

• security requirement is encryption up to APISIX: terminate at APISIX, LB in TCP passthrough mode;
• requirement is LB-level HTTP routing/WAF: terminate at LB, then re-encrypt to APISIX;
• avoid LB termination followed by plaintext to APISIX if the requirement says the data must stay encrypted.

Also protect etcd/control-plane access carefully, because if SSL private keys are stored in APISIX SSL resources, etcd/config access becomes part of your certificate security boundary.

[chandreshreddyg17]

@pick7
I got the complete picture.
Thank you.

[chandreshreddyg17]

@dosu @pick7
I have a Linux RHEL virtual machine.
I do not want to expose the certs and keys, for that I need to implement the vault integration.
Can you tell me how to do that.

[dosubot[bot]]

APISIX has native HashiCorp Vault integration for this exact use case. Here's how to set it up:

Important prerequisite: APISIX only supports Vault KV engine version 1 [1]. KV v2 is not supported.

1. Set up Vault (on your RHEL VM or a separate Vault server)

# Enable KV v1 engine
vault secrets enable -path=kv -version=1 kv

# Store your certificate and key
vault kv put kv/apisix/ssl/example.com \
  cert=@/path/to/server.crt \
  key=@/path/to/server.key

# Create a least-privilege policy for APISIX
vault policy write apisix-ssl - <<EOF
path "kv/apisix/ssl/*" {
  capabilities = ["read"]
}
EOF

# Generate a token for APISIX
VAULT_TOKEN=$(vault token create -policy=apisix-ssl -format=json | jq -r '.auth.client_token')

2. Register Vault as a Secret resource in APISIX [2] [3]

curl -i "http://127.0.0.1:9180/apisix/admin/secrets/vault/my-vault" \
  -X PUT \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "uri": "http://YOUR_VAULT_ADDRESS:8200",
    "prefix": "kv/apisix",
    "token": "your-vault-token"
  }'

You can also protect the Vault token itself using an environment variable: "token": "$ENV://VAULT_TOKEN" [2].

3. Create an SSL object that references Vault [3]

Instead of passing the raw PEM cert/key, use the $secret:// reference format:

curl -i "http://127.0.0.1:9180/apisix/admin/ssls/1" \
  -X PUT \
  -H "X-API-KEY: ${APISIX_ADMIN_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "sni": "example.com",
    "cert": "$secret://vault/my-vault/ssl/example.com/cert",
    "key": "$secret://vault/my-vault/ssl/example.com/key"
  }'

The reference format is $secret://vault/{secret-resource-id}/{vault-path}/{field}, where the vault path is relative to the prefix you configured in step 2 [2].

How it works at runtime: During TLS handshake, APISIX resolves the secret URIs by calling Vault's API, retrieves the actual cert/key values, and uses them for the connection. This way, your certificates and private keys never exist in plaintext in etcd or APISIX configuration [4].

Key notes:

• No special config.yaml changes are needed — Vault is configured entirely through the Admin API [1]
• Since you're in decoupled mode, the Secret resource gets stored in etcd and is available to all data planes automatically
• Use a read-only, scoped Vault token in production — never use a root token [5]
• Secrets are cached, so certificate rotations in Vault may not take effect immediately

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[chandreshreddyg17]

@dosu
Thank you for the plane to do that.
Are there any other things in APISIX both the control plane and data plane to secure using the hashicrop vault integration.
I will cover those things also

[dosubot[bot]]

@chandreshreddyg17 Yes — the $secret://vault/... reference works across many APISIX resources beyond just SSL. Here's a comprehensive picture of what you can protect:

TLS / SSL (already covered)

• cert, key, certs, keys in SSL objects — TLS private keys [1]

---

Authentication plugins (consumer configuration) [2]

• key-auth → key
• basic-auth → password
• hmac-auth → secret_key
• jwt-auth → secret (and RSA private key)
• openid-connect → client_secret, client_rsa_private_key
• authz-keycloak → client_secret
• authz-casdoor → client_secret
• jwe-decrypt → key, secret

---

Logger plugins (fields marked as encrypt_fields in their schemas)

• clickhouse-logger → password
• elasticsearch-logger → auth.password
• sls-logger → access_key_secret
• rocketmq-logger → secret_key
• tencent-cloud-cls → secret_key
• google-cloud-logging → auth_config.private_key
• error-log-logger → clickhouse.password

---

Other plugins

• kafka-proxy → sasl.password
• csrf → key
• openwhisk → service_token

Note: The specific logger and other plugin fields listed above are based on their encrypt_fields schema definitions, but I wasn't able to verify every individual field directly. You should double-check each plugin's schema in the source code to confirm the exact field names.

---

Core / startup config (these use $ENV:// rather than $secret://, since they're read before APISIX can reach Vault) [2]

• config.yaml: etcd username/password, etcd client TLS certs/keys
• apisix.ssl.key_encrypt_salt — the key used to encrypt private keys before they're written to etcd (especially sensitive since compromising it exposes all stored private keys)
• apisix.data_encryption.keyring — the keyring used to encrypt encrypt_fields values at rest in etcd

For these startup config values, the recommended approach is environment variables ($ENV://), set before starting APISIX on each node.

---

The general rule: any plugin field that has encrypt_fields defined in its schema can reference Vault via $secret://vault/.... The pattern is the same everywhere — store the value in Vault, register your Vault Secret resource once, then reference it wherever needed [2] [1].

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[chandreshreddyg17]

@dosu
Can we do the vault integration for the config file credentilas,like the admin key, keys and certs .

[dosubot[bot]]

Yes, Vault integration is possible, but the method depends on which credentials you want to protect.

Config file values (admin key, etcd password, key_encrypt_salt, etc.)

The $secret://vault/... URI syntax is a runtime feature that only works in plugin configurations and SSL objects — it is not supported in config.yaml [1]. However, config.yaml has its own ${{VAR}} environment variable substitution that is resolved at startup [2] [3]. You can pull secrets from Vault into env vars before starting APISIX:

export APISIX_ADMIN_KEY=$(vault kv get -field=admin_key kv/apisix/config)
apisix start

Then reference them in config.yaml:

deployment:
  admin:
    admin_key:
      - name: admin
        key: ${{APISIX_ADMIN_KEY}}
        role: admin

The same pattern works for etcd mTLS cert paths, passwords, and other startup values. A common approach is to use Vault Agent to populate the environment before APISIX launches, keeping Vault as the single source of truth.

SSL certs/keys for client-facing TLS (data plane)

These go through SSL objects and fully support native Vault integration via the $secret://vault/... syntax [4] [1]:

# Store in Vault
vault kv put apisix/my-cert cert=@server.pem key=@server.key

# Reference in an SSL object
curl http://127.0.0.1:9180/apisix/admin/ssls/1 \
  -H "X-API-KEY: $admin_key" -X PUT -d '{
    "sni": "example.com",
    "cert": "$secret://vault/1/my-cert/cert",
    "key":  "$secret://vault/1/my-cert/key"
  }'

Plugin secrets (API keys, passwords, etc.)

Also fully supported with $secret://vault/... — for example, in the key-auth plugin: "key": "$secret://vault/1/jack/auth-key" [5].

Quick summary:

Credential	How to use Vault
Admin API key, etcd password, key_encrypt_salt (config.yaml)	${{VAR}} env var, populated from Vault before startup
SSL certs/keys (data plane TLS)	Native $secret://vault/... in SSL objects
Plugin secrets (auth keys, passwords)	Native $secret://vault/... in plugin config

Note that APISIX only supports Vault KV engine version V1 [6].

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.