Adding Envoy Gateway as control plane ingress

[laurabarton]

Summary

This is the first iteration on deploying Envoy Gateway (EG) as a replacement for Ingress NGINX. This creates the necessary charts for adding EG and HTTP/GRPC routes.

This supports running both EG and Ingress NGINX in the same cluster, using the INGRESS_PROVIDER global value. You can choose either nginx, envoy, or both.

Included Features

Control Plane

• Backend Traffic Policy - For defining connection timeout, tcp keep alive, http2 support, and rate limiting for the EG proxy to the downstream GRPC service/connections.
• ClientTrafficPolicy - Used to set timeout and buffer size limits
• Gateway - configures the gateway itself. Serverless is currently still support, but will be pulled out once we tear down serverless.
• All the routes

Envoy Proxy

• Patch policy - Used to define the http filters used for auth token interception/injection. And caching of requests to auth-proxy to avoid hitting rate limits.
• Proxy configuration itself
• Redis configuration - used by the rate limiter.

• jan/wip-selfhosted - ⚠️ No PR associated with branch
  • update #323
    • Adding Envoy Gateway as control plane ingress 👈

[aviator-app]

Current Aviator status

Aviator will automatically update this comment as the status of the PR changes.
Comment /aviator refresh to force Aviator to re-examine your PR (or learn about other /aviator commands).

This PR was merged using Aviator.

---

See the real-time status of this PR on the Aviator webapp.

Use the Aviator Chrome Extension to see the status of your PR within GitHub.

[xjerod]

Could you add a section somewhere that explains a bit more on how the setup looks like with both enabled? The pre and post migration states are pretty clear, but it's a bit fuzzy how ingress will function during the dual deployment phase

[laurabarton]

I'll update the comment in a follow-up PR. But essentially, they aren't meant to operate both for long. This is so when we are in the process of migrating we can switch back if something goes wrong. There is a flag (set the cloud repo in the terraform) which will update the external-dns to either use "ingress" or "httproute"/"grpcroute" as it's target which tell the dns to to either use nginx or envoy. I have also setup weighted routing so we can filter only a small percentage of requests during testing if we would like

[xjerod]

What happens if a user specifies something other than the required options, is this a helm error because it can't render? Or is there a way to validate the field? (I'm still learning helm details)

[laurabarton]

That's a good question. I think if the value is neither than the gated logic in charts/controlplane/templates/flyte-core-app.yaml will not render either chart I guess.

Here you can see that if the value if missing completely it will default to nginx.

if or (eq (default "nginx" .Values.global.INGRESS_PROVIDER) "nginx") (eq .Values.global.INGRESS_PROVIDER "both") }}

I could probably update the logic to look for something other than "envoy" then render "nginx" if we are concerned about that

[xjerod]

Okay cool, makes sense, I don't think we need to add any extra logic, this behavior makes sense

[xjerod]

Is this for patching envoy gateway's configuration?

[laurabarton]

This is patching EnvoyGateway itself. This is how the plugins work. We build a .so binary from the envoy-gateway contrib base image and then build in our plugins that are accessible through a mounted library path.
It's hard to tell, because this is the cloud repo. I believe that if the filters aren't yet commited this fails, but gracefully...well sort of no auth an therefore 500 errors. This is why I was thinking I might break up my PR into smaller PRs.