Error Ensuring Firewall rules when running on Clusters in a Shared VPC

[boredabdel]

Hi,

I'm having an issue with kubemci trying to create a multi cluster ingress across two clusters that are attached to a sharedvpc.

Shared VPC in project A. This is the Host project
Cluster 1 in project B Attached to the Shared VPC in project A
Cluster 2 in project B Attached to the Shared VPC in project A

Same service running on both clusters listening to the same node port.

I0227 15:24:45.316852 52427 gen.go:5430] GCEFirewalls.Insert(context.Background, Key{"mci1-fr--nginx-mci"}, ...): projectID = PROJECT_B, rk = &{ProjectID:PROJECT_B Operation:Insert Version:ga Service:Firewalls}
I0227 15:24:45.408566 52427 gen.go:5441] GCEFirewalls.Insert(context.Background, Key{"mci1-fr--nginx-mci"}, ...) = googleapi: Error 400: Invalid value for field 'resource.network': 'https://www.googleapis.com/compute/v1/projects/PROJECT_A/global/networks/PROJECT_A'. Cross project referencing is not allowed for this resource., invalid
Error ensuring firewall rule: Error googleapi: Error 400: Invalid value for field 'resource.network': 'https://www.googleapis.com/compute/v1/projects/PROJECT_A/global/networks/PROJECT_A
. Cross project referencing is not allowed for this resource., invalid in ensuring firewall rule
Error: Error in creating load balancer: 1 error occurred:

• Error googleapi: Error 400: Invalid value for field 'resource.network': 'https://www.googleapis.com/compute/v1/projects/PROJECT_A/global/networks/PROJECT_A'. Cross project referencing is not allowed for this resource., invalid in ensuring firewall rule.

What i think happens is the following

1 - kubemci check the project hosting the cluster (PROJECT_B in my case) for a specific firewall rule.
2 - The call fails, so it calls the EnsureFirewallRule function to create those.
3 - The call fails because the API called behind apparently doesn't support creating a firewall rule in a different project then the one where the call is made. hence the error "Cross project referencing is not allowed for this resource"

I understand that kubemci is trying to create fw rules for the LB Health Check and Backend Service IP Ranges. But we could create those in the shared VPC project manually or via a script. will it be possible to have a flag to kubemci to skip trying to create firewall rules ?

[jawlitkp]

Having the same issue. please provide work around

[boredabdel]

This have been fixed already.

You just need to add the following annotation to the ingress definition to silence the alerts about firewall rules.

networking.gke.io/suppress-firewall-xpn-error: "true"

[boredabdel]

Got confused with this and an other issue, reopening as we do need a workaround for this issue,

[artazar]

I am facing exactly the same issue, is there any known workaround available?

[nikhiljindal]

Yes this is a known issue. kubemci does not support XPN: #56.

I havent tried them myself, but 2 potential workarounds I can think of are:

1. Create the Firewall rule manually. Or
2. Try the patch: Networks #222

To create the firewall rule manually: Run the kubemci create command so that it creates all other resources. Then create the necessary firewall rule yourself. Ensure that you use the right project.
You can create the firewall rule manually by using cloud console, or by running the appropriate gcloud command: https://cloud.google.com/sdk/gcloud/reference/compute/firewall-rules/create.