Error while ensuring backend services

[hermanbanken]

When running kubemci I consistently get issues with the backend services:

kubemci create moria-api-global --ingress=.hue/mci_ingress.yaml --gcp-project=cl-dev --kubeconfig=clusters.yaml

The output is:

Ingress already exists and matches; moving on.
Ingress already exists and matches; moving on.
Ensuring health checks
Path for healthcheck is /
Ensuring health check for port: {SvcName:default/moria-api-global SvcPort:{Type:0 IntVal:80 StrVal:} NodePort:32080 Protocol:HTTP SvcTargetPort: NEGEnabled:false}
Health check mci1-hc-32080--moria-api-global exists already. Checking if it matches our desired health check
Desired health check exists already
Determining instance groups for cluster gke_cl-dev_europe-west1-b_hue-niverse
Determining instance groups for cluster gke_cl-dev_us-east1_hue-us-east1-dev-tmp
Fetching instance group: us-east1-b k8s-ig--cf453fa4ca0cd8bd
Fetched instance group: us-east1-b/k8s-ig--cf453fa4ca0cd8bd, got named ports: port: &{Name:port30301 Port:30301 ForceSendFields:[] NullFields:[]} port: &{Name:port32443 Port:32443 ForceSendFields:[] NullFields:[]} port: &{Name:port32500 Port:32500 ForceSendFields:[] NullFields:[]} port: &{Name:port31354 Port:31354 ForceSendFields:[] NullFields:[]} port: &{Name:port32080 Port:32080 ForceSendFields:[] NullFields:[]}
Ensuring backend services
Ensuring backend service for port: {default/moria-api-global {0 80 } 32080 HTTP  false}
Creating backend service mci1-be-32080--moria-api-global
Backend service mci1-be-32080--moria-api-global created successfully
Error ensuring backend service for port {default/moria-api-global {0 80 } 32080 HTTP  false}: Error googleapi: Error 404: The resource 'projects/cl-dev/global/backendServices/mci1-be-32080--moria-api-global' was not found, notFound in ensuring backend service for port {default/moria-api-global {0 80 } 32080 HTTP  false}. Continuing.
Error ensuring backend service for moria-api-global: 1 error occurred:

* Error googleapi: Error 404: The resource 'projects/cl-dev/global/backendServices/mci1-be-32080--moria-api-global' was not found, notFound in ensuring backend service for port {default/moria-api-global {0 80 } 32080 HTTP  false}
Ensuring url map
Error getting backend service moria-api-global: unexpected: No backend service found for service: moria-api-global, must have been an error in ensuring backend servicesError getting URL map from Ingress: 1 error occurred:

* unexpected: No backend service found for service: moria-api-global, must have been an error in ensuring backend services
Error ensuring urlmap for moria-api-global: error 1 error occurred:

* unexpected: No backend service found for service: moria-api-global, must have been an error in ensuring backend services in computing desired url map
Ensuring ssl cert
ssl cert mci1-ssl--moria-api-global exists already. Checking if it matches our desired ssl cert mci1-ssl--moria-api-global
Desired ssl cert exists already
Ensuring http target proxy.
Ensuring target https proxy
Creating target HTTPS proxy mci1-tps--moria-api-global
Creating target https proxy mci1-tps--moria-api-global
Error ensuring HTTPS target proxy: 1 error occurred:

* Error in ensuring https target proxy: googleapi: Error 400: Invalid value for field 'resource.urlMap': ''. A UrlMap must be specified for TargetHttpsProxy creation., invalid
Ensuring https forwarding rule
Creating forwarding rule mci1-fws--moria-api-global
Error ensuring https forwarding rule: googleapi: Error 400: Invalid value for field 'resource.target': ''. No target or backend service specified for forwarding rule., invalid
Ensuring firewall rule
Firewall rule mci1-fr--moria-api-global exists already. Checking if it matches our desired firewall rule
Desired firewall rule exists already.
Error: Error in creating load balancer: 4 errors occurred:

* Error ensuring backend service for moria-api-global: 1 error occurred:

* Error googleapi: Error 404: The resource 'projects/cl-dev/global/backendServices/mci1-be-32080--moria-api-global' was not found, notFound in ensuring backend service for port {default/moria-api-global {0 80 } 32080 HTTP  false}
* Error ensuring urlmap for moria-api-global: error 1 error occurred:

* unexpected: No backend service found for service: moria-api-global, must have been an error in ensuring backend services in computing desired url map
* Error ensuring HTTPS target proxy: 1 error occurred:

* Error in ensuring https target proxy: googleapi: Error 400: Invalid value for field 'resource.urlMap': ''. A UrlMap must be specified for TargetHttpsProxy creation., invalid
* Error ensuring https forwarding rule: googleapi: Error 400: Invalid value for field 'resource.target': ''. No target or backend service specified for forwarding rule., invalid

It complains about urlMap issues too, while the ingress.yaml has no complex urlMap:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: moria-api-global
  annotations:
    kubernetes.io/ingress.class: gce-multi-cluster
    kubernetes.io/ingress.global-static-ip-name: redacted
    kubernetes.io/ingress.allow-http: "false"
spec:
  tls:
  - secretName: redacted-tls
  backend:
    serviceName: moria-api-global
    servicePort: 80

$ kubemci version
Client version: 0.4.0

[nikhiljindal]

This is interesting:

Creating backend service mci1-be-32080--moria-api-global
Backend service mci1-be-32080--moria-api-global created successfully
Error ensuring backend service for port {default/moria-api-global {0 80 } 32080 HTTP  false}: Error googleapi: Error 404: The resource 'projects/cl-dev/global/backendServices/mci1-be-32080--moria-api-global' was not found, notFound in ensuring backend service for port {default/moria-api-global {0 80 } 32080 HTTP  false}. Continuing.

Looks like the CLI creates the backend service successfully but then receives a 404 when it tries to fetch it later. Can you check if the backend service mci1-be-32080--moria-api-global exists? Maybe its on a different network/subnetwork?

gcloud compute backend-services list or you can view list of all backend services in Cloud console

[hermanbanken]

Nope. I tried again today. The backends should be showing at:
https://console.cloud.google.com/net-services/loadbalancing/backends/list?project=cl-dev, right? No matter the network/subnetwork?

Note: we are on legacy networks, not on VPC networks. Part of creating this MCI is to migrate to VPC networks without downtime.

[nikhiljindal]

Yes that is correct, they should be showing up in the UI.

Have you tried the same in a different project which is not using legacy networks and see if creating a multicluster ingress works fine. That will confirm that the YAMLs are correct and it is actually some project specific setting that is the problem here.
Sorry I havent tried kubemci with a legacy network so can not be certain if it will or will not work.

[hermanbanken]

I did not have time to try that. I want to try next week.

However, I feel that there is something that we can improve in kubemci already here: kubemci reports that the backend is created, but it is not. Isn’t that check wrong then? Even if legacy networks is the culprit, the check is not working. If the check would be working it might be able to show the correct (helpful) error message.

[nikhiljindal]

Yes. I will wait for us to find a root cause to be sure.
If it is indeed the case that a backend service was not created, then we should not show "backend service created successfully" message.
If the backend service was created, but our call to fetch it later is not correct, then we need to fix that.

At a cursory glance, the code looks correct to me:

k8s-multicluster-ingress/app/kubemci/pkg/gcp/backendservice/backendservicesyncer.go

Line 222 in 415afa6

fmt.Println("Backend service", name, "created successfully")

[hermanbanken]

When I run kubemci in the same project with 2 VPC-network clusters it works. When I include the original legacy cluster it fails. With the same sequence "Backend service created successfully; Not found". It actually is not there, when checking with gcloud compute backend-services.

I wanted to dive into the bowels of kubemci to debug, but I was unable to install kubemci from source. Are there any quick guidelines for running this go app from source? It seems that the makefile is referencing a different directory than the files are actually in. And the package name of kubemci/cmd is cmd which is not main and therefore not runnable.

[hermanbanken]

Ah, I found out. Apparently I had no GOPATH (using go 11.5 with modules mainly)...

brew install glide
go get github.com/GoogleCloudPlatform/k8s-multicluster-ingress
export GOPATH=~/go
cd $GOPATH/src/github.com/GoogleCloudPlatform/k8s-multicluster-ingress
make build

[hermanbanken]

Now the build fails:

go build -a -installsuffix cgo /Users/hbanken/go//src/github.com/GoogleCloudPlatform/k8s-multicluster-ingress/cmd/kubemci/kubemci.go
# github.com/GoogleCloudPlatform/k8s-multicluster-ingress/app/kubemci/pkg/kubeutils
app/kubemci/pkg/kubeutils/utils.go:140:27: invalid operation: port.SvcPort.Type == "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/apimachinery/pkg/util/intstr".Int (mismatched types "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/intstr".Type and "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/apimachinery/pkg/util/intstr".Type)
app/kubemci/pkg/kubeutils/utils.go:141:25: invalid operation: port.SvcPort.Type == "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/apimachinery/pkg/util/intstr".String (mismatched types "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/intstr".Type and "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/apimachinery/pkg/util/intstr".Type)
app/kubemci/pkg/kubeutils/utils.go:229:46: cannot use svc (type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/api/core/v1".Service) as type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/k8s.io/api/core/v1".Service in argument to annotations.FromService
app/kubemci/pkg/kubeutils/utils.go:265:3: cannot use "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/apimachinery/pkg/types".NamespacedName literal (type "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/apimachinery/pkg/types".NamespacedName) as type "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/types".NamespacedName in field value
app/kubemci/pkg/kubeutils/utils.go:266:3: cannot use be.ServicePort (type "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/apimachinery/pkg/util/intstr".IntOrString) as type "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/intstr".IntOrString in field value
# github.com/GoogleCloudPlatform/k8s-multicluster-ingress/app/kubemci/pkg/gcp/forwardingrule
app/kubemci/pkg/gcp/forwardingrule/forwardingrulesyncer.go:89:27: cannot use existingFR (type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/google.golang.org/api/compute/v1".ForwardingRule) as type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".ForwardingRule in argument to forwardingRuleMatches
app/kubemci/pkg/gcp/forwardingrule/forwardingrulesyncer.go:95:33: cannot use existingFR (type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/google.golang.org/api/compute/v1".ForwardingRule) as type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".ForwardingRule in argument to s.updateForwardingRule
app/kubemci/pkg/gcp/forwardingrule/forwardingrulesyncer.go:148:19: cannot use httpFr (type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/google.golang.org/api/compute/v1".ForwardingRule) as type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".ForwardingRule in argument to getStatus
app/kubemci/pkg/gcp/forwardingrule/forwardingrulesyncer.go:155:19: cannot use httpsFr (type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/google.golang.org/api/compute/v1".ForwardingRule) as type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".ForwardingRule in argument to getStatus
app/kubemci/pkg/gcp/forwardingrule/forwardingrulesyncer.go:179:16: cannot assign []*"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/google.golang.org/api/compute/v1".ForwardingRule to rules (type []*"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".ForwardingRule) in multiple assignment
app/kubemci/pkg/gcp/forwardingrule/forwardingrulesyncer.go:255:58: cannot use existingFR (type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/google.golang.org/api/compute/v1".ForwardingRule) as type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".ForwardingRule in argument to s.desiredForwardingRuleWithoutClusters
app/kubemci/pkg/gcp/forwardingrule/forwardingrulesyncer.go:261:31: cannot use existingFR (type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/google.golang.org/api/compute/v1".ForwardingRule) as type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".ForwardingRule in argument to s.updateForwardingRule
app/kubemci/pkg/gcp/forwardingrule/forwardingrulesyncer.go:293:41: cannot use desiredFR (type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".ForwardingRule) as type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/google.golang.org/api/compute/v1".ForwardingRule in argument to s.frp.CreateGlobalForwardingRule
# github.com/GoogleCloudPlatform/k8s-multicluster-ingress/app/kubemci/pkg/gcp/targetproxy
app/kubemci/pkg/gcp/targetproxy/targetproxysyncer.go:127:48: cannot use *existingHTTPProxy (type "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/google.golang.org/api/compute/v1".TargetHttpProxy) as type "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".TargetHttpProxy in argument to targetHTTPProxyMatches
app/kubemci/pkg/gcp/targetproxy/targetproxysyncer.go:155:42: cannot use desiredHTTPProxy (type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".TargetHttpProxy) as type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/google.golang.org/api/compute/v1".TargetHttpProxy in argument to s.tpp.SetUrlMapForTargetHttpProxy
app/kubemci/pkg/gcp/targetproxy/targetproxysyncer.go:155:42: cannot use urlMap (type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".UrlMap) as type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/google.golang.org/api/compute/v1".UrlMap in argument to s.tpp.SetUrlMapForTargetHttpProxy
app/kubemci/pkg/gcp/targetproxy/targetproxysyncer.go:173:36: cannot use desiredHTTPProxy (type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".TargetHttpProxy) as type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/google.golang.org/api/compute/v1".TargetHttpProxy in argument to s.tpp.CreateTargetHttpProxy
app/kubemci/pkg/gcp/targetproxy/targetproxysyncer.go:224:50: cannot use *existingHTTPSProxy (type "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/google.golang.org/api/compute/v1".TargetHttpsProxy) as type "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".TargetHttpsProxy in argument to targetHTTPSProxyMatches
app/kubemci/pkg/gcp/targetproxy/targetproxysyncer.go:251:43: cannot use desiredHTTPSProxy (type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".TargetHttpsProxy) as type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/google.golang.org/api/compute/v1".TargetHttpsProxy in argument to s.tpp.SetUrlMapForTargetHttpsProxy
app/kubemci/pkg/gcp/targetproxy/targetproxysyncer.go:251:79: cannot use &"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".UrlMap literal (type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".UrlMap) as type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/google.golang.org/api/compute/v1".UrlMap in argument to s.tpp.SetUrlMapForTargetHttpsProxy
app/kubemci/pkg/gcp/targetproxy/targetproxysyncer.go:267:37: cannot use desiredHTTPSProxy (type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".TargetHttpsProxy) as type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/google.golang.org/api/compute/v1".TargetHttpsProxy in argument to s.tpp.CreateTargetHttpsProxy
# github.com/GoogleCloudPlatform/k8s-multicluster-ingress/app/kubemci/pkg/gcp/sslcert
app/kubemci/pkg/gcp/sslcert/sslcertsyncer.go:58:40: cannot use ing (type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/api/extensions/v1beta1".Ingress) as type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/k8s.io/api/extensions/v1beta1".Ingress in argument to annotations.FromIngress
app/kubemci/pkg/gcp/sslcert/sslcertsyncer.go:66:43: cannot use ing (type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/api/extensions/v1beta1".Ingress) as type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/k8s.io/api/extensions/v1beta1".Ingress in argument to annotations.FromIngress
app/kubemci/pkg/gcp/sslcert/sslcertsyncer.go:93:35: cannot use *existingCert (type "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/google.golang.org/api/compute/v1".SslCertificate) as type "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".SslCertificate in argument to sslCertMatches
app/kubemci/pkg/gcp/sslcert/sslcertsyncer.go:145:37: cannot use desiredCert (type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".SslCertificate) as type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/google.golang.org/api/compute/v1".SslCertificate in argument to s.scp.CreateSslCertificate
app/kubemci/pkg/gcp/sslcert/sslcertsyncer.go:161:38: cannot use desiredCert (type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/google.golang.org/api/compute/v1".SslCertificate) as type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/google.golang.org/api/compute/v1".SslCertificate in argument to s.scp.CreateSslCertificate
app/kubemci/pkg/gcp/sslcert/sslcertsyncer.go:188:46: cannot use client (type "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/client-go/kubernetes".Interface) as type "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/k8s.io/client-go/kubernetes".Interface in field value:
	"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/client-go/kubernetes".Interface does not implement "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/k8s.io/client-go/kubernetes".Interface (wrong type for Admissionregistration method)
		have Admissionregistration() "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/client-go/kubernetes/typed/admissionregistration/v1beta1".AdmissionregistrationV1beta1Interface
		want Admissionregistration() "github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/k8s.io/client-go/kubernetes/typed/admissionregistration/v1beta1".AdmissionregistrationV1beta1Interface
app/kubemci/pkg/gcp/sslcert/sslcertsyncer.go:189:29: cannot use ing (type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/api/extensions/v1beta1".Ingress) as type *"github.com/GoogleCloudPlatform/k8s-multicluster-ingress/vendor/k8s.io/ingress-gce/vendor/k8s.io/api/extensions/v1beta1".Ingress in argument to tlsLoader.Load
make: *** [build] Error 2

[hermanbanken]

Fixed above build error by switching to Go Modules (#219).

[hermanbanken]

The reason the error during the backendservice creation is not detected is due to a bug in op.go which is resolved in the latest version, compare function below with the link above:

https://github.com/GoogleCloudPlatform/k8s-cloud-provider/blob/master/pkg/cloud/op.go#L83-L90

I think we should update to a more up to date version of kubernetes/kubernetes, but that might hurt a bit...

[hermanbanken]

Finally, I was able to extract the underlying GlobalOperation error:

Code:INCOMPATIBLE_BACKEND_SERVICES Location: Message:Validation failed for instance group 'projects/cl-dev/zones/europe-west1-b/instanceGroups/k8s-ig':
backend services 'projects/cl-dev/global/backendServices/k8s-be-30772' and 'projects/cl-dev/global/backendServices/mci1-be-32080--mci-global' point to the same instance group but the backends have incompatible balancing_mode. Values should be the same. ForceSendFields:[] NullFields:[]} &{INCOMPATIBLE_BACKEND_SERVICES  Validation failed for instance group 'projects/cl-dev/zones/europe-west1-b/instanceGroups/k8s-ig': backend services 'projects/cl-dev/global/backendServices/k8s-be-30772' and 'projects/cl-dev/global/backendServices/mci1-be-32080--mci-global' point to the same instance group but the backends have incompatible balancing_mode. Values should be the same.

This is the exact same as what I see when trying to create a Load Balancer from the console: in that case the option for the Rate-based balancing mode is greyed out for this (old cluster) instance group. In the console it is not a problem to pick a different balancing mode (Utilization) for this cluster and the 'normal' Rate based balancing mode for new clusters. However, kubemci explicitly chose the Rate balancing mode which is incompatible with our backend instance group (because it has existing Utilization balancing).

[hermanbanken]

Given a compute command like this:

gcloud compute backend-services list --flatten="backends[]" --format "csv(name,backends.group,backends.balancingMode)"

I can easily list the current backend-services and which instance groups they are using with which balancing mode. Given that list, it would be trivial to pick the 'correct' (eg. already used) balancing mode.

[hermanbanken]

This also means that reproduction is much easier to accomplish than I thought: just create 2 instance groups and add 1 backend-service with Utilization to one of those, then try to setup kubemci for them. That should fail too.

[nikhiljindal]

Thanks a lot Herman for this great debugging!

I understand this issue now.

Looks like you have 2 different backend services with different load balancing modes, pointing to the same instance group. This is not allowed. All backend services pointing to the same instance group should have the same balancing mode.

Most users do not run into this problem since the in-cluster k8s ingress-gce controller and kubemci CLI both create backend services with the same balancing mode (RATE). Since these 2 are mostly the only ones creating backend services, there is never a conflict.

In your case, for some reason the in-cluster k8s ingress-gce controller is creating backend services with a different load balancing mode (UTILIZATION) than kubemci (RATE).

Why could this be happening and ways to fix:

• Why: You are running an older version of ingress-gce controller in your cluster. How to fix: Upgrade your cluster.
• Why: You were running an older version, you have upgraded now, but the ingress-gce controller is still creating backend services with old balancing mode. Backstory: ingress-gce controller moved from using UTILIZATION to RATE as load balancing mode. But for existing clusters with existing ingresses, it still uses UTILIZATION mode to not break existing backend services. Ways to fix: Delete all existing ingresses in the cluster. After that when you create an ingress in that cluster, backend service should be created with load balancing mode RATE - which will work with kubemci CLI.

I understand that deleting all existing ingresses might not be a viable solution if this is a production cluster, but thats what we have right now.

Hope this helps

[hermanbanken]

Yes, that is how I understand it as well. Removing the ingresses is not viable as we need this in our production cluster too. We are utilizing kubemci as part of our cluster migration strategy 😉 so áfterwards we can drop the whole cluster, but for now we need a workaround.

I want to take it one step further by making the balancing mode a parameter in (a fork of) kubemci. And possibly even be smart about it and detect the reuse of an instance group and reuse its current balancing mode. Like I said gcloud compute backend-services list --flatten="backends[]" --format "csv(name,backends.group,backends.balancingMode)" shows you the current mode for the instance group. I'll try to create a patch that does exactly that via Golang.