High cardinality from metric "rest_client_request_latency_seconds_sum" #269
Description
Is this a BUG REPORT or FEATURE REQUEST?:
BUG
What happened:
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D", verb="GET"} | 0.026166687
-- | --
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D/eviction", verb="POST"} | 3.3592428640000005
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/api/v1/nodes", verb="GET"} | 15600.239082934992
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/api/v1/nodes/%7Bname%7D", verb="PATCH"} | 1.2786773510000002
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/api/v1/pods?fieldSelector=%7Bvalue%7D", verb="GET"} | 130.45066546799998
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/api/v1?timeout=32s", verb="GET"} | 0.009988171000000002
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/api?timeout=32s", verb="GET"} | 3.033722622
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/activemonitor.keikoproj.io/v1alpha1?timeout=32s", verb="GET"} | 0.002481853
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/addonmgr.keikoproj.io/v1alpha1?timeout=32s", verb="GET"} | 0.599171469
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/admiral.io/v1alpha1?timeout=32s", verb="GET"} | 0.002490315
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/admissionregistration.k8s.io/v1?timeout=32s", verb="GET"} | 0.002397739
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/admissionregistration.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.002413015
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/apiextensions.k8s.io/v1?timeout=32s", verb="GET"} | 1.098874921
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/apiextensions.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.002451179
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/apiregistration.k8s.io/v1?timeout=32s", verb="GET"} | 0.001346273
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/apiregistration.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.002670913
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/apps/v1/namespaces/%7Bnamespace%7D/daemonsets/%7Bname%7D", verb="GET"} | 61.567477207999985
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/apps/v1?timeout=32s", verb="GET"} | 1.298807949
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/argoproj.io/v1alpha1?timeout=32s", verb="GET"} | 0.499365619
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/authentication.istio.io/v1alpha1?timeout=32s", verb="GET"} | 0.399400264
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/authentication.k8s.io/v1?timeout=32s", verb="GET"} | 1.248660241
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/authentication.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.099412211
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/authorization.k8s.io/v1?timeout=32s", verb="GET"} | 0.002372744
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/authorization.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.299374821
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/autoscaling/v1?timeout=32s", verb="GET"} | 1.19871405
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/autoscaling/v2beta1?timeout=32s", verb="GET"} | 1.448753136
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/autoscaling/v2beta2?timeout=32s", verb="GET"} | 1.148790586
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/batch/v1?timeout=32s", verb="GET"} | 0.699155697
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/batch/v1beta1?timeout=32s", verb="GET"} | 0.998960333
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/certificates.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.002329583
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/config.istio.io/v1alpha2?timeout=32s", verb="GET"} | 0.19947485
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/coordination.k8s.io/v1?timeout=32s", verb="GET"} | 0.00255896
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/coordination.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.149677943
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/crd.k8s.amazonaws.com/v1alpha1?timeout=32s", verb="GET"} | 0.549296545
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/crd.projectcalico.org/v1?timeout=32s", verb="GET"} | 0.002270277
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/custom.metrics.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.012747798
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/discovery.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.349364577
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/dynatrace.com/v1alpha1?timeout=32s", verb="GET"} | 0.001302129
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/elbv2.k8s.aws/v1alpha1?timeout=32s", verb="GET"} | 0.049350027
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/elbv2.k8s.aws/v1beta1?timeout=32s", verb="GET"} | 0.002372812
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/events.k8s.io/v1beta1?timeout=32s", verb="GET"} | 1.348550366
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/extensions/v1beta1?timeout=32s", verb="GET"} | 0.002311416
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/hpa.orkaproj.io/v1alpha1?timeout=32s", verb="GET"} | 0.002432777
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/iammanager.keikoproj.io/v1alpha1?timeout=32s", verb="GET"} | 0.002576066
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/install.istio.io/v1alpha1?timeout=32s", verb="GET"} | 0.002513564
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/instancemgr.keikoproj.io/v1alpha1?timeout=32s", verb="GET"} | 0.649275446
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/litmuschaos.io/v1alpha1?timeout=32s", verb="GET"} | 0.749110148
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/metrics.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.951508069
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/monitoring.coreos.com/v1?timeout=32s", verb="GET"} | 0.002462623
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/monitoring.coreos.com/v1alpha1?timeout=32s", verb="GET"} | 1.049249703
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/networking.istio.io/v1alpha3?timeout=32s", verb="GET"} | 1.498888937
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/networking.istio.io/v1beta1?timeout=32s", verb="GET"} | 0.002323638
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/networking.k8s.io/v1?timeout=32s", verb="GET"} | 0.002411563
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/networking.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.249401094
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/node.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.849161453
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/policy/v1beta1?timeout=32s", verb="GET"} | 0.899041388
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/rbac.authorization.k8s.io/v1?timeout=32s", verb="GET"} | 0.002423248
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/rbac.authorization.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.002327157
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/rbac.istio.io/v1alpha1?timeout=32s", verb="GET"} | 0.002481619
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/scheduling.k8s.io/v1?timeout=32s", verb="GET"} | 0.00236329
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/scheduling.k8s.io/v1beta1?timeout=32s", verb="GET"} | 0.002318702
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/security.istio.io/v1beta1?timeout=32s", verb="GET"} | 0.002456169
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/storage.k8s.io/v1?timeout=32s", verb="GET"} | 0.002295293
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/storage.k8s.io/v1beta1?timeout=32s", verb="GET"} | 1.3985861019999999
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/upgrademgr.keikoproj.io/v1alpha1/namespaces/%7Bnamespace%7D/rollingupgrades/%7Bname%7D/status", verb="PUT"} | 91.31436567799945
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/upgrademgr.keikoproj.io/v1alpha1/rollingupgrades?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D", verb="GET"} | 0.01303986
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/upgrademgr.keikoproj.io/v1alpha1?timeout=32s", verb="GET"} | 0.449370049
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis/vpcresources.k8s.aws/v1beta1?timeout=32s", verb="GET"} | 0.799149227
rest_client_request_latency_seconds_sum{app="rolling-upgrade-controller", container="main", endpoint="iks-metric", instance="10.235.26.7:8080", job="upgrade-manager-metrics", namespace="addon-upgrademgr-ns", pod="rolling-upgrade-controller-5bb484bc58-hknvs", pod_template_hash="5bb484bc58", service="upgrade-manager-metrics", url="https://172.20.0.1:443/apis?timeout=32s", verb="GET"}
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:
- rolling-upgrade-controller version
- Kubernetes version :
$ kubectl version -o yaml
Other debugging information (if applicable):
- RollingUpgrade status:
$ kubectl describe rollingupgrade <rollingupgrade-name>
- controller logs:
$ kubectl logs <rolling-upgrade-controller pod>