cli is asking for oAuth

[soanni86]

Hello, i built cli v0.0.3 but it doesn't work with my perforator i deployed on k8s

export PERFORATOR_ENDPOINT="https://perforator.dev-corp.com"
export PERFORATOR_SECURE=true

./cli list services
16:21:00.908 ERROR cli/cli.go:66 Failed to initialize CLI {"error": "no OAuth token found"}
failed to initialize CLI: failed to initialize perforator client: no OAuth token found

~/src/perforator/perforator/bundle$ ./cli list profiles --selector '{cluster="cluster-***", timestamp>="now-30m"}'
16:32:26.750 ERROR cli/cli.go:66 Failed to initialize CLI {"error": "no OAuth token found"}
failed to initialize CLI: failed to initialize perforator client: no OAuth token found

if i set PEFORATOR_SECURE to false

export PERFORATOR_ENDPOINT="perforator.dev-corp.com:443"

./cli list profiles --selector '{cluster="cluster-***", timestamp>="now-30m"}'
16:35:23.612 WARN client client/client.go:109 No OAuth token found
16:35:23.613 INFO client client/client.go:214 List profiles {"filters": {"FromTS":"0001-01-01T00:00:00Z","ToTS":"2025-02-13T16:35:23.61369461Z","Selector":"{cluster=\"cluster-***\", timestamp>=\"now-30m\"}"}, "offset": 0, "limit": 500, "trace.id": "1a131d31472308982e63cdf5a3e69c1a", "span.id": "73c932ba3e193d33"}
rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"

/src/perforator/perforator/bundle$ ./cli list services
16:35:37.944 WARN client client/client.go:109 No OAuth token found
16:35:37.945 INFO client client/client.go:158 List services {"offset": 0, "limit": 500, "order": "services", "trace.id": "ab44f0388648e40959a0e57dde6d6c1c", "span.id": "02afebf44fcd3cf6"}
rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"

also please note that looks like PERFORATOR_URL env var is ignored https://perforator.tech/docs/en/howto/cli/install and PERFORATOR_EDPOINT is required instead.
Thanks

[MikailBag]

Hello.

This is bug in CLI, it will be fixed in the next release.

error reading server preface: http2: frame too large

This is expected: PERFORATOR_SECURE=false disabled TLS, so CLI tried to open plaintext connection to TLS-only endpoint.

like PERFORATOR_URL env var is ignored

It was a mistake in docs. I've fixed it.

Thank you for the report!

[MikailBag]

Release v0.0.4 is finally aviailable.

[soanni86]

thank you!

[soanni86]

still cant make it work. Am i doing smth wrong ?

export PERFORATOR_ENDPOINT=https://perforator.dev-corp.com

./cli list services
12:25:50.143 INFO client client/client.go:153 List services {"offset": 0, "limit": 500, "order": "services", "trace.id": "667fa0b9e41655dbc06061cdf2c1fd34", "span.id": "68103449066e9f43"}
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup tcp///perforator.dev-corp.com: unknown port"

export PERFORATOR_SECURE=true

./cli list services
12:26:19.372 WARN client client/client.go:104 No OAuth token found
12:26:19.373 INFO client client/client.go:153 List services {"offset": 0, "limit": 500, "order": "services", "trace.id": "d0efa3ddb5057e0930ce1431b9a1b8e4", "span.id": "682cfe83afbfbe33"}
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup tcp///perforator.dev-corp.com: unknown port"


./cli version
Git info:
    Commit: 28ce16ea4c5d3c3becab396019af53486127be11
    Branch: tags/v0.0.4
    Author: tangyatsu <[email protected]>
    Summary: [perforator][helm] rename host override field to make it clearer commit_hash:a5503422c0fbf677575951ae4909f4ef16b0502d

[soanni86]

looks like it doesnt like protocol in the env var and needs a port. could you please tell what is wrong here ?

export PERFORATOR_ENDPOINT=perforator.dev-corp.com:443

./cli list services
12:30:57.034 WARN client client/client.go:104 No OAuth token found
12:30:57.035 INFO client client/client.go:153 List services {"offset": 0, "limit": 500, "order": "services", "trace.id": "e710c04231f84a719d6cf9cffda3f0c2", "span.id": "2a40cd723a762a61"}
rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"

[MikailBag]

it doesnt like protocol in the env var

Yes. In fact, https is not right scheme anyway since CLI uses gRPC API.

needs a port

This is also true (but we are going to add defaults here).

what is wrong here

Which CA has issued certificate for the endpoint?

Looking at our code, it seems that we're doing pretty weird thing here. If you use internal CA whose root is installed system-wide, please try adding CERTIFI_USE_SYSTEM_CA=yes.

[soanni86]

i issued the cert using acme and used this cert/key in perforator installation. may be i miss smth ?

k -n perforator get secret perforator-storage-grpc-issued-tls -ojson | jq -r '.data."ca.crt"' | base64 -d | sudo tee /usr/local/share/ca-certificates/acme-ca.crt > /dev/null

sudo update-ca-certificates

1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

Adding debian:acme-ca.pem
done.
done.


ls -l /etc/ssl/certs/ | grep acme
lrwxrwxrwx 1 root root     44 Feb 17 13:43 acme-ca.pem -> /usr/local/share/ca-certificates/acme-ca.crt

openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt /usr/local/share/ca-certificates/acme-ca.crt
/usr/local/share/ca-certificates/acme-ca.crt: OK

now i added ca to the system and now seeing this

export PERFORATOR_ENDPOINT=perforator.dev-corp.com:443

./cli list services
13:40:07.577 INFO client client/client.go:153 List services {"offset": 0, "limit": 500, "order": "services", "trace.id": "c02262ba6a71979d47ea1710f721e852", "span.id": "4ccafd662442766f"}
rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"

export CERTIFI_USE_SYSTEM_CA=yes

./cli list services
13:40:34.439 INFO client client/client.go:153 List services {"offset": 0, "limit": 500, "order": "services", "trace.id": "6f9dd857535a2b938ce62cf3bcced9c7", "span.id": "ccced351561f2839"}
rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"

export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

./cli list services
13:40:42.990 INFO client client/client.go:153 List services {"offset": 0, "limit": 500, "order": "services", "trace.id": "83a07b597b36cdcd9b28cd0168f28aa8", "span.id": "e3b0fec71631dc63"}
rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"

thanks

[cezarnik]

Do you have https server on perforator.dev-corp.com:443? It may be the case that grpc client can't connect to https server. Similar error occured here open-telemetry/opentelemetry-helm-charts#646 (comment)

[soanni86]

yes it's a https server (actually http as ingress controller terminates tls and traffic goes to port 80 of the perforator proxy service), i'm just following the guide https://perforator.tech/docs/en/guides/cli/install trying to make cli work.
so should i have 2 ingresses both https and grpc(s) ? https will show me UI and grpc i should use for PERFORATOR_ENDPOINT env var like grpc://perforator.dev-corp.com:7618 ? thanks

[cezarnik]

Yes, just try with perforator.dev-corp.com:7618. perforator.dev-corp.com, in turn, should forward to grpc

[soanni86]

no it doesnt work

k -nperforator get ing perforator-ingress-grpc -oyaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/issuer: al-pki-issuer
    cert-manager.io/issuer-group: cas-issuer.jetstack.io
    cert-manager.io/issuer-kind: GoogleCASClusterIssuer
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
  labels:
    app.kubernetes.io/instance: perforator
    app.kubernetes.io/name: perforator
    argocd.argoproj.io/instance: perforator
  name: perforator-ingress-grpc
  namespace: perforator
spec:
  ingressClassName: ingress-controller-internal
  rules:
  - host: perforator-grpc.dev-corp.com
    http:
      paths:
      - backend:
          service:
            name: perforator-web-service
            port:
              number: 7618
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - perforator-grpc.dev-corp.com
    secretName: perforator-grpc-tls-secret

grpcurl perforator-grpc.dev-corp.com:443 list
NPerforator.NProto.MicroscopeService
NPerforator.NProto.Perforator
NPerforator.NProto.TaskService
grpc.health.v1.Health
grpc.reflection.v1.ServerReflection
grpc.reflection.v1alpha.ServerReflection

export PERFORATOR_ENDPOINT=perforator-grpc.dev-corp.com:443

cd src/perforator/perforator/bundle/

cli list services
11:33:06.375 INFO client client/client.go:153 List services {"offset": 0, "limit": 500, "order": "services", "trace.id": "4b8855a4d3c6c1b0fb3858845e5026d2", "span.id": "e3421412bed21fdf"}
rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"