[ci.jenkins.io] Container agents in a degraded state

[basil]

Service(s)

ci.jenkins.io

Summary

I've now run 3 core PR builds in a row against a Linux JDK 8 Kubernetes container (label: maven) and all three have failed with:

Cannot contact jnlp-maven-8-45549: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@3ba7fb1d:JNLP4-connect connection from ec2-3-143-84-83.us-east-2.compute.amazonaws.com/3.143.84.83:64651": Remote call on JNLP4-connect connection from ec2-3-143-84-83.us-east-2.compute.amazonaws.com/3.143.84.83:64651 failed. The channel is closing down or has closed down`

• https://ci.jenkins.io/job/Core/job/jenkins/job/PR-6499/2/console
• https://ci.jenkins.io/job/Core/job/jenkins/job/PR-6499/3/console
• https://ci.jenkins.io/job/Core/job/jenkins/job/PR-6499/4/console

https://status.jenkins.io/ shows no issues.

Clearly the cluster is in a severely degraded state and monitoring is insufficient.

Reproduction steps

Try to run a core PR build. Observe that the build fails within a minute rather than compiling and testing the code.

[basil]

The acute problem I was facing seems to have now cleared up:

• https://ci.jenkins.io/job/Core/job/jenkins/job/PR-6499/5/console

This ticket can be closed, since the short-term pain is gone. I am still trying to think of the best way to file a ticket to track the long-term pain of agent instability we are seeing.

[dduportal]

Some elements to add:

• There is a lot of jnlp agent that are in "idle" state. It's weird, no idea why/what/how.
• I have zero idea how to monitor such behaviors. Maybe collecting all logs and have a sytem grepping it for such errors? Monitoring Jenkins have always been a pain, any idea welcome to be honest.
• DockerHub suffered issues ~2 hours ago, it was timeout to pull images: might be related.

[dduportal]

https://status.jenkins.io/ shows no issues.

It's basically a static website: there isn't really automation on this. Do not hesitate to open a PR on it if you see such pain in agents allocation: that would help other users a lot.

[dduportal]

For info, I've triggered a "reload from disk". It usually trigger whatever "agent garbage collector" process in Jenkins usually.

[dduportal]

Currently checking the logs to see what is going wrong

[dduportal]

A lot of errors

io.jenkins.plugins.casc.ConfiguratorException: Invalid configuration elements for type class org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate : env.
Available attributes : alwaysPullImage, args, command, envVars, image, livenessProbe, name, ports, privileged, resourceLimitCpu, resourceLimitEphemeralStorage, resourceLimitMemory, resourceRequestCpu, resourceRequestEphemeralStorage, resourceRequestMemory, runAsGroup, runAsUser, shell, ttyEnabled, workingDir

Sounds related to jenkins-infra/jenkins-infra#2128.

[dduportal]

Stopped puppet on the VM to validate the correct casc syntax.

[async ping @lemeurherve @smerle33 : it looks like that your work on doks is also generating a lot of errors. It will be deleted by the operation I'm doing right now]

[basil]

[…] Invalid configuration elements […] : env. Available attributes : […] envVars […]

=> jenkins-infra/jenkins-infra#2129

[dduportal]

Many thanks @basil for reporting the issue + sending the PR! I've added a suggestion that I just tested in the production.