[BUG] CumulusCommunity/cumulus-vx:5.9.0 fails to deploy on Libvirt

[jbemmel]

5.10.0 (default image) works, but 5.9.0 fails with an error:

The configured shell (config.ssh.shell) is invalid and unable
to properly execute commands. The most common cause for this is
using a shell that is unavailable on the system. Please verify
you're using the full path to the shell and that the shell is
executable by the SSH user.

Not sure which variable this is, or which binary is missing

[ipspace]

According to https://developer.hashicorp.com/vagrant/docs/vagrantfile/ssh_settings#config-ssh-shell, the default shell Vagrant tries to use is bash, so it looks like they forgot to add it to the Vagrant box.

There is no way to change that to sh within netlab, you'd have to use a custom Vagrant template. See https://blog.ipspace.net/2022/06/netsim-custom-vagrant-boxes/ for details. I don't think the template search path has changed since that blog post was written, if it did please let me know (or submit a PR against https://github.com/ipspace/blog 😎)

[ipspace]

There's not much worth doing here (from our side). Changing default CL 5.x bash to sh just because NVIDIA messed up one release makes little sense.

[ipspace]

The default password expiration policy kicks in after a while and a Vagrant box becomes unusable. 5.10.0 stopped working (for me) today.

Here's the relevant part of vagrant up --debug:

DEBUG ssh: Checking key permissions: /home/pipi/.vagrant.d/insecure_private_keys/vagrant.key.ed25519
DEBUG ssh: Checking key permissions: /home/pipi/.vagrant.d/insecure_private_keys/vagrant.key.rsa
 INFO interface: detail: SSH address: 192.168.121.250:22
 INFO interface: detail:     dut: SSH address: 192.168.121.250:22
    dut: SSH address: 192.168.121.250:22
 INFO interface: detail: SSH username: vagrant
 INFO interface: detail:     dut: SSH username: vagrant
    dut: SSH username: vagrant
 INFO interface: detail: SSH auth method: private key
 INFO interface: detail:     dut: SSH auth method: private key
    dut: SSH auth method: private key
DEBUG ssh: Checking key permissions: /home/pipi/.vagrant.d/insecure_private_keys/vagrant.key.ed25519
DEBUG ssh: Checking key permissions: /home/pipi/.vagrant.d/insecure_private_keys/vagrant.key.rsa
 INFO ssh: Attempting SSH connection...
 INFO ssh: Attempting to connect to SSH...
 INFO ssh:   - Host: 192.168.121.250
 INFO ssh:   - Port: 22
 INFO ssh:   - Username: vagrant
 INFO ssh:   - Password? false
 INFO ssh:   - Key Path: ["/home/pipi/.vagrant.d/insecure_private_keys/vagrant.key.ed25519", "/home/pipi/.vagrant.d/insecure_private_keys/vagrant.key.rsa"]
DEBUG ssh:   - connect_opts: {:auth_methods=>["none", "hostbased", "publickey"], :config=>false, :forward_agent=>false, :send_env=>false, :keys_only=>true, :verify_host_key=>:never, :password=>nil, :port=>22, :timeout=>15, :user_known_hosts_file=>[], :verbose=>:debug, :logger=>#<Logger:0x00007f131c0b6310 @level=0, @progname=nil, @default_formatter=#<Logger::Formatter:0x00007f13194c8668 @datetime_format=nil>, @formatter=nil, @logdev=#<Logger::LogDevice:0x00007f131c0b62c0 @shift_period_suffix=nil, @shift_size=nil, @shift_age=nil, @filename=nil, @dev=#<StringIO:0x00007f13194c8758>, @binmode=false, @mon_data=#<Monitor:0x00007f13194c8550>, @mon_data_owner_object_id=2640>, @level_override={}>, :keys=>["/home/pipi/.vagrant.d/insecure_private_keys/vagrant.key.ed25519", "/home/pipi/.vagrant.d/insecure_private_keys/vagrant.key.rsa"], :remote_user=>"vagrant", :keepalive=>true, :keepalive_interval=>5}
DEBUG ssh: == Net-SSH connection debug-level log START ==
DEBUG ssh: D, [2025-02-17T14:43:28.357640 #5967] DEBUG -- net.ssh.transport.session[a64]: establishing connection to 192.168.121.250:22
D, [2025-02-17T14:43:28.358214 #5967] DEBUG -- net.ssh.transport.session[a64]: connection established
I, [2025-02-17T14:43:28.358257 #5967]  INFO -- net.ssh.transport.server_version[a78]: negotiating protocol version
D, [2025-02-17T14:43:28.358264 #5967] DEBUG -- net.ssh.transport.server_version[a78]: local is `SSH-2.0-Ruby/Net::SSH_7.3.0 x86_64-linux'
D, [2025-02-17T14:43:28.365658 #5967] DEBUG -- net.ssh.transport.server_version[a78]: remote is `SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u3'
I, [2025-02-17T14:43:28.365913 #5967]  INFO -- net.ssh.transport.algorithms[a8c]: sending KEXINIT
D, [2025-02-17T14:43:28.366039 #5967] DEBUG -- socket[aa0]: queueing packet nr 0 type 20 len 940
D, [2025-02-17T14:43:28.366084 #5967] DEBUG -- socket[aa0]: sent 944 bytes
D, [2025-02-17T14:43:28.366576 #5967] DEBUG -- socket[aa0]: read 752 bytes
D, [2025-02-17T14:43:28.366610 #5967] DEBUG -- socket[aa0]: received packet nr 0 type 20 len 748
I, [2025-02-17T14:43:28.366632 #5967]  INFO -- net.ssh.transport.algorithms[a8c]: got KEXINIT from server
I, [2025-02-17T14:43:28.366662 #5967]  INFO -- net.ssh.transport.algorithms[a8c]: negotiating algorithms
D, [2025-02-17T14:43:28.366709 #5967] DEBUG -- net.ssh.transport.algorithms[a8c]: negotiated:
* kex: ecdh-sha2-nistp521
* host_key: ecdsa-sha2-nistp256
* encryption_server: aes256-ctr
* encryption_client: aes256-ctr
* hmac_client: [email protected]
* hmac_server: [email protected]
* compression_client: none
* compression_server: none
* language_client:
* language_server:
D, [2025-02-17T14:43:28.366727 #5967] DEBUG -- net.ssh.transport.algorithms[a8c]: exchanging keys
D, [2025-02-17T14:43:28.369492 #5967] DEBUG -- socket[aa0]: queueing packet nr 1 type 30 len 148
D, [2025-02-17T14:43:28.369527 #5967] DEBUG -- socket[aa0]: sent 152 bytes
D, [2025-02-17T14:43:28.370551 #5967] DEBUG -- socket[aa0]: read 376 bytes
D, [2025-02-17T14:43:28.370590 #5967] DEBUG -- socket[aa0]: received packet nr 1 type 31 len 356
D, [2025-02-17T14:43:28.375611 #5967] DEBUG -- socket[aa0]: queueing packet nr 2 type 21 len 20
D, [2025-02-17T14:43:28.375646 #5967] DEBUG -- socket[aa0]: sent 24 bytes
D, [2025-02-17T14:43:28.375668 #5967] DEBUG -- socket[aa0]: received packet nr 2 type 21 len 12
D, [2025-02-17T14:43:28.375841 #5967] DEBUG -- net.ssh.authentication.session[ab4]: beginning authentication of `vagrant'
D, [2025-02-17T14:43:28.375864 #5967] DEBUG -- socket[aa0]: using encrypt-then-mac
D, [2025-02-17T14:43:28.375984 #5967] DEBUG -- socket[aa0]: queueing packet nr 3 type 5 len 32
D, [2025-02-17T14:43:28.375993 #5967] DEBUG -- socket[aa0]: sent 100 bytes
D, [2025-02-17T14:43:28.419166 #5967] DEBUG -- socket[aa0]: read 100 bytes
D, [2025-02-17T14:43:28.419346 #5967] DEBUG -- socket[aa0]: received packet nr 3 type 6 len 32
D, [2025-02-17T14:43:28.419419 #5967] DEBUG -- net.ssh.authentication.session[ab4]: trying none
D, [2025-02-17T14:43:28.419475 #5967] DEBUG -- socket[aa0]: using encrypt-then-mac
D, [2025-02-17T14:43:28.419515 #5967] DEBUG -- socket[aa0]: queueing packet nr 4 type 50 len 48
D, [2025-02-17T14:43:28.419574 #5967] DEBUG -- socket[aa0]: sent 116 bytes
D, [2025-02-17T14:43:28.433991 #5967] DEBUG -- socket[aa0]: read 232 bytes
D, [2025-02-17T14:43:28.434181 #5967] DEBUG -- socket[aa0]: received packet nr 4 type 53 len 64
I, [2025-02-17T14:43:28.434217 #5967]  INFO -- net.ssh.authentication.session[ab4]: Welcome to NVIDIA Cumulus (R) Linux (R)

D, [2025-02-17T14:43:28.434262 #5967] DEBUG -- socket[aa0]: received packet nr 5 type 51 len 32
D, [2025-02-17T14:43:28.434283 #5967] DEBUG -- net.ssh.authentication.session[ab4]: allowed methods: publickey,password
D, [2025-02-17T14:43:28.434312 #5967] DEBUG -- net.ssh.authentication.methods.none[ac8]: none failed
D, [2025-02-17T14:43:28.434339 #5967] DEBUG -- net.ssh.authentication.session[ab4]: trying publickey
D, [2025-02-17T14:43:28.437152 #5967] DEBUG -- net.ssh.authentication.agent[adc]: connecting to ssh-agent
E, [2025-02-17T14:43:28.437190 #5967] ERROR -- net.ssh.authentication.agent[adc]: could not connect to ssh-agent: Agent not configured
D, [2025-02-17T14:43:28.437281 #5967] DEBUG -- net.ssh.authentication.methods.publickey[b04]: trying publickey (c8:ac:1c:f8:c5:f9:7f:fc:86:97:a4:96:a3:a1:2e:d8) alg ssh-ed25519
D, [2025-02-17T14:43:28.437332 #5967] DEBUG -- socket[aa0]: using encrypt-then-mac
D, [2025-02-17T14:43:28.437385 #5967] DEBUG -- socket[aa0]: queueing packet nr 5 type 50 len 128
D, [2025-02-17T14:43:28.437430 #5967] DEBUG -- socket[aa0]: sent 196 bytes
D, [2025-02-17T14:43:28.444800 #5967] DEBUG -- socket[aa0]: read 100 bytes
D, [2025-02-17T14:43:28.444995 #5967] DEBUG -- socket[aa0]: received packet nr 6 type 51 len 32
D, [2025-02-17T14:43:28.445035 #5967] DEBUG -- net.ssh.authentication.session[ab4]: allowed methods: publickey,password
D, [2025-02-17T14:43:28.445140 #5967] DEBUG -- net.ssh.authentication.methods.publickey[b04]: trying publickey (dd:3b:b8:2e:85:04:06:e9:ab:ff:a8:0a:c0:04:6e:d6) alg rsa-sha2-256
D, [2025-02-17T14:43:28.445174 #5967] DEBUG -- socket[aa0]: using encrypt-then-mac
D, [2025-02-17T14:43:28.445211 #5967] DEBUG -- socket[aa0]: queueing packet nr 6 type 50 len 352
D, [2025-02-17T14:43:28.445249 #5967] DEBUG -- socket[aa0]: sent 420 bytes
D, [2025-02-17T14:43:28.445699 #5967] DEBUG -- socket[aa0]: read 372 bytes
D, [2025-02-17T14:43:28.445761 #5967] DEBUG -- socket[aa0]: received packet nr 7 type 60 len 304
D, [2025-02-17T14:43:28.446933 #5967] DEBUG -- socket[aa0]: using encrypt-then-mac
D, [2025-02-17T14:43:28.446972 #5967] DEBUG -- socket[aa0]: queueing packet nr 7 type 50 len 640
D, [2025-02-17T14:43:28.446992 #5967] DEBUG -- socket[aa0]: sent 708 bytes
D, [2025-02-17T14:43:28.460150 #5967] DEBUG -- socket[aa0]: read 164 bytes
D, [2025-02-17T14:43:28.461509 #5967] DEBUG -- socket[aa0]: received packet nr 8 type 53 len 96
I, [2025-02-17T14:43:28.461550 #5967]  INFO -- net.ssh.authentication.session[ab4]: You are required to change your password immediately (password expired).

D, [2025-02-17T14:43:28.500663 #5967] DEBUG -- socket[aa0]: read 84 bytes
D, [2025-02-17T14:43:28.500839 #5967] DEBUG -- socket[aa0]: received packet nr 9 type 52 len 16
D, [2025-02-17T14:43:28.500875 #5967] DEBUG -- net.ssh.authentication.methods.publickey[b04]: publickey succeeded (dd:3b:b8:2e:85:04:06:e9:ab:ff:a8:0a:c0:04:6e:d6)

DEBUG ssh: == Net-SSH connection debug-level log END ==
DEBUG ssh: Checking whether SSH is ready...
DEBUG ssh: Re-using SSH connection.
 INFO ssh: SSH is ready!
DEBUG ssh: Re-using SSH connection.
 INFO ssh: Execute:  (sudo=false)
DEBUG ssh: stderr: WARNING: Your password has expired.
Password change required but no TTY available.

DEBUG ssh: Exit status: 1
 INFO ssh: SSH not ready: #<Vagrant::Errors::SSHInvalidShell:"The configured shell (config.ssh.shell) is invalid and unable\nto properly execute commands. The most common cause for this is\nusing a shell that is unavailable on the system. Please verify\nyou're using the full path to the shell and that the shell is\nexecutable by the SSH user.">
ERROR warden: Error occurred: The configured shell (config.ssh.shell) is invalid and unable
to properly execute commands. The most common cause for this is
using a shell that is unavailable on the system. Please verify
you're using the full path to the shell and that the shell is
executable by the SSH user.
 INFO warden: Beginning recovery process...