After update from 9.5p1 to version 9.8p1 we started having Connections aborted.

[JoaoPereira1994]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest version
• Search the existing issues.

Steps to reproduce

After we updated to version 9.8p1 we started to have aborted connections in a intermittent fashion.

We found out so far that the issue occurs when we hace a couple of competing connections from the same client doing for example some list and gets files from our Server.

Expected behavior

We espect to not have this issue since we are using Openssh for Windows since version 8.1p1 and never had issues with competing connections.

Actual behavior

When we have competing connections the server starts to abort connections for no apparent reason.

Error details

sshd-session.log

6656 2024-11-05 08:22:12.523 debug1: SSH2_MSG_KEXINIT sent [preauth]
6656 2024-11-05 08:22:12.523 debug3: send - WSASend() ERROR:10053, io:0000024654A58EB0 [preauth]
6656 2024-11-05 08:22:12.523 ssh_dispatch_run_fatal: Connection from 10.1.1.94 port 44654: Connection aborted [preauth]
6656 2024-11-05 08:22:12.523 debug1: do_cleanup [preauth]

----------------------------------------//////////------------------------------------------------------
sshd.log

3876 2024-11-05 08:22:12.523 debug3: write - ERROR:109 on prior unblocking write, io:000002BD25407AD0
3876 2024-11-05 08:22:12.523 error: ssh_msg_send: write: Broken pipe
3876 2024-11-05 08:22:12.523 error: send_rexec_state: ssh_msg_send failed
3876 2024-11-05 08:22:12.523 debug3: send_rexec_state: done
3876 2024-11-05 08:22:12.523 debug1: child_reap: preauth child 1524 for connection from 10.11.7.68 to 10.1.1.94 exited with status 255
3876 2024-11-05 08:22:12.539 debug1: child_reap: preauth child 11676 for connection from 10.11.7.68 to 10.1.1.94 exited with status 255 (early)
3876 2024-11-05 08:22:12.539 debug1: child_reap: preauth child 6656 for connection from 10.11.7.68 to 10.1.1.94 exited with status 255 (early)
--------------------------------------/////////---------------------------------------------------------
sshd-session.log

8372 2024-11-05 08:52:23.252 debug3: send packet: type 20 [preauth]
8372 2024-11-05 08:52:23.252 debug1: SSH2_MSG_KEXINIT sent [preauth]
8372 2024-11-05 08:52:23.252 debug3: send - WSASend() ERROR:10053, io:00000203D2326E30 [preauth]
8372 2024-11-05 08:52:23.252 ssh_dispatch_run_fatal: Connection from 10.1.1.94 port 50447: Connection aborted [preauth]
8372 2024-11-05 08:52:23.252 debug1: do_cleanup [preauth]

--------------------------------------/////////---------------------------------------------------------
sshd.log

3876 2024-11-05 08:52:23.267 debug3: write - ERROR from cb:109, io:000002BD25408220
3876 2024-11-05 08:52:23.267 error: ssh_msg_send: write: Destination address required
3876 2024-11-05 08:52:23.267 error: send_rexec_state: ssh_msg_send failed
3876 2024-11-05 08:52:23.267 debug3: send_rexec_state: done
3876 2024-11-05 08:52:23.267 debug1: child_reap: preauth child 8372 for connection from 10.11.7.68 to 10.1.1.94 exited with status 255 (early)
3876 2024-11-05 08:52:23.267 debug1: child_reap: preauth child 3984 for connection from 10.11.7.68 to 10.1.1.94 exited with status 255

Environment data

PS C:\> $PSVersionTable 
Name                           Value
----                           -----
PSVersion                      5.1.14393.7254
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.14393.7254
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Version

OpenSSH_for_Windows_9.8p1 Win32-OpenSSH-GitHub, LibreSSL 3.9.2

Visuals

No response

[StevenBucher98]

This may be related to some upstream changes to help prevent DDOS from the client. Can you try to adding known IPs to this directive in SSHD to see if it gets more reliable? This will help us know if this has to do with the upstream changes. https://man.openbsd.org/sshd_config#PerSourcePenaltyExemptList

[JoaoPereira1994]

@StevenBucher98 -

We tried your suggestion and we got the following result we keep having the same errors:

11744 2024-11-11 19:43:02.807 debug3: write - ERROR from cb:109, io:0000020447ED48C0
11744 2024-11-11 19:43:02.807 error: ssh_msg_send: write: Destination address required
11744 2024-11-11 19:43:02.807 error: send_rexec_state: ssh_msg_send failed
11744 2024-11-11 19:43:02.807 debug3: send_rexec_state: done
11744 2024-11-11 19:43:02.822 debug1: child_reap: preauth child 13164 for connection from 10.15.18.54 to 10.15.18.55 exited with status 255 (early)
11744 2024-11-11 19:43:02.822 debug3: srclimit_penalise: address 10.15.18.55 is exempt
11744 2024-11-11 19:43:05.037 debug3: spawning "c:\program files\openssh/sshd-session.exe" -R as subprocess
11744 2024-11-11 19:43:05.146 debug3: send_rexec_state: entering fd = 16 config len 2565
11744 2024-11-11 19:43:05.146 debug3: ssh_msg_send: type 0 len 4829
11744 2024-11-11 19:43:06.960 debug3: write - ERROR:109 on prior unblocking write, io:0000020447ED4B30
11744 2024-11-11 19:43:06.960 error: ssh_msg_send: write: Broken pipe
11744 2024-11-11 19:43:06.960 error: send_rexec_state: ssh_msg_send failed
11744 2024-11-11 19:43:06.960 debug3: send_rexec_state: done
11744 2024-11-11 19:43:06.960 debug1: child_reap: preauth child 15608 for connection from 10.15.18.54 to 10.15.18.55 exited with status 255 (early)
11744 2024-11-11 19:43:06.960 debug3: srclimit_penalise: address 10.15.18.55 is exempt

Meanwhile on the SSHD_Config we made the following change: (We uncomment the hostkey ecdsa and ed25519)

#HostKey PROGRAMDATA/ssh/ssh_host_rsa_key
#HostKey PROGRAMDATA/ssh/ssh_host_dsa_key
HostKey PROGRAMDATA/ssh/ssh_host_ecdsa_key
HostKey PROGRAMDATA/ssh/ssh_host_ed25519_key

After this change we stop having the errors we comented before and started having a new error:

16520 2024-11-11 19:57:47.551 debug1: should_drop_connection: p 30, r 25
16520 2024-11-11 19:57:47.551 error: beginning MaxStartups throttling
16520 2024-11-11 19:57:47.551 drop connection #10 from [10.15.18.55]:62319 on [10.15.18.54]:22 past Maxstartups
16520 2024-11-11 19:57:47.567 debug3: fd 10 is not O_NONBLOCK
16520 2024-11-11 19:57:47.583 debug1: should_drop_connection: p 30, r 64

After this new error we increased the param MaxStartups from 10 to 50 and stopped having errors and connections aborted so far.

Note: We are using two curl versions has a client:

debug1: Remote protocol version 2.0, remote software version libssh2_1.10.0

debug1: Remote protocol version 2.0, remote software version libssh2_1.11.1

[JoaoPereira1994]

@StevenBucher98 - Adding just a bit more information on our tries we found out that if we have HostKey PROGRAMDATA/ssh/ssh_host_rsa_key uncomment we start having the same errors that we mencion before again, the errors happen even if we have the rest of the Hostkeys commented or uncommented.

[tgauth]

@StevenBucher98 - Adding just a bit more information on our tries we found out that if we have HostKey PROGRAMDATA/ssh/ssh_host_rsa_key uncomment we start having the same errors that we mencion before again, the errors happen even if we have the rest of the Hostkeys commented or uncommented.

Which error do you start seeing again? Is it the:
11744 2024-11-11 19:43:06.960 debug3: write - ERROR:109 on prior unblocking write, io:0000020447ED4B30
11744 2024-11-11 19:43:06.960 error: ssh_msg_send: write: Broken pipe
11744 2024-11-11 19:43:06.960 error: send_rexec_state: ssh_msg_send failed
or
16520 2024-11-11 19:57:47.551 error: beginning MaxStartups throttling
16520 2024-11-11 19:57:47.551 drop connection from [10.15.18.55]:62319 on [10.15.18.54]:22 past Maxstartups

[JoaoPereira1994]

Hello @tgauth,

When we have sshd.config like this:

#HostKey PROGRAMDATA/ssh/ssh_host_rsa_key
#HostKey PROGRAMDATA/ssh/ssh_host_dsa_key
HostKey PROGRAMDATA/ssh/ssh_host_ecdsa_key
HostKey PROGRAMDATA/ssh/ssh_host_ed25519_key

we had the this errors - 16520 2024-11-11 19:57:47.551 error: beginning MaxStartups throttling
16520 2024-11-11 19:57:47.551 drop connection from [10.15.18.55]:62319 on [10.15.18.54]:22 past Maxstartups

that we then fixed with param - Maxstartups 50:30:100

When we have sshd.config like this:

HostKey PROGRAMDATA/ssh/ssh_host_rsa_key
#HostKey PROGRAMDATA/ssh/ssh_host_dsa_key
HostKey PROGRAMDATA/ssh/ssh_host_ecdsa_key
HostKey PROGRAMDATA/ssh/ssh_host_ed25519_key

or like this (default)

#HostKey PROGRAMDATA/ssh/ssh_host_rsa_key
#HostKey PROGRAMDATA/ssh/ssh_host_dsa_key
#HostKey PROGRAMDATA/ssh/ssh_host_ecdsa_key
#HostKey PROGRAMDATA/ssh/ssh_host_ed25519_key

We have the same errors that we had in the beggining:

write - ERROR:109 on prior unblocking write
error: ssh_msg_send: write: Broken pipe
error: send_rexec_state: ssh_msg_send failed
error: ssh_msg_send: write: Destination address required
ssh_dispatch_run_fatal: Connection from 10.1.1.94 port 44654: Connection aborted [preauth]

Note - This error happens when doing around 15 conections at the same time using curl.

[tgauth]

@JoaoPereira1994 - what version of curl are you using? I'm going to try to setup a local repro.

And are you connecting via sftp or via ssh?

[darikprescott]

I am having a bit of trouble with someone that poisoned my dns and has been spoofing my keys so I had to tear everything down and go through everything. Have been targeted by big businesses and so I'll be up and routinely involved on alot more once I have secured funds to reliably continue labeling the data and aggravating this.

[JoaoPereira1994]

Hello @tgauth,

The connections are always via sftp.

We did a lab test with the curl version (8.12.1) that uses libssh2_1.11.1 to reproduce this problem, we also got the same errors before with the libssh2_1.10.0.

To reproduce the error we had to use batch files with one of the following codes either:

LIST - curl -k --user username:password sftp://fqdn/

or

DOWNLOAD - curl -O -k --user username:password sftp://fqdn/file

Note - This error happens when running around 15 of the mencioned batch files at the same time.

The errors are the following:

write - ERROR:109 on prior unblocking write
error: ssh_msg_send: write: Broken pipe
error: send_rexec_state: ssh_msg_send failed
error: ssh_msg_send: write: Destination address required

The following error we couldn't reproduce on our lab but we had this issue in our production environment before we applied our workaround:

ssh_dispatch_run_fatal: Connection from 10.1.1.1 port 44654: Connection aborted [preauth]