October 2024 Windows updates --> Error 1067 on SSHD start

[teknowledgist]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest version
• Search the existing issues.

Steps to reproduce

Our Server 2019 with OpenSSH worked without issue for several years. Installed the October 2024 updates, rebooted, and OpenSSH won't start with a "Error 1067".

I saw that is commonly due to additional permissions on c:\ProgramData\ssh (specifically the "logs" subfolder) and possibly C:\Windows\System32\OpenSSH. I also noted that the .exe files were v9.5.2.1 with a modify date in October.

• Deleted the "logs" folder. Same problem.
• Ran the "FixHostFilePermissions.ps1" script. It reported success, but same problem.
• Removed the OpenSSH Server and Client features and re-installed them. Same problem.
• Noticed that even after removal, the service was still installed and the C:\Windows\System32\OpenSSH folder was still there. Couldn't delete it without permission from "TrustedInstaller". The SYSTEM account couldn't even see the directory. Took ownership of the directory and deleted it (after a reboot). Re-installed using the Add Features GUI. Then I get a 1058 error. Noticed that the .exe files are v7.7.2.1.

What in the world is going on here!? This shouldn't be that hard!

I installed the v9.8.1.0 .MSI from here, and it gets into an endless start-service -> unexpectedly terminated -> restart service loop (Event 7031) and also will not run.

Basically, because of some screwup with the standard monthly update, I no longer have an SSH service that is necessary for a business process.

I don't care what flavor of SSH, but please help me fix this!

Expected behavior

OpenSSH SSH Server service runs.

Actual behavior

* With v9.5.2.1 (installed after Oct. 2024 Windows updates):  Error 1067
* With v7.7.2.1 (installed via "Add a feature"): Error 1058
* With v9.8.1.0 (installed via MSI): The service starts and immediately terminates "unexpectedly" (Event 7031) over and over and over...

Error details

No response

Environment data

Name                           Value
----                           -----
PSVersion                      5.1.17763.6414
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.17763.6414
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Version

7.7.2.1, 9.5.2.1, 9.8.1.0

Visuals

No response

[tgauth]

For v9.4 and above, to fix the service startup failure, please run the commands from #2290 (comment)

[teknowledgist]

Changing the Service from "Disabled" (because even with "Manual", it constantly loops), and I immediately get:

And the same 7036 event, "The OpenSSH SSH Server service entered the running state." followed by 7031 event, "The OpenSSH SSH Server service terminated unexpectedly. It has done this 5051 time(s). The following corrective action will be taken in 0 milliseconds: Restart the service."

[tgauth]

What OpenSSH version is this with now?

And what is the path to the ssh server executable for the service? Can be found by opening services, navigating to OpenSSH SSH Server, right-clicking and selecting properties, then viewing the General tab.

[teknowledgist]

Version 9.8.1.0:

[tgauth]

Are you able to start sshd in debug mode - sshd.exe -ddd?

[tgauth]

if you have multiple versions of OpenSSH on the machine simultaneously, there may be conflicting libcrypto.dll files depending on the paths/order in the PATH environment variable

[teknowledgist]

I'm not aware of any additional versions/installs of OpenSSH. It's not in C:\Windows\System32\OpenSSH. Where else should I look?

[tgauth]

Looks like the sshd_config line 88 syntax is preventing startup

[teknowledgist]

changed from

DenyUsers

to

# DenyUsers

And it is running. Thank you!

Why can't there be a generic "config error"? It doesn't even say "error" in the debug messages.