InvalidKeySize_DoesNotInvalidateKey: An internal consistency check failed.

[EgorBo]

  Discovering: System.Security.Cryptography.Cng.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Security.Cryptography.Cng.Tests (found 1179 of 1344 test cases)
  Starting:    System.Security.Cryptography.Cng.Tests (parallel test collections = on [4 threads], stop on fail = off)
    System.Security.Cryptography.Encryption.Aes.Tests.AesModeTests.Windows7DoesNotSupportCFB128 [SKIP]
      Condition(s) not met: "IsWindows7"
    System.Security.Cryptography.Dsa.Tests.DSASignVerify_Array.InvalidKeySize_DoesNotInvalidateKey [FAIL]
      System.Security.Cryptography.CryptographicException : An internal consistency check failed.
      Stack Trace:
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CngKey.Create.cs(58,0): at System.Security.Cryptography.CngKey.Create(CngAlgorithm algorithm, String keyName, CngKeyCreationParameters creationParameters)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CngAlgorithmCore.cs(73,0): at System.Security.Cryptography.CngAlgorithmCore.GetOrGenerateKey(Int32 keySize, CngAlgorithm algorithm)
        /_/src/libraries/Common/src/System/Security/Cryptography/DSACng.SignVerify.cs(147,0): at System.Security.Cryptography.DSACng.ComputeQLength()
        /_/src/libraries/Common/src/System/Security/Cryptography/DSACng.SignVerify.cs(125,0): at System.Security.Cryptography.DSACng.AdjustHashSizeIfNecessary(ReadOnlySpan`1 hash, Span`1 stackBuf)
        /_/src/libraries/Common/src/System/Security/Cryptography/DSACng.SignVerify.cs(25,0): at System.Security.Cryptography.DSACng.CreateSignature(Byte[] rgbHash)
        /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/DSA.cs(149,0): at System.Security.Cryptography.DSA.SignData(Byte[] data, Int32 offset, Int32 count, HashAlgorithmName hashAlgorithm)
        /_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/DSA/DSASignVerify.cs(14,0): at System.Security.Cryptography.Dsa.Tests.DSASignVerify_Array.SignData(DSA dsa, Byte[] data, HashAlgorithmName hashAlgorithm)
        /_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/DSA/DSASignVerify.cs(123,0): at System.Security.Cryptography.Dsa.Tests.DSASignVerify.InvalidKeySize_DoesNotInvalidateKey()
        /_/src/coreclr/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.CoreCLR.cs(36,0): at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs(57,0): at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=915376&view=ms.vss-test-web.build-test-results-tab&runId=24176138&resultId=113288&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

(net10.0-windows-Release-arm64-jitstress_random_2-Windows.11.Arm64.Open on runtime-coreclr libraries-jitstress-random)

I couldn't use "Known build fauilure" template because https://helix.dot.net/BuildAnalysis/CreateKnownIssues is down.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[v-wenyuxu]

Failed in: runtime-coreclr libraries-jitstress-random 20250113.1

Failed tests:

net10.0-windows-Release-x86-jitstress_random_1-Windows.10.Amd64.Open
    - System.Security.Cryptography.Dsa.Tests.DSASignVerify_Array.InvalidKeySize_DoesNotInvalidateKey

Error message:

 System.Security.Cryptography.CryptographicException : An internal consistency check failed.

Stack trace:

   at System.Security.Cryptography.CngKey.Create(CngAlgorithm algorithm, String keyName, CngKeyCreationParameters creationParameters) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CngKey.Create.cs:line 58
   at System.Security.Cryptography.CngAlgorithmCore.GetOrGenerateKey(Int32 keySize, CngAlgorithm algorithm) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/CngAlgorithmCore.cs:line 73
   at System.Security.Cryptography.DSACng.ComputeQLength() in /_/src/libraries/Common/src/System/Security/Cryptography/DSACng.SignVerify.cs:line 147
   at System.Security.Cryptography.DSACng.AdjustHashSizeIfNecessary(ReadOnlySpan`1 hash, Span`1 stackBuf) in /_/src/libraries/Common/src/System/Security/Cryptography/DSACng.SignVerify.cs:line 125
   at System.Security.Cryptography.DSACng.CreateSignature(Byte[] rgbHash) in /_/src/libraries/Common/src/System/Security/Cryptography/DSACng.SignVerify.cs:line 25
   at System.Security.Cryptography.DSA.SignData(Byte[] data, Int32 offset, Int32 count, HashAlgorithmName hashAlgorithm) in /_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/DSA.cs:line 149
   at System.Security.Cryptography.Dsa.Tests.DSASignVerify_Array.SignData(DSA dsa, Byte[] data, HashAlgorithmName hashAlgorithm) in /_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/DSA/DSASignVerify.cs:line 14
   at System.Security.Cryptography.Dsa.Tests.DSASignVerify.InvalidKeySize_DoesNotInvalidateKey() in /_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/DSA/DSASignVerify.cs:line 123
   at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args) in /_/src/coreclr/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.CoreCLR.cs:line 36
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) in /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs:line 57

[jeffhandley]

Assigned to @bartonjs for triage

[bartonjs]

Different callstack, but it's the same "Windows CNG is in a bad state", and we've not been successful in getting their help to investigate.