Microsoft.Data.SqlClient.SNI.x86.dll, version: 5.2.0.0 crashed host application

[xiangdguo]

Describe the bug

Microsoft.Data.SqlClient.SNI.x86.dll, version: 5.2.0.0 crashed hosted application. The host application BPHost runs multiple backend tasks in parallel in different app domains. Each task run will use the sample code below to retrieve data from a SQL Server database. It crashes multiple times every day randomly.

Application Error from Windows Event Log:
Faulting application name: Eclipse.BPHost.exe, version: 1.5.35.43, time stamp: 0x6740e2c0
Faulting module name: Microsoft.Data.SqlClient.SNI.x86.dll, version: 5.2.0.0, time stamp: 0x65d526aa
Exception code: 0xc0000409
Fault offset: 0x000013fb
Faulting process id: 0x62d0
Faulting application start time: 0x01db41e100d49446
Faulting application path: E:\Smartflow\Halo\SmartFlow.Halo.Agent\Services\Sky2\Eclipse\Eclipse.BPHost.exe
Faulting module path: E:\Smartflow\Halo\SmartFlow.Halo.Agent\Services\Sky2\Eclipse\DTE\Microsoft.Data.SqlClient.SNI.x86.dll
Report Id: 4dbbfafa-baf6-456b-a345-ad2f96b1c3a1

To reproduce

    public static async Task<Dictionary<string, Object>> GetColumnValuesAsync(string connectionString, string userId, SecureString password, List<string> columnNames, string tableName,
                             string whereClauseWithParameterNames, List<(string paramName, SqlDbType dataType, object paramValue, int size, byte scale, byte precision)> parameterNamesWithValues,
                             CommandBehavior commandBehavior = CommandBehavior.Default, int? commandTimeout = null, ILogger? logger = null)
    {

#if NETSTANDARD2_0 || NETSTANDARD2_1 || NETCOREAPP3_1
        _ = connectionString ?? throw new ArgumentNullException(nameof(connectionString), "GetColumnValuesAsync requires connection string");
        _ = userId ?? throw new ArgumentNullException(nameof(userId), "GetColumnValuesAsync requires UserId to access DB");
        _ = password ?? throw new ArgumentNullException(nameof(password), "GetColumnValuesAsync requires Password to access DB");
        _ = tableName ?? throw new ArgumentNullException(nameof(tableName), "GetColumnValuesAsync requires tableName");
        _ = columnNames ?? throw new ArgumentNullException(nameof(columnNames), "GetColumnValuesAsync requires columnNames");
        _ = whereClauseWithParameterNames ?? throw new ArgumentNullException(nameof(whereClauseWithParameterNames), "GetColumnValuesAsync requires whereClauseWithParameterNames");
        _ = parameterNamesWithValues ?? throw new ArgumentNullException(nameof(parameterNamesWithValues), "GetColumnValuesAsync requires parameterNamesWithValues");
#else
        ArgumentNullException.ThrowIfNull(connectionString);
        ArgumentNullException.ThrowIfNull(userId);
        ArgumentNullException.ThrowIfNull(password);
        ArgumentNullException.ThrowIfNull(tableName);
        ArgumentNullException.ThrowIfNull(columnNames);
        ArgumentNullException.ThrowIfNull(whereClauseWithParameterNames);
        ArgumentNullException.ThrowIfNull(parameterNamesWithValues);
#endif

        Dictionary<string, Object> returnDictionary = [];
        StringBuilder sb = new();
        sb.Append("SELECT ");

        foreach (string colName in columnNames)
        {
            sb.AppendFormat(CultureInfo.InvariantCulture, "{0},", colName);
        }
        sb.Length--;

        sb.AppendFormat(CultureInfo.InvariantCulture, " FROM {0} WHERE {1}", tableName, whereClauseWithParameterNames);

        string cmdText = sb.ToString();

        using (SqlConnection conn = new(connectionString) { Credential = new SqlCredential(userId, password) })
        {
            using SqlCommand cmd = new()
            {
                Connection = conn,
                CommandText = cmdText,
                // Also need to specify that this is a stored procedure command (default is Text)
                CommandType = CommandType.Text
            };

            if (commandTimeout.HasValue && commandTimeout.Value >= 0)
                cmd.CommandTimeout = commandTimeout.Value;

            if (parameterNamesWithValues != null)
            {
                foreach (var (paramName, dataType, paramValue, size, scale, precision) in parameterNamesWithValues)
                {
                    if (dataType == SqlDbType.VarChar || dataType == SqlDbType.NVarChar || dataType == SqlDbType.Char || dataType == SqlDbType.NChar || dataType == SqlDbType.Text || dataType == SqlDbType.NText)
                        cmd.Parameters.Add(new SqlParameter { SqlDbType = dataType, ParameterName = paramName, Direction = ParameterDirection.Output, Value = paramValue, Size = size });
                    else if (dataType == SqlDbType.Decimal)
                        cmd.Parameters.Add(new SqlParameter { SqlDbType = dataType, ParameterName = paramName, Direction = ParameterDirection.Output, Value = paramValue, Scale = scale, Precision = precision, });
                    else if (dataType == SqlDbType.DateTime2)
                        cmd.Parameters.Add(new SqlParameter { SqlDbType = dataType, ParameterName = paramName, Direction = ParameterDirection.Output, Value = paramValue, Scale = scale, });
                    else
                        cmd.Parameters.Add(new SqlParameter { SqlDbType = dataType, ParameterName = paramName, Direction = ParameterDirection.Input, Value = paramValue });
                }
            }
            // Create the input parameter, set the properties and add to command.

            await conn.OpenAsync().ConfigureAwait(false);
            using var reader = await cmd.ExecuteReaderAsync(commandBehavior).ConfigureAwait(false);
            await reader.ReadAsync().ConfigureAwait(false);
            returnDictionary = Enumerable.Range(0, reader.FieldCount)
                        .ToDictionary(reader.GetName, reader.GetValue);
        }
        return returnDictionary;
    }

Expected behavior

Do not crash the hosted app and let the tasks continue in next run later even there is an exception in the code.

Further technical details

Microsoft.Data.SqlClient version: 5.2.2
.NET target: .NET Framework 4.7.2,
SQL Server version: SQL Server 2017
Operating system: Windows Server 2019

[mdaigle]

Hi @xiangdguo, is there any more information you can provide on the exception you're seeing? Additional error messages or stack traces would be helpful.

Are there any other characteristics of the failure (e.g. high load on the server) that can help us pinpoint the issue?

[xiangdguo]

Hi @mdaigle, thank you for the reply above! We tried to capture more exception message by adding exception handling code in the host app, but the process crashed the whole host app before reaching the exception handling code.

As I described above, the host app runs multiple processes in parallel, and each process calls MSD 5.2 client to retrieve data from a SQL Server DB. Can a 32 bit host app running on Windows 2019 64 bit OS be a root cause?

[cheenamalhotra]

As I described above, the host app runs multiple processes in parallel, and each process calls MSD 5.2 client to retrieve data from a SQL Server DB. Can a 32 bit host app running on Windows 2019 64 bit OS be a root cause?

Hi @xiangdguo

A few follow up questions:

• How many processes are running in parallel?
• Does modifying that number to a lower count help prevent crashes?
• Do you have any Server side logs you can provide from ERRORLOG file?
• Has this issue always occurred with MDS or this started happening after a recent upgrade? What version worked for you before, if it did?

Would appreciate more details. Thanks!

[xiangdguo]

Hi @cheenamalhotra,

Below are my answers following your questions.

• How many processes are running in parallel?
  It happens with 5 parallel processes in our testing environment.

• Does modifying that number to a lower count help prevent crashes?
  No, it happens in both testing (much less parallel process) and production (many more parallel processes) environment.

• Do you have any Server side logs you can provide from ERRORLOG file?
  We added exception handling and logging code for the code that calls MDS, but nothing has been logged for the crashes.

• Has this issue always occurred with MDS or this started happening after a recent upgrade? What version worked for you before, if it did?
  Always occurred with 5.2. We had not used the lower/older versions of MDS.

Thank you!

[xiangdguo]

@cheenamalhotra @mdaigle Hello. In the release notes of 5.0.1 it was said "Updated Microsoft.Data.SqlClient.SNI (.NET Framework dependency) and Microsoft.Data.SqlClient.SNI.runtime (.NET Core/Standard dependency) version to 5.0.1 #1795, which includes the fix for AppDomain crash introducing in issue #1418."

Does it mean you have not updated the SNI dlls for .Net framework 4.x yet? I'm asking this question because our host application targets .Net framework 4.7.

Thanks!

[cheenamalhotra]

Hi @xiangdguo

Microsoft.Data.SqlClient targets net462 as the lowest framework version supported and so our DLLs and dependencies are also applicable to .NET Framework 4.7 as in your case. You can check the list of dependencies on NuGet.org for Microsoft.Data.SqlClient v5.0.1.

P.S. v5.0.1 contains a vulnerability, please upgrade to a non-vulnerable version as soon as possible.

[xiangdguo]

Hi @cheenamalhotra,

Can you please confirm that the previous fix released within 5.0.1 applies to Microsoft.Data.SqlClient.SNI.x86.dll, i.e. 32 bit DLL too? Because the 32 bit dll is what we used and crashed.

BTW, as pointed out in the subject, we are using version 5.2 of Microsoft.Data.SqlClient.SNI.x86.dll.

Thanks.

[mdaigle]

Hi @xiangdguo, yes, that fix should apply to the 5.2 release.
I think it will be pretty tough for us to help given the amount of information available to us at the moment. If there's any way you can scope down the issue and get an error log, that's really the information we'll need.

Other info that would also be helpful:

• Does the issue repro outside of the host app environment? If so, can you provide setup steps so that we can reproduce the same.
• Is there any way you can capture which operation it fails on? Is it when opening the connection, when executing the command, or when reading results?
• Can you provide details on the connection string you're using?

Some other things you can try if collecting error info hits a dead end:

1. Upgrade to Microsoft.Data.SqlClient 6.0.1. There have been other small changes to SNI since 5.2. Without knowing the true nature of the issue you're facing, I can't say if it will help, but could be worth a try.
2. Use managed networking on windows. This isn't our officially recommended setting for windows at the moment due to low adoption (we don't have as much data on how it performs in the real world). If you validate that it works for your workload, it could let you skip over native SNI code.

[xiangdguo]

Hello @mdaigle,

Thank you for providing more information!

In debug mode we found the crash happened at this line "await conn.OpenAsync().ConfigureAwait(false);" in the sample code above. Our connection string is like "Data Source=;Initial Catalog=;encrypt=true;trustServerCertificate=false".

The crash happens only in the app that runs multiple DB access tasks in multiple app domains.

Thanks,
Xiang