fix(auth): handle binary SASL initial response correctly

[KrisnaDana]

Summary

Fix intermittent SCRAM-SHA-256 authentication failures when connecting to PgDoorman using the Rust SQLx PostgreSQL driver.

The issue is caused by converting the entire binary SASL initial response into a UTF-8 string before extracting the SCRAM payload. This can corrupt byte offsets and cause authentication failures depending on the length of the SCRAM client-first-message.

This PR introduces binary-safe SASL parsing while preserving the existing Cow<str> signature of parse_client_first_message().

Problem

During testing, SQLx produced inconsistent authentication results when establishing repeated connections through PgDoorman using the same connection configuration.

SQLx — 200 sequential connections:

• Successful: 137
• Failed: 63
• Failure rate: 31.5%

The failures occurred during connection authentication, before executing SELECT 1.

The following PostgreSQL error was returned:

SQLSTATE: 28P01
Authentication protocol error.
Your client may not support SCRAM authentication properly.

Some connections succeeded while others failed despite using identical credentials and connection settings.

This behavior suggests an intermittent issue in PgDoorman's SCRAM authentication handling, triggered by specific SASL message lengths generated by SQLx.

Root Cause

The previous implementation processed the binary SASL initial response using:

String::from_utf8_lossy(&first_message)

However, the SASL initial response contains:

1. A null-terminated SASL mechanism name.
2. A 4-byte big-endian response length.
3. The SCRAM client-first-message payload.

When the encoded response length contains bytes that are not valid standalone UTF-8 sequences, String::from_utf8_lossy() replaces them with Unicode replacement characters.

This changes the byte representation and can invalidate the fixed byte offsets used to extract the SCRAM payload.

SQLx generates variable-length SCRAM nonces, making the issue appear intermittently depending on the resulting message length.

Changes

This PR:

• Introduces parse_sasl_initial_response() to parse the SASL initial response directly from raw bytes.
• Validates the SASL authentication mechanism and declared response length.
• Extracts the SCRAM payload without converting the binary header to UTF-8.
• Converts only the extracted SCRAM payload into UTF-8.
• Preserves the existing Cow<str> parameter in parse_client_first_message().
• Updates the SCRAM authentication call sites, including the auth_query path.
• Adjusts existing SCRAM unit tests to match the updated parser input contract.
• Adds regression coverage for variable-length SCRAM messages.

Testing

1. SQLx Connection Test (Before Patch)

The original SQLx test performed 200 sequential connections against PgDoorman.

DONE OK=137 FAIL=63

Authentication failures occurred intermittently with SQLSTATE 28P01.

The failure rate was approximately 31.5%, even though all connections used the same credentials and configuration.

These results were obtained before the patch and are not an end-to-end verification of the patched PgDoorman v3.11.0 build.

2. Binary-Safe Parsing Regression Test

A deterministic 200-iteration reproduction was used to compare the legacy parsing approach with binary-safe parsing.

Legacy parser:

• Total iterations: 200
• Successful: 152
• Failed: 48
• Success rate: 76%

Binary-safe parser:

• Total iterations: 200
• Successful: 200
• Failed: 0
• Success rate: 100%

The legacy parsing approach failed when the SCRAM client-first-message length reached 128 bytes or more.

The binary-safe parser successfully handled all 200 simulated inputs and rejected malformed response lengths.

These results demonstrate that binary-safe parsing resolves the reproduced byte-offset issue.

Note: This deterministic reproduction validates the parsing logic at the unit-test level. It is not an end-to-end authentication test against the patched PgDoorman server.

3. Existing SCRAM Unit Tests

The existing good_parse_client_first_message unit test was updated to reflect the new input contract.

Previously, the parser received a string representation of the complete binary SASL initial response.

After the change, the parser receives only the extracted SCRAM text payload, while the binary header is handled separately by parse_sasl_initial_response().

This separates binary protocol parsing from SCRAM message parsing and maintains the existing Cow<str> function signature.

4. Full Library Unit Test Suite

The complete PgDoorman library unit test suite was executed after applying the patch and updating the existing SCRAM tests.

Command:

cargo test --lib

Result:

test result: ok. 1211 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 2.70s

Status: PASSED

• 1,211 tests passed
• 0 tests failed
• 1 test ignored
• No regressions detected by the existing library unit test suite.

This confirms that the patched implementation passes the complete library unit test suite, including the updated SCRAM authentication tests.

Expected Behavior

PgDoorman should correctly process valid SCRAM-SHA-256 SASL initial responses regardless of their length.

SQLx connections should no longer experience intermittent authentication failures caused by binary-to-UTF-8 conversion.

The parser should also reject malformed SASL responses, including:

• Invalid authentication mechanisms.
• Incorrect response lengths.
• Invalid UTF-8 payloads.

Environment

• Target PgDoorman version: v3.11.0
• PostgreSQL authentication: SCRAM-SHA-256
• PostgreSQL driver: SQLx (Rust)
• Pooling mode: Transaction
• Container runtime: Podman

Additional Notes

This patch focuses specifically on the parsing of SASL initial responses.

It does not change the SCRAM cryptographic algorithm, password verification, or connection pooling behavior.

The goal is to improve PostgreSQL protocol correctness and compatibility with SQLx clients that generate variable-length SCRAM messages.

All 1,211 library unit tests pass after the changes. End-to-end SQLx connection testing against a patched PgDoorman container remains a separate verification step.