initializeTestEnvironment sometimes failed due to Node.js fetch failed by "SocketError: other side closed"

[jpreynat]

[REQUIRED] Environment info

firebase-tools: 13.35.1
firebase: 11.5.0
@firebase/rules-unit-testing: 4.0.1

**Platform:**macOS

[REQUIRED] Test case

Running multiple tests in CI using initializeTestEnvironment, eventually a random test fails with a TypeError: fetch failed error with a Cause: SocketError: other side closed in the discoverEmulators function from @firebase/rules-unit-testing:

TypeError: fetch failed
    ...
    at discoverEmulators (node_modules/.pnpm/@[email protected][email protected]/node_modules/@firebase/rules-unit-testing/src/impl/discovery.ts:31:15)
    at initializeTestEnvironment (node_modules/.pnpm/@[email protected][email protected]/node_modules/@firebase/rules-unit-testing/src/initialize.ts:66:32)
    at <internal-code>

Cause:
SocketError: other side closed

[REQUIRED] Steps to reproduce

Only one test is failing randomly over hundreds that we have, so reproducing is very difficult.
It also happens only in CI and not locally.

It looks like it's happening randomly for other users since starting using firebase@11 and an issue has already been opened on the firebase-js-sdk repo: firebase/firebase-js-sdk#8690

They recommended opening an issue here since it looks related to the emulator rather than the SDK code.

[REQUIRED] Expected behavior

Tests shouldn't fail, as they didn't before moving to firebase@11, although keeping the previously working [email protected] version or the newest [email protected] doesn't change anything.

[REQUIRED] Actual behavior

Described in the Test case

[aalej]

Thanks for the detailed report @jpreynat! At the moment, it’s still difficult to determine if the issue is caused by the client SDK or the emulator suite. Reading through firebase/firebase-js-sdk#8690, it looks like it’s only reproducible on GitHub workflows. It’s also possible that GitHub actions might be throttling the request, but I can’t really say for sure.

I created this repo to try and replicate the issue, but so far, I haven’t encountered any errors. I feel like I may be missing something here, could you let me know if I’m doing anything wrong with the setup? Also, how often does the issue occur? For example, in 100 runs, how many will encounter the error?

[jpreynat]

Thanks for the reply @aalej!

The repro looks good, but it might be a scale effect indeed.

We have multiple hundreds of tests, I'd say at least 500, split into 10 parallel CI runs, and only one of those hundreds of tests is failing.