Outgoing Http Requests "faulting". How can I get logs from the TCP stack?

[Leonardo-Ferreira]

First of all, sorry for not being able to produce a minimal reproduction. This is because my app uses mTLS and encryption of the body and a lot of sensitive things.

Description

My app consumes an external API and since the begging of the week, app insights has been logging a few result codes "Faulted"... not 400, not 500, "Faulted". We ran for about 65 days without a hiccup.

These "Faulted" requests are not consistent with anything. A simple retry might do the trick, but sometimes after 3 retries (spaced by 1, 3 and 7 seconds) won't do it. At the pod next door, no errors so we don't believe the problem is at the destination.

We ran a TCP dump and noticed that our pod is responding some packages with RESET. These appear to be random but the fact is that whenever there is a RESET, we get a "FAULTED".

In order to further troubleshoot this I was wondering if I can get TCP level logs to help me understand why is my code issuing "reset" packages?

Details

Dotnet + SO Version: Asp.Net 8.0.17-alpine

[MihuBot]

I'm a bot. Here is a possible related and/or duplicate issue (I may be wrong):

• Connection reset by peer error when sending requests using HttpClient #95516

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[wfurt]

you can enable diag logs on Sockets but that can be quite verbose and it may be harder to correlate back to particular request. I'm also not sure if that is something caused by the application e.g. the HTTP layer. You may get RST from the endpoint or any intermediate nodes like firewalls and load balancers. It may be caused by trying to re-use connection that was idle for a while. If the idle timeout is longer than what the network is willing to tolerate you may get to this situation.

If you want to dig deeper SocketsHttpHandler.ConnectCallback may be your best option. You can create sitcom stream with tracking and correlate back to your app. The TCP is all handled by Kernel so even with the tracing you may not see much.

[dotnet-policy-service]

This issue has been marked needs-author-action and may be missing some important information.