[TCP] Client hangs when it receives an invalid address for reconnection

[hayley-jean]

Describe the bug
This is related to issue EventStore/EventStore#2840

When a node returns NotHandled.NotLeader to the client, it tries to provide the leader address as part of the response.
If the node does not have a leader then it falls back to providing its own address.

This causes an issue when the node has an ExtIp of 0.0.0.0, as it will provide that in the response to the client.
When the client tries to connect to this address it will throw the exception IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address..

The client fails to reconnect and gets stuck. For some reason the connection is also not closed during the reconnection, logging CloseTcpConnection IGNORED because was closed. instead.
The client is also able to receive and respond to heartbeats on the original connection.

The client will remain hanging until a heartbeat failure (e.g. shutting the node down) or the client is forcibly closed.

To Reproduce
Steps to reproduce the behavior:

1. Start a read only replica to force it into read only leaderless state with the following config:

---
IntIp: 127.0.0.1
ExtIp: 0.0.0.0
ClusterSize: 3
DiscoverViaDns: false
GossipSeed: 127.0.0.1:1111,127.0.0.1:2222 # These nodes must not be running
Insecure: true
EnableExternalTcp: true
ReadOnlyReplica: true

2. Run the following client code:

var settings = ConnectionSettings.Create()
    .PerformOnLeaderOnly()
    .UseConsoleLogger()
    .EnableVerboseLogging()
    .DisableTls();
var connection = EventStoreConnection.Create(settings, new IPEndPoint(IPAddress.Loopback, 1113));

await connection.ConnectAsync();

await connection.AppendToStreamAsync(
    "test-stream",
    ExpectedVersion.Any,
    new EventData(
        Guid.NewGuid(),
        "test-event",
        true,
        Encoding.ASCII.GetBytes("{\"foo\":\"bar\"}"),
        null));

Expected behavior
The client logs an error that the address provided is invalid, and then reverts back to node discovery or reconnections.

Actual behavior
The client hangs until forcibly shut down.

Config/Logs/Screenshots
Example client log:

[09,22:44:31.495,DEBUG] EventStoreConnection 'ES-4abfce9d-0bd7-4e1a-9924-91f7a49f3730': enqueueing message EventStore.ClientAPI.Internal.HandleTcpPackageMessage..
[05,22:44:31.495,DEBUG] EventStoreConnection 'ES-4abfce9d-0bd7-4e1a-9924-91f7a49f3730': HandleTcpPackage connId 378f66f9-a6f4-4e39-b279-74ffa165e76f, package NotHandled, 4776adf3-ed8d-41b2-ae17-b5f27ffe20b3..
[05,22:44:31.496,DEBUG] EventStoreConnection 'ES-4abfce9d-0bd7-4e1a-9924-91f7a49f3730': HandleTcpPackage OPERATION DECISION Reconnect (NotHandled - NotLeader), Operation AppendToStreamOperation (4776adf3-ed8d-41b2-ae17-b5f27ffe2
0b3): Stream: test-stream, ExpectedVersion: -2, retry count: 1, created: 22:44:29.117, last updated: 22:44:31.492.
[05,22:44:31.497,INFO] EventStoreConnection 'ES-4abfce9d-0bd7-4e1a-9924-91f7a49f3730': going to reconnect to [Unspecified/0.0.0.0:3112]. Current endpoint: [127.0.0.1:3112, L127.0.0.1:61050].
[05,22:44:31.497,DEBUG] EventStoreConnection 'ES-4abfce9d-0bd7-4e1a-9924-91f7a49f3730': CloseTcpConnection IGNORED because was closed.
[05,22:44:31.497,DEBUG] EventStoreConnection 'ES-4abfce9d-0bd7-4e1a-9924-91f7a49f3730': EstablishTcpConnection to [Unspecified/0.0.0.0:3112].
[05,22:44:31.497,ERROR] Error processing EventStore.ClientAPI.Internal.HandleTcpPackageMessage
EXCEPTION(S) OCCURRED:
System.ArgumentException: IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address.
   at System.Net.Dns.GetHostAddresses(String hostNameOrAddress)
   at EventStore.ClientAPI.Internal.EndPointExtensions.ResolveDnsToIPAddress(EndPoint endpoint)
   at EventStore.ClientAPI.Transport.Tcp.TcpClientConnector.ConnectTo(ILogger log, Guid connectionId, EndPoint remoteEndPoint, Boolean ssl, Boolean validateServer, TimeSpan timeout, Action`1 onConnectionEstablished, Action`2 onC
onnectionFailed, Action`2 onConnectionClosed)
   at EventStore.ClientAPI.Transport.Tcp.TcpPackageConnection..ctor(ILogger log, EndPoint remoteEndPoint, Guid connectionId, Boolean ssl, Boolean validateServer, TimeSpan timeout, Action`2 handlePackage, Action`2 onError, Action
`1 connectionEstablished, Action`2 connectionClosed)
   at EventStore.ClientAPI.Internal.EventStoreConnectionLogicHandler.EstablishTcpConnection(NodeEndPoints endPoints)
   at EventStore.ClientAPI.Internal.EventStoreConnectionLogicHandler.ReconnectTo(NodeEndPoints endPoints)
   at EventStore.ClientAPI.Internal.EventStoreConnectionLogicHandler.HandleTcpPackage(TcpPackageConnection connection, TcpPackage package)
   at EventStore.ClientAPI.Internal.EventStoreConnectionLogicHandler.<.ctor>b__23_7(HandleTcpPackageMessage msg)
   at EventStore.ClientAPI.Internal.SimpleQueuedHandler.<>c__DisplayClass5_0`1.<RegisterHandler>b__0(Message msg)
   at EventStore.ClientAPI.Internal.SimpleQueuedHandler.ProcessQueue(Object state)

EventStore details

• EventStore server version: >= 20.6.0
• EventStore TCP client version (if applicable): >= 20.6.0, likely v5 as well