Report connectivity information when connectivity fails

[ramonsmits]

Describe the feature.

Is your feature related to a problem? Please describe.

A customer was having Azure Service Bus connectivity issues which they were unable to resolve themselves. From their perspective things were working as they used Azure Service Bus Explorer which "just worked".

The difference is that Azure Service Bus Explorer uses websockets by default which uses port 443 and our transport uses AMQP ports 5671 (AMQP TLS) and 5672 (AMQP plain).

When connecting fails this is logged by ServiceControl

2024-11-20 07:19:15.7207|29|Warn|NServiceBus.CustomChecks.TimerBasedPeriodicCheck|Failed to report periodic check to ServiceControl.
Azure.Messaging.ServiceBus.ServiceBusException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond ErrorCode: TimedOut (ServiceCommunicationProblem) ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

It isn't clear to what its connecting (IP or hostname), what port number(s), and what protocol(s) like TCP/UDP

Second, it is strange that NServiceBus.CustomChecks.TimerBasedPeriodicCheck is reporting this but not any of the ingesters for ServiceControl.

Describe the requested feature

1. Improve any connectivity exceptions to include connection information. This will make it easier to resolve issues because WARN log entries are also logged to the windows event log which would now contain all information to resolve this issue
2. As custom checks NEED the broker connectivity to work these can't be updated thus nothing can be observed in ServicePulse
3. Connectivity exceptions should be handled gracefully and not have the .ToString() invoked as that just isn't providing any useful information. This will

Additional Context

Full exception:

2024-11-20 07:19:16.0091|24|Warn|ServiceControl.CustomChecks.InternalCustomCheckManager|Failed to update periodic check status.
Azure.Messaging.ServiceBus.ServiceBusException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond ErrorCode: TimedOut (ServiceCommunicationProblem) ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.Azure.Amqp.Transport.AmqpTransportInitiator.<>c.<ConnectAsync>b__17_1(IAsyncResult r)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Messaging.ServiceBus.Amqp.AmqpConnectionScope.<CreateAndOpenConnectionAsync>d__72.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Amqp.FaultTolerantAmqpObject`1.<OnCreateAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Amqp.Singleton`1.<GetOrCreateAsync>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Amqp.Singleton`1.<GetOrCreateAsync>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Messaging.ServiceBus.Amqp.AmqpConnectionScope.<OpenSenderLinkAsync>d__70.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<CreateLinkAndEnsureSenderStateAsync>d__28.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Amqp.FaultTolerantAmqpObject`1.<OnCreateAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Amqp.Singleton`1.<GetOrCreateAsync>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Amqp.Singleton`1.<GetOrCreateAsync>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<SendBatchInternalAsync>d__19.MoveNext()
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<SendBatchInternalAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<SendAsync>b__20_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.<>c__20`1.<<RunOperation>b__20_0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.<RunOperation>d__21`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.<RunOperation>d__21`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.<RunOperation>d__20`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<SendAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Messaging.ServiceBus.ServiceBusSender.<SendMessagesAsync>d__26.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Messaging.ServiceBus.ServiceBusSender.<SendMessageAsync>d__25.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.SerializeMessageConnector.<Invoke>d__1.MoveNext() in /_/src/NServiceBus.Core/Pipeline/Outgoing/SerializeMessageConnector.cs:line 39
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.SendConnector.<Invoke>d__1.MoveNext() in /_/src/NServiceBus.Core/Routing/Routers/SendConnector.cs:line 23
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at ServiceControl.Infrastructure.DomainEvents.DomainEvents.<Raise>d__2`1.MoveNext() in /_/src/ServiceControl/Infrastructure/DomainEvents/DomainEvents.cs:line 15
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at ServiceControl.CustomChecks.CustomChecksStorage.<UpdateCustomCheckStatus>d__1.MoveNext() in /_/src/ServiceControl/CustomChecks/CustomChecksStorage.cs:line 72
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at ServiceControl.CustomChecks.InternalCustomCheckManager.<Run>d__2.MoveNext() in /_/src/ServiceControl/CustomChecks/InternalCustomChecks/InternalCustomCheckManager.cs:line 49