After downgrading an AKS cluster to free tier I have seen this happening. It's probably due to the free tier not having as good uptime.
I looked at the proto actor code and it seems that proto actor should try to reconnect when it loses connection but in this case it seems like it got an Unhandled exception instead.
Checking KubernetesClusterMonitor.cs I think the culprint is the tcs.SetResult(); inside Restart().
I suggest changing it into tcs.TrySetResult() to avoid throwing exception. Or setting a flag that indicates that we already called restart?
{"Timestamp":"2026-02-17T04:21:42.7382017Z","Message":"[Cluster][KubernetesProvider] Unable to watch the cluster status","level":"Error","Exception":"System.Net.Http.HttpProtocolException: The HTTP/2 server reset the stream. HTTP/2 error code 'NO_ERROR' (0x0). (HttpProtocolError)\n at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)\n at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead)\n at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)\n at k8s.LineSeparatedHttpContent.CancelableStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)\n at System.IO.StreamReader.ReadBufferAsync(CancellationToken cancellationToken)\n at System.IO.StreamReader.ReadLineAsyncInternal(CancellationToken cancellationToken)\n at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()\n at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)\n--- End of stack trace from previous location ---\n at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)\n at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)\n--- End of stack trace from previous location ---\n at k8s.Watcher`1.CreateWatchEventEnumerator(Func`1 streamReaderCreator, Action`1 onError, CancellationToken cancellationToken)+MoveNext()\n at k8s.Watcher`1.CreateWatchEventEnumerator(Func`1 streamReaderCreator, Action`1 onError, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()\n at k8s.Watcher`1.WatcherLoop(CancellationToken cancellationToken)\n at k8s.Watcher`1.WatcherLoop(CancellationToken cancellationToken)","SourceContext":"Proto.Cluster.Kubernetes.KubernetesClusterMonitor","ExceptionDetail":{"HResult":-2146232800,"Message":"The HTTP/2 server reset the stream. HTTP/2 error code 'NO_ERROR' (0x0). (HttpProtocolError)","Source":"System.Net.Http","TargetSite":"Void ThrowRequestAborted(System.Exception)","ErrorCode":0,"HttpRequestError":"HttpProtocolError","Type":"System.Net.Http.HttpProtocolException"},"Pod":"pod-xxx","ApplicationName":"xxx"}
{"Timestamp":"2026-02-17T04:21:43.0064750Z","Message":"[Cluster][KubernetesProvider] Recreating Kubernetes client due to connectivity error","level":"Warning","SourceContext":"Proto.Cluster.Kubernetes.KubernetesClusterMonitor","Pod":"pod-xxx","ApplicationName":"xxx"}
Unhandled exception. System.InvalidOperationException: An attempt was made to transition a task to a final state when it had already completed.
at Proto.Cluster.Kubernetes.KubernetesClusterMonitor.<>c__DisplayClass17_0.<Watch>g__Closed|1()
at k8s.Watcher`1.WatcherLoop(CancellationToken cancellationToken)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
--- End of stack trace from previous location ---
at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__124_1(Object state)
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
at System.Threading.Thread.StartCallback()
ProtoActor version 1.8.0
After downgrading an AKS cluster to free tier I have seen this happening. It's probably due to the free tier not having as good uptime.
I looked at the proto actor code and it seems that proto actor should try to reconnect when it loses connection but in this case it seems like it got an Unhandled exception instead.
Checking KubernetesClusterMonitor.cs I think the culprint is the tcs.SetResult(); inside Restart().
I suggest changing it into tcs.TrySetResult() to avoid throwing exception. Or setting a flag that indicates that we already called restart?
Microsoft docs:
My application crash log: