[Bug] Workflow task not found after SDK upgrade

[btravers]

What are you really trying to do?

We are upgrading Temporal SDK from 1.11.8 to 1.12.1.

Describe the bug

We have a long-running workflow. After upgrading the SDK, we are experiencing the following error: "Workflow task not found."
When we roll back to the previous version, the error disappears.

[mjameswh]

Do you see any other error message besides the "Workflow task not found."? Any information that could help diagnose or reproduce what you are observing?

[shaz1v]

We are seeing similar issues as-well with the new version, the workflows we tested the new SDK version with were short-running ones with large number activities, and the new SDK resulted in very slow processing and workflow being stuck for few good minutes before picking up the next activity, reverting resolved and everything instantly closed

The full log line we saw was

2025-07-31T07:46:46.723583Z WARN temporal_sdk_core::worker::workflow: Task not found when completing error=status: NotFound, message: "Workflow task not found.", details: [8, 5, 18, 24, 87, 111, 114, 107, 102, 108, 111, 119, 32, 116, 97, 115, 107, 32, 110, 111, 116, 32, 102, 111, 117, 110, 100, 46, 26, 66, 10, 64, 116, 121, 112, 101, 46, 103, 111, 111, 103, 108, 101, 97, 112, 105, 115, 46, 99, 111, 109, 47, 116, 101, 109, 112, 111, 114, 97, 108, 46, 97, 112, 105, 46, 101, 114, 114, 111, 114, 100, 101, 116, 97, 105, 108, 115, 46, 118, 49, 46, 78, 111, 116, 70, 111, 117, 110, 100, 70, 97, 105, 108, 117, 114, 101], metadata: MetadataMap { headers: {"content-type": "application/grpc", "server": "temporal", "date": "Thu, 31 Jul 2025 07:46:46 GMT"} } run_id="01985f6f-b5ea-73a8-bb25-93b37c8db50d"

[koliaVasyliv]

hey, @mjameswh have you been able to look into this issue? it's blocking us to upgrade to newer versions.

[mjameswh]

Hey! Sorry, I was out for a few days.

Unfortunately, I haven't been able to reproduce this error myself, and I don't observe any issue with any of our regular performance tests. It also seems like this is affecting only a very small subset of users, which suggests there might be some particular conditions required to reproduce this.

The log entry reported above isn't helping either; the "details" shown contain nothing more than the error status and message that are already printed out as plain text, just above. And in any case, the Workflow task not found. error is almost certainly just a secondary symptom; the real problem is that the Worker is not completing task within the allowed time (i.e. 10 seconds by default). The performance problem is what needs to be figured out.

Before anything else, I would recommend trying this again, with v1.13.0. That release notably fixed a bug that could surface as an apparent memory leak in some Worker configurations. The increased memory pressure caused by this bug might have increased time spent in GC, thus considerably reducing the Worker's performance.

If this is not fixed by v1.13.0, could you please provide the following:

• Your Runtime's options
• Your Worker's options
• Graphs of your CPU usage and Memory usage
• When does that happen? Is that after the Worker's has been quietly running for a long time? Are you seeing many of those errors in short time, then no more for some time, and then again?
• Anything in your Worker's logs that seems potentially pertinent.