Make a Google ADK (adk-go) agent durable
and replay-safe under Temporal without rewriting the agent. The agent's
orchestration loop runs inside a Temporal Workflow; each LLM call becomes a
durable Temporal Activity, and any tool that does I/O opts into an Activity
too — so calls are retried, timed-out, visible in the Temporal UI, and replayable.
You keep building agents the native ADK way — llmagent.New(...) with a
model.LLM, tool.Tools / tool.Toolsets and SubAgents, wrapped in
runner.New(...) and driven by r.Run(...). You change two things:
- Use
googleadk.NewModel("<model-name>")as your agent'sModel. It is amodel.LLMwhose calls are dispatched to theInvokeModelActivity; the real model is reconstructed worker-side (never in the workflow). - Pass the bridged context from
googleadk.NewContext(workflowCtx)tor.Run, which installs Temporal-deterministic time / UUID / task-fan-out providers.
Tools run in-workflow by default — the idiomatic Temporal model: your workflow
is deterministic, and anything that touches the network, clock, or disk goes
through an Activity. Opt a tool into an Activity with googleadk.ActivityAsTool,
or use googleadk.NewMCPToolset for MCP. The real model and any activity/MCP tool
handlers live worker-side in the registry declared by googleadk.Config and
wired onto the worker by googleadk.NewPlugin(...).
The module path is go.temporal.io/googleadk, replacing
go.temporal.io/sdk/contrib/googleadk.
From your application's Go module, run:
go get go.temporal.io/googleadk@latestimport "go.temporal.io/googleadk"This package depends on the deterministic ADK platform seams
(WithTimeProvider, WithUUIDProvider, WithTaskRunner), tool/toolutils.PackTool,
and the model.NewLLM registry lookup from upstream google.golang.org/adk/v2
(the registry itself stays application-owned; this package never registers into it).
Those seams have been in tagged ADK releases since v2.1.0; go.mod requires
v2.3.0 for compatibility with OpenTelemetry log SDK v0.21.0. Request-order
confirmation resume (google/adk-go#1169), available since v2.2.0, makes
multi-decision confirmation resumes replay-stable. ADK sets the Go floor:
1.26.6+. The replay-safe telemetry gate composes workflow.IsReadOnly.
Runnable end-to-end samples live in
temporalio/samples-go.
Those upstream samples use the former module path; update their Google ADK
import to go.temporal.io/googleadk when using this module.
Two halves: the worker registers the real model handler as an Activity; the workflow builds a vanilla ADK agent and drives it.
package main
import (
"context"
"log"
"go.temporal.io/sdk/client"
"go.temporal.io/sdk/worker"
"go.temporal.io/sdk/workflow"
"google.golang.org/adk/v2/agent"
"google.golang.org/adk/v2/agent/llmagent"
"google.golang.org/adk/v2/model"
"google.golang.org/adk/v2/model/gemini"
"google.golang.org/adk/v2/runner"
"google.golang.org/adk/v2/session"
"google.golang.org/genai"
"go.temporal.io/googleadk"
)
const taskQueue = "adk"
// AgentWorkflow runs a native ADK agent. The model call inside r.Run is
// dispatched to a Temporal Activity by googleadk.NewModel.
func AgentWorkflow(ctx workflow.Context, question string) (string, error) {
// The model is a TemporalModel: in-workflow it only carries the model name;
// the real gemini model is reconstructed worker-side by the ModelFactory.
root, err := llmagent.New(llmagent.Config{
Name: "assistant",
Description: "a helpful assistant",
Model: googleadk.NewModel("gemini-2.0-flash"),
Instruction: "Answer concisely.",
})
if err != nil {
return "", err
}
r, err := runner.New(runner.Config{
AppName: "hello",
Agent: root,
SessionService: session.InMemoryService(),
AutoCreateSession: true,
})
if err != nil {
return "", err
}
// NewContext bridges the workflow.Context into the context ADK reads its
// determinism/executor seams from. Pass it straight to Run.
adkCtx := googleadk.NewContext(ctx)
msg := genai.NewContentFromText(question, genai.RoleUser)
var answer string
for ev, err := range r.Run(adkCtx, "user-1", "session-1", msg, agent.RunConfig{}) {
if err != nil {
return "", err
}
if ev != nil && ev.Content != nil {
for _, p := range ev.Content.Parts {
if p != nil && p.Text != "" {
answer = p.Text
}
}
}
}
return answer, nil
}
func main() {
c, err := client.Dial(client.Options{})
if err != nil {
log.Fatal(err)
}
defer c.Close()
// Worker-side registry, wired as a worker plugin: the real model lives here.
// API keys are captured in the factory closure and never cross the Activity
// boundary. The plugin registers the Activities at worker start and closes
// cached MCP toolsets at worker stop.
adkPlugin, err := googleadk.NewPlugin(googleadk.Config{
Models: map[string]googleadk.ModelFactory{
"gemini-2.0-flash": func(ctx context.Context, name string) (model.LLM, error) {
// nil config reads GEMINI_API_KEY / GOOGLE_API_KEY from the env, worker-side.
return gemini.NewModel(ctx, name, nil)
},
},
})
if err != nil {
log.Fatal(err)
}
w := worker.New(c, taskQueue, worker.Options{Plugins: []worker.Plugin{adkPlugin}})
w.RegisterWorkflow(AgentWorkflow)
if err := w.Run(worker.InterruptCh()); err != nil {
log.Fatal(err)
}
}Config.Models is optional: when a model name is absent, InvokeModel falls
back to ADK's model registry (model.NewLLM), which stays application-owned —
this package never registers into it, so your own model.Register calls are
honored — and gemini-* names the registry does not know resolve via a
built-in zero-config Gemini fallback (a nil config reads GEMINI_API_KEY /
GOOGLE_API_KEY worker-side). Supply a factory to inject credentials, disable
the model SDK's own retries (see below), or override the fallbacks.
- Every LLM turn is a durable Activity.
NewModel(name)returns amodel.LLMthat dispatches to theInvokeModelActivity. The workflow only ships the model name; the Activity reconstructs the model from yourModelFactory. - Deterministic tools, in-workflow by default. Ordinary
functiontool.New(...)tools run on Temporal's deterministic dispatcher inside the workflow — no Activity overhead, and their session-state mutations (ctx.State().Set,ctx.Actions()) propagate normally. - Issue workflow commands from a tool.
WorkflowContext(ctx)returns theworkflow.Contextthe bridged context dispatches on, so an in-workflow tool can start a child workflow, set a timer, or signal another execution. Use the returned Context rather than closing over the workflow function's own: during concurrent fan-out the tool runs on a different coroutine, and blocking with another coroutine's Context can panic or stall workflow execution. The accessor reports false outsideNewContext(a local ADK run), so a tool can fall back to a non-durable path. - Opt a tool into an Activity when it does I/O.
ActivityAsTool(myActivity, ...)exposes an existingfunc(context.Context, TArgs) (TResults, error)Temporal activity to the agent as a tool (parameter schema inferred fromTArgs); itsRundispatches the activity. Register the same activity on the worker as usual. - MCP, statelessly.
NewMCPToolset(...)is a workflow-side proxy: it lists remote tools (full declarations, including parameters) viaListMcpToolsand executes calls viaCallMcpTool. The live, statefulmcptoolset.New(...)runs worker-side, never in the workflow. YourMCPFactoryruns at most once per toolset name — the toolset is cached and shared across calls — and the plugin closes any cached toolset that implementsClose() errorat worker stop automatically; with manual wiring, callActivities.Closeyourself afterworker.Runreturns. - Deterministic by construction.
NewContextbinds ADK'splatform.Nowtoworkflow.Now,platform.NewUUIDto a deterministic generator drawn from the workflow random stream, andplatform.RunTasksto aworkflow.Gofan-out, so the agent loop replays deterministically. Concurrent tool fan-out is on by default; useNewContext(ctx, googleadk.WithSequentialToolFanout())for a serial fallback. - Typed failures. Model / tool / MCP failures surface as Temporal
ApplicationErrors taggedgoogleadk.ModelError/.ToolError/.McpError. Classify withIsNonRetryable(err); never string-match. Upstream HTTP status drives retryability (408/409/429/5xxretryable, other4xxnot). - Disable model-SDK retries in your
ModelFactory.InvokeModelalready runs under Temporal'sRetryPolicy; leaving the model client's own retries on retries a transient failure twice over. Let Temporal own retries. - Test without a live LLM.
testing.goshipsFakeModel,FakeMCPServer, andTextResponse/FunctionCallResponseso you can unit-test workflows with no network. The test environments do not run plugins; register the Activities directly withNewActivities+Register(as this repo's own tests do).
Determinism note. Because plain tools run in-workflow, their code must be deterministic and replay-safe — no direct network, clock, randomness, or goroutines. Anything that isn't belongs in an
ActivityAsTool(or an MCP tool), where it runs worker-side under Temporal's retry/timeout policy.
A tool that needs approval calls ADK's ctx.RequestConfirmation(hint, payload).
ADK records the request and emits a function call named adk_request_confirmation,
ending the turn. Because the tool runs in-workflow, the request lands in the
workflow's own event actions. MCP tools participate too: a confirmation the
worker-side tool requests (e.g. via mcptoolset's RequireConfirmation option)
is tunneled back across the Activity boundary and re-recorded workflow-side, so
the agent pauses the same way. Drive it from your workflow like this:
for {
var events []*session.Event
for ev, err := range r.Run(adkCtx, userID, sessionID, msg, agent.RunConfig{}) {
if err != nil {
return err
}
events = append(events, ev)
}
pending := googleadk.PendingConfirmations(events)
if len(pending) == 0 {
break // done
}
// Ask the human. Deliver decisions via a Temporal signal or update.
var decisions []googleadk.ConfirmationDecision
for _, p := range pending {
var d googleadk.ConfirmationDecision // {FunctionCallID, Confirmed}
// e.g. workflow.GetSignalChannel(ctx, googleadk.ConfirmationSignalName).Receive(ctx, &d)
decisions = append(decisions, d)
}
// Resume: re-run with the confirmation responses.
msg = googleadk.ConfirmationResponse(decisions...)
}PendingConfirmations exposes each pending call's OriginalCall and Hint for
your UI; ConfirmationResponse builds the resume message ADK expects.
A conversation's history lives in the ADK session. To keep a workflow's history bounded, snapshot the session and continue-as-new:
if workflow.GetInfo(ctx).GetContinueAsNewSuggested() {
snap, err := googleadk.ExportSession(adkCtx, svc, appName, userID, sessionID)
if err != nil {
return err
}
return workflow.NewContinueAsNewError(ctx, AgentWorkflow, snap /* + next input */)
}On the next run, rebuild the session before driving the agent:
svc := session.InMemoryService()
if snap != nil {
if _, err := googleadk.ImportSession(adkCtx, svc, snap); err != nil {
return err
}
}SessionSnapshot is JSON-serializable (session-scoped state + full event
history); every value in session state and every tool result must be
JSON-encodable. App/user-scoped state (managed across sessions by the session
service) is not carried — use a durable session service for that.
NewModel(name, googleadk.WithStreaming(topic, 0)) drives the model in streaming
mode: the InvokeModel Activity calls the model with stream=true, heartbeats,
and publishes each chunk to a per-run
workflowstreams
topic for external (UI) consumers, then returns the aggregated final response into
the workflow so replay stays deterministic.
Call googleadk.StreamServer(ctx) once near the top of the workflow that drives
r.Run, and set agent.RunConfig{StreamingMode: agent.StreamingModeSSE}:
func StreamingAgentWorkflow(ctx workflow.Context, q string) (string, error) {
if err := googleadk.StreamServer(ctx); err != nil { // required when streaming
return "", err
}
topic := "run-" + workflow.GetInfo(ctx).WorkflowExecution.ID
root, _ := llmagent.New(llmagent.Config{
Model: googleadk.NewModel("gemini-2.0-flash", googleadk.WithStreaming(topic, 0)),
// ...
})
// ... build runner, set agent.RunConfig{StreamingMode: agent.StreamingModeSSE}, drive r.Run as above
}External consumers read chunks with workflowstreams.NewClient(c, workflowID, ...).Subscribe(...).
The bidirectional RunLive path (hard-coded goroutines/channels) is not supported.
This integration's plugin only registers its Activities at worker start and
closes cached MCP toolsets at worker stop — no interceptors, no data converter —
so it composes with other entries in worker.Options.Plugins (e.g. interceptor-
or converter-based plugins like sdk-go/contrib/opentelemetry) without conflict.
On the ADK side, add other ADK plugins to runner.PluginConfig.Plugins as usual.
ADK also emits its own OpenTelemetry telemetry from inside the workflow — see the
next section before wiring exporters.
ADK records its telemetry through the OpenTelemetry process globals from code
that runs inside the workflow: at this adk-go pin that means spans (with
token usage as gen_ai.usage.* span attributes and span durations as latency)
and gen_ai.* log events; OTel metrics are pending upstream
(adk-go#479). Workflow code
re-executes on every history replay — worker restarts, redeploys, sticky-cache
eviction — and each replay re-reads the recorded Activity results from history and
re-emits identical telemetry, so observed counts inflate by one full copy per
replay, without bound over a workflow's lifetime. Temporal's replay-safe telemetry
(workflow.GetMetricsHandler, replay-gated tracing interceptors) never applies,
because ADK bypasses it via the OTel globals it captured at package init.
Wrap your real providers in this package's replay-safe wrappers and install them
as the globals. The log and metric wrappers drop any emission whose context
carries a replaying workflow (recovered from the context bridged by
googleadk.NewContext); the tracer wrapper re-creates workflow spans during
replay and suppresses their End instead (the span contract below). Everything
else — worker, client, and Activity telemetry — delegates unchanged:
import (
"context"
"go.opentelemetry.io/otel"
otellogglobal "go.opentelemetry.io/otel/log/global"
sdklog "go.opentelemetry.io/otel/sdk/log"
sdkmetric "go.opentelemetry.io/otel/sdk/metric"
"go.temporal.io/googleadk"
)
func main() {
// NewReplaySafeTracerProvider builds and owns the tracer provider, so it
// always installs the span-ID generator that gives a workflow span
// re-created on replay the same trace and span IDs it drew on first
// execution (span contract below). Pass the usual sdktrace options
// (exporters, resource, sampler). The logger and meter wrappers wrap your
// own providers.
tracerProvider := googleadk.NewReplaySafeTracerProvider( /* sdktrace.WithBatcher(exporter), ... */ )
myLoggerProvider := sdklog.NewLoggerProvider( /* processors ... */ )
myMeterProvider := sdkmetric.NewMeterProvider( /* readers ... */ )
// Must be the FIRST global providers set in the process: ADK captures the
// global proxy tracer/logger at package init, and the proxy binds its
// delegate on the first Set call only — a provider installed earlier
// permanently bypasses the wrappers.
otel.SetTracerProvider(tracerProvider)
otellogglobal.SetLoggerProvider(googleadk.NewReplaySafeLoggerProvider(myLoggerProvider))
otel.SetMeterProvider(googleadk.NewReplaySafeMeterProvider(myMeterProvider))
// You own the tracer provider; shut it down after your clients and workers
// stop so buffered spans flush.
defer func() { _ = tracerProvider.Shutdown(context.Background()) }()
// ... Temporal client, worker, googleadk.NewPlugin wiring as usual ...
}googleadk.NewPlugin logs a best-effort warning at worker start and at
workflow replayer creation when a global provider is a raw OTel SDK provider
installed unwrapped.
Only that positively-recognized case warns; wrapped, no-op, never-set, and
custom providers stay silent. The check is best-effort — the OTel global proxy
binds its delegate on the first Set*Provider call permanently, so it cannot
see through a process that sets a global more than once — but a raw SDK
provider installed once and never wrapped, the realistic misconfiguration, is
recognized.
With the wrappers installed, replays add nothing. Point telemetry (log events,
metric recordings) is recorded on first execution, at-least-once rather
than exactly-once: a workflow task that fails or times out after emitting
re-executes live and records again — the same semantics and caveat as
workflow.GetMetricsHandler.
Spans get the same at-least-once contract through a different mechanism,
because ADK holds each span open across the model call's Activity await. A
span started from sequenced workflow code is a real span in every execution
mode — a replay re-creates it with a workflow-time start and the same trace and
span IDs it drew on first execution (the owned provider always installs the
span-ID generator) — but its End is suppressed while the workflow is
replaying. Tracers derived from a workflow span
(span.TracerProvider().Tracer(...)) produce spans under the same contract.
Each span is therefore exported by whichever execution reaches its End live:
- Sticky-cache eviction (graceful): eviction teardown exports nothing —
the SDK marks teardown as replay before coroutine defers run, so ADK's
deferred force-End of a still-open span is suppressed. The catch-up replay
on the next workflow task re-creates the span and its live
Endexports it exactly once, complete: agenerate_contentspan evicted mid-model-call keeps itsgen_ai.usage.*token attributes, its parent linkage, and its original identity. Withworker.SetStickyWorkflowCacheSize(0)(orWORKFLOW_CACHE_SIZE=0in this repo's tests) every model call straddles an eviction and span counts, attributes, and point telemetry all stay exact. - Worker shutdown, crash, or redeploy: a span open at that moment is
recovered the same way — the catch-up replay on the next worker re-creates
it and exports it once when its
Endruns live. Spans are lost only when their workflow never continues (terminated or timed-out runs). - Workflow task retry: a task that fails after a span ended re-executes live and exports the span again — at-least-once, the caveat all point telemetry shares — but the copies carry one span ID, so ID-deduplicating trace backends collapse them; span-count pipelines (e.g. the spanmetrics connector) see one extra copy per retried task, the same as they do for retried metric and log recordings.
NewReplaySafeMeterProvider is forward-looking: it gates synchronous
instrument recordings and reports their Enabled false while suppressed
(observable instruments pass through — their callbacks never run under a
workflow context), covering both your own workflow-side recordings through
the global meter today and ADK's metrics once adk-go#479 lands.
Telemetry from query handlers and update validators always records:
the gate composes workflow.IsReplaying with !workflow.IsReadOnly
(Experimental). Both are once-per-request operations that never re-execute
from history — a query served right after a catch-up replay (e.g. an agent
awaiting InvokeModel on a restarted worker) still observes IsReplaying
true, because the flag retains whatever the last processed history event left
there, but IsReadOnly excludes read-only contexts from suppression, so the
recording is kept rather than lost. Side-effect functions are read-only
contexts too, harmlessly so: they never execute during replay at all — their
recorded markers supply the value.
OTel Logs API status: NewReplaySafeLoggerProvider is built on the
pre-1.0 go.opentelemetry.io/otel/log (v0.19.x at this pin), which may
change shape between minor releases; upgrading it can require a matching
upgrade of this package. A surface test in this package fails on any method
an upgrade would newly pass through ungated.
- Supported: single- and multi-agent (
SubAgents) trees, in-workflow function tools,ActivityAsTool, stateless MCP, Gemini built-in tools (executed server-side insideInvokeModel), HITL tool confirmation, continue-as-new state carry, the in-memory session service, and SSE streaming. - Not yet:
RunLive(bidirectional streaming), sub-agent-as-child-workflow, live memory/artifact tools that require in-workflow network I/O, and DB/Vertex session services. These raise or are documented rather than silently degrading.