feat(schema): add max_tokens field to AgentDef for per-agent override

[nskun (nskun)]

Summary

Add max_tokens to AgentDef, enabling per-agent output tokens limit configuration.

Background

agent_builder.py already has the fallback logic, but the field was missing from the schema:

agent_max_tokens = getattr(agent, "max_tokens", None)
max_tokens = agent_max_tokens if agent_max_tokens is not None else default_max_tokens

Since AgentDef uses extra="forbid", specifying max_tokens in YAML caused a validation error, and getattr always returned None.

Changes

・Add max_tokens: int | None = Field(None, ge=1, le=200000) to AgentDef (matches RuntimeConfig.max_tokens constraints)
・Add forbidden-field checks in validate_agent_type for non-LLM agent types: script, workflow, wait, set, terminate
・Add TestAgentDefMaxTokens(11 tests)

Usage

agents:
  - name: creator
    max_tokens: 32768
    prompt: !file prompt_creator.md

  - name: reviewer
    max_tokens: 8192
    prompt: !file prompt_reviewer.md

When omitted inherits runtime.max_tokens as before. No logic change needed in agent_builder.py

Closes #470

[nskun (nskun)]

@microsoft-github-policy-service agree

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@2dbb712). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #471   +/-   ##
=======================================
  Coverage        ?   91.90%           
=======================================
  Files           ?      144           
  Lines           ?    23272           
  Branches        ?        0           
=======================================
  Hits            ?    21389           
  Misses          ?     1883           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Jason Robert (jrob5756)]

The schema work here is solid. The field sits in the right place next to max_agent_iterations and session_key, the bounds agree with both RuntimeConfig.max_tokens and the runtime assertion at claude.py:365, and using is not None instead of copying the truthiness check on the line above was the right call. Ruff, ty and the full test_config suite are all green.

My concerns are all about what happens once the schema has accepted the value.

agent_builder.py:307 is the only place in the repo that reads this field, and it is reachable from claude.py alone. copilot, which is the default provider, has no max_tokens reference anywhere. hermes:283 reads only the runtime value. aca's AcaAgentPayload is extra="forbid" and does not list the field, so it cannot cross the wire. On four of five providers this validates clean and then quietly does nothing. The two fields immediately above it in AgentDef are plumbed into all five (copilot:1296,1301, hermes:247,252, claude_agent_sdk:883,891, aca:763,764), so the surrounding convention sets a different expectation.

On the one provider that does read it, _coerce_for_thinking can raise the value without saying so. max_tokens=1 with reasoning.effort=max comes out as 64000, and where the clamp-down branch at agent_builder.py:244 logs at INFO, the up-bump at :240-242 logs nothing at any level. A small per-agent cap on an agent that also sets reasoning is the most likely way anyone reaches for this field, so in its most probable configuration it does the reverse of what it advertises.

There is also an asymmetry with claude-agent-sdk. factory.py:195 raises ProviderError for runtime.max_tokens, and the comment above it says silently dropping the value "would quietly violate user intent". The per-agent field never reaches that check, so moving one line two levels deeper in the same YAML file gets past the guard. That also makes docs/providers/comparison.md:170 inaccurate, since it currently states both settings are rejected at the factory.

A ProviderCapabilities flag plus a check in _check_agent_capabilities would settle the first and third points together. session_key, max_session_seconds, working_dir and skills all use that shape already, and it would catch claude-agent-sdk at validate time rather than partway into a run.

Smaller things: the restriction lists in docs/workflow-syntax.md (lines 936, 1022, 1162) enumerate every rejected field and now omit this one, and the same lists appear in plugins/conductor/skills/conductor/references/authoring.md and yaml-schema.md, which ship to agents as skill content. There is no CHANGELOG.md entry under Unreleased.

Nothing above is a criticism of the mechanics, which are clean. Happy to look again once the provider question is settled.

[Jason Robert (jrob5756)]

This validates clean on copilot, hermes, aca and claude-agent-sdk, and then has no effect on any of them. agent_builder.py:307 is the only read in the repo and it only runs under claude.

The two fields directly above this one are honoured by every provider (copilot:1296,1301, hermes:247,252, claude_agent_sdk:883,891, aca:763,764), so anyone reading AgentDef would expect the same treatment here.

Two ways to close it. Either wire up the remaining four providers, or declare the support explicitly and let validation refuse the rest. The second is what session_key and max_session_seconds already do:

# providers/capabilities.py
max_tokens: bool = False
"""``True`` when the provider applies a per-agent ``max_tokens`` output cap.

``False`` means the value would be silently ignored, so workflows that set it
fail validation instead."""

# config/validator.py, inside _check_agent_capabilities
if agent.max_tokens is not None and not caps.max_tokens:
    errors.append(
        f"Agent '{agent.name}' sets max_tokens={agent.max_tokens!r} but provider "
        f"'{provider_name}' does not apply per-agent output token caps "
        f"(capabilities.max_tokens=False). Remove it, use runtime.max_tokens where "
        f"the provider honours it, or override the agent to a provider that does."
    )

Then set max_tokens=True on ClaudeProvider and leave the default everywhere else. That also picks up claude-agent-sdk, which today refuses runtime.max_tokens at factory.py:195 but lets this one through.

[Jason Robert (jrob5756)]

human_gate accepts max_tokens as things stand. I checked against a valid gate fixture and it takes the field without complaint, even though it rejects reasoning and session_key. questions accepts it too, which is odder still given it rejects model with the reason "no provider is invoked".

max_session_seconds and max_agent_iterations have the same hole, so this is inherited rather than introduced. But this is the line where max_tokens's contract gets written down, and it currently names the one type the code does not cover.

Suggested change

	Overrides the workflow-level runtime.max_tokens for this agent.
	Only applies to provider-backed agents (not script or human_gate).
	Overrides the workflow-level runtime.max_tokens for this agent. Controls
	response length, not the context window (that budget is context.max_tokens).
	Rejected on script, workflow, wait, set, and terminate steps.

[Jason Robert (jrob5756)]

This block is repeated verbatim five times, and the copying is what let human_gate and questions slip through.

validate_agent_type already has a standalone-guard idiom for this, at lines 1929, 1944 and 1956. The comment on the stdin one states the reasoning outright: being a standalone guard rather than a per-branch check, it also covers the types that have no branch of their own.

The same shape replaces all five and closes the gap:

if self.type not in (None, "agent") and self.max_tokens is not None:
    raise ValueError(
        f"'{self.type}' agents cannot have 'max_tokens' "
        "(only provider-backed agents support this field)"
    )

It is a net reduction in lines, and nothing existing can break: extra="forbid" meant no workflow could carry the field at all before this PR, so the new rejection is strictly tighter than nothing. validator.py:1733 already defines _LLM_AGENT_TYPES = frozenset({None, "agent"}) if you would rather have one source of truth, and schema.py does not import validator.py, so there is no cycle.

Your five existing rejection tests still pass against this, since they only match on the field name.

[Jason Robert (jrob5756)]

All eleven tests here exercise AgentDef.__init__. None of them reach the line that consumes the field, which was dead code until this PR. Delete agent_builder.py:307-308 and this suite still goes green.

tests/test_integration/test_parameter_flow_verification.py was written for this exact worry (its module docstring names it) and already covers the workflow-level value. The cheapest addition is two synchronous tests in test_pydantic_ai_agent_builder.py::TestSamplingSettings:

def test_agent_max_tokens_overrides_workflow_default(self) -> None:
    """A per-agent max_tokens must win over the workflow-level default."""
    agent_def = AgentDef(name="sampler", max_tokens=1000)
    pydantic_agent = build_agent(
        agent_def, system_prompt="", rendered_prompt="", default_max_tokens=4096
    )
    assert pydantic_agent.model_settings["max_tokens"] == 1000


def test_workflow_default_used_when_agent_max_tokens_unset(self) -> None:
    """With no per-agent override the workflow default still applies."""
    agent_def = AgentDef(name="sampler")
    pydantic_agent = build_agent(
        agent_def, system_prompt="", rendered_prompt="", default_max_tokens=4096
    )
    assert pydantic_agent.model_settings["max_tokens"] == 4096

I ran both against this branch and they pass.

One gap in the range coverage too: 0, -100 and 200001 are all tested, but the accepted endpoints 1 and 200000 are not, so swapping ge/le for gt/lt would go unnoticed. TestAgentDefMaxSessionSeconds has test_minimum_boundary for the same reason.

Worth adding a case for reasoning as well. max_tokens=1000 with reasoning.effort=low currently produces model_settings["max_tokens"] == 6144, which may be correct for the Anthropic API but is worth pinning so it cannot drift unnoticed.

[Jason Robert (jrob5756)]

This assertion cannot fail for the reason it looks like it is checking. Pydantic v2 echoes the input dict into the error message, so "max_tokens" in str(exc_info.value) is true for any ValidationError raised on this input. Drop the command kwarg and it still passes, on a completely unrelated "script agents require 'command'" error.

pytest.raises is carrying the test on its own here. The range tests further up (line 687) already match on real message text, so this is inconsistent within the same class rather than a house style:

Suggested change

	assert "max_tokens" in str(exc_info.value)
	assert "script agents cannot have 'max_tokens'" in str(exc_info.value)

Same applies to the workflow, wait, set and terminate cases below.

[nskun (nskun)]

Thank you so much for the incredibly thorough review — I really appreciate the time and care you put into it.
I’ve read through all the feedback, and it’s extremely helpful.
It may take me a couple of days to work through everything and push an update, but I’ll ping you once it’s ready for another look.

[nskun (nskun)]

Thank you again for the detailed review. I’ve pushed an initial update addressing the concrete issues you identified.

My original intent was to support the stable-tier providers, Copilot and Claude. Claude can apply a per-agent output-token setting through Pydantic AI, but the current Copilot SDK does not expose an equivalent setting that Conductor can pass through.

The initial update therefore followed the second approach you suggested: it enabled per-agent max_tokens on Claude only and added checks in conductor validate for unsupported providers.

The current update includes:

• ProviderCapabilities.max_tokens, defaulting to False and currently enabled only on ClaudeProvider
• Capability checks in conductor validate for unsupported providers
• Schema validation for step types that do not invoke a provider
• Exact validation assertions and coverage for the previously missed human_gate and questions cases
• Override precedence, fallback, boundary-value, and reasoning-related tests
• INFO logging for the current automatic reasoning adjustment
• Updates to the related documentation, distributed skill references, provider comparison table, and changelog

The directly related test selection passes (490 passed, 4 skipped).

This is an interim update. I’m still working through the remaining changes and will post a follow-up when the PR is ready for another review.