Skip to content

feat(redteam): public as_target_session and default run_attack max_turns Part 3 - #417

Open
nhungbi wants to merge 4 commits into
strands-agents:mainfrom
nhungbi:feat/redteam-as-target-session
Open

nhungbi wants to merge 4 commits into
strands-agents:mainfrom
nhungbi:feat/redteam-as-target-session

Conversation

@nhungbi

@nhungbi nhungbi commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Description

Writing a custom red-team task, for example one that builds a fresh target per case or drives
strategies outside RedTeamExperiment, needed two things that were private or manual:

  1. as_target_session(target) (new public helper). It wraps a strands.Agent in
    StrandsAgentSession and a MultiAgentBase (Graph/Swarm) in StrandsMultiAgentSession,
    and returns a ready TargetSession unchanged. Anything else raises TypeError. The wrapping
    logic (_build_session) moves from task.py to strategies/target_session.py, and the
    parallel task runner now uses the public helper. The helper is exported from strategies and
    from strands_evals.experimental.redteam.

    session = as_target_session(build_agent())
    result = strategy.run_attack(case, session)
  2. run_attack's max_turns defaults to MAX_ALLOWED_TURNS. Custom tasks previously had to
    pass max_turns=MAX_ALLOWED_TURNS on every call, and nothing enforced the cap.

    • MAX_ALLOWED_TURNS = 50 moves from task.py to strategies/base.py and is exported from
      strategies and the package root. It has to live in base.py because task.py imports
      strategies. task.MAX_ALLOWED_TURNS is still re-exported, so existing imports work.
    • AttackStrategy.MAX_ALLOWED_TURNS (a ClassVar) and a _turn_cap(max_turns, own=None)
      helper. The helper uses the cap when max_turns is None, clamps larger values to it, and
      then applies the strategy's own budget when that is smaller.
    • All built-in strategies now take max_turns: int | None = None and call _turn_cap.
    • The task runner still passes max_turns explicitly, now as strategy.MAX_ALLOWED_TURNS.
      Custom strategies written against the old contract declare max_turns with no default, and
      omitting it would raise a TypeError for them.

Behavior change

Calling a built-in strategy's run_attack directly with max_turns > 50 now clamps to 50 instead
of honoring the value. Experiments run through RedTeamExperiment are unaffected, because the
runner already passed 50.

Known limitation

Python does not inherit a parameter's default into an override. A user's custom strategy gets the
default and the clamp only if it declares max_turns: int | None = None and calls
self._turn_cap(...). Making this automatic for every subclass, for example by wrapping
run_attack in __init_subclass__, is left out of this PR to keep it small.

Related Issues

Documentation PR

N/A. The module README, the as_target_session docstring example and AGENTS.md are updated in
this PR.

Type of Change

New feature

Testing

  • New test_target_session.py tests for as_target_session: wrapping an Agent or a
    MultiAgentBase, returning a TargetSession subclass unchanged instead of re-wrapping it, and
    raising TypeError for an object without a trace list.

  • New test_strategies.py tests: omitting max_turns gives 50, max_turns=100 is clamped to 50,
    _turn_cap resolution cases, and MAX_ALLOWED_TURNS importable from base, strategies,
    task and the package root.

  • I ran hatch run prepare

Checklist

  • I have read the CONTRIBUTING document
  • I have reviewed and understand every line of code in this PR, including any generated by AI tools, and I can explain why it works
  • My change is focused and reasonably small; I have split unrelated work into separate PRs
  • I have added any necessary tests that prove my fix is effective or my feature works
  • I have updated the documentation accordingly
  • I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@nhungbi

nhungbi commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

@strandly-the-agent review the changes

@github-actions github-actions Bot added the enhancement New feature or request label Oct 8, 2026
@nhungbi
nhungbi deployed to auto-approve October 8, 2026 20:26 — with GitHub Actions Active
@github-actions github-actions Bot added area-redteam Red teaming: adversarial generation, attack strategies, attack success evaluation area-devx Developer experience: papercuts, confusing public APIs, error messages, ergonomics, usability labels Oct 8, 2026

@strandly-the-agent strandly-the-agent left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested — no blockers; two API-shape gaps in the new surface and some doc drift. Both features work as claimed (pure move of _build_session, _turn_cap resolves correctly, 4/4 mutants killed). The two things worth fixing before un-drafting: as_target_session can't capture a baseline, so a custom task that shares one target gets a reset() that leaks agent.state between cases (repro below); and the runner now passes strategy.MAX_ALLOWED_TURNS, so a subclass ClassVar lifts the "hard" 50 cap the README still promises (runner passed 10 000 in my repro). Inline threads carry suggestions. Also: this changes a public ABC signature and adds root exports, with no needs-api-review label.

✅ Verified (head ecdfc89)
  • pytest tests/strands_evals/experimental/redteam → 359 passed; ruff check/format --check clean; mypy src/strands_evals/experimental/redteam clean.
  • Mutation checks on the new tests — all 4 killed: _turn_cap ignores own; _turn_cap doesn't clamp; runner passes max_turns=None; as_target_session returns the target unwrapped.
  • _build_session body in target_session.py:366-395 is textually identical to the block removed from task.py; as_target_session(make_target()) ≡ old _build_session(make_target(), baseline=None).
  • Repro, shared target: agent.state.set("mode","x") after as_target_session(agent), then session.reset() → state.mode still "x", messages cleared. Same with StrandsAgentSession(agent, baseline=agent.take_snapshot(preset="session")) → restored.
  • Repro, cap: subclass with MAX_ALLOWED_TURNS = 10_000 driven through task._run_attack → strategy received max_turns=10000; on main the runner always passed 50.
  • mypy on a user subclass still declaring max_turns: int (MYPYPATH=src): new Liskov error on this branch, none on main.
Questions (non-blocking)
  1. Default on an abstract method. base.py:78 gives run_attack a default that Python won't inherit into overrides, and base.py:96 documents an underscore method (_turn_cap) as the thing implementers must call. Would a template method be simpler — public non-abstract run_attack that resolves the cap and calls an abstract _run_attack(..., max_turns: int)? Subclasses then get a resolved int and can't skip the clamp. It's a bigger rename for existing subclasses, so a maintainer call — but if the subclass contract breaks anyway (mypy Liskov already does), once in the final shape beats twice.
  2. Silent clamp of an explicit value. run_attack(..., max_turns=100) was honoured and now silently becomes 50. AGENTS.md:491 asks for a DeprecationWarning cycle on changed defaults; a logger.warning when max_turns is not None and max_turns > cap (matching target_session.py:46) would be the cheap version. The None path should stay silent.
  3. Why keep _build_session at all? as_target_session is return _build_session(target) plus a duplicated docstring, and task.py:16 imports the private name across modules for one call. One as_target_session(target, *, baseline=None) would remove both (and answers the first inline thread).
Reading order

strategies/base.py (constant, ClassVar, _turn_cap) → task.py:150 → strategies/target_session.py:344-395 → one strategy (pair/__init__.py:148-156) as representative of the six → tests.

Appendix — non-blocking (7)
  • README.md:76 still shows run_attack(case, target_session, *, max_turns, model) as if max_turns were required; :87 says it can be omitted. Suggest max_turns=None, model=None.
  • SKILL.md:429 still says "task.py enforces MAX_ALLOWED_TURNS = 50"; SKILL.md:421 lists the session symbols without as_target_session. README/AGENTS.md were updated, SKILL.md wasn't.
  • README.md:229 still describes task.py as "wraps Agent / MultiAgentBase into a TargetSession" — that moved. as_target_session has zero README mentions despite being the custom-task entry point.
  • task.py:15 re-export alias: MAX_ALLOWED_TURNS wasn't in any __all__ before this PR and everything else in task.py is private — the alias (and test_strategies.py:103 pinning it) is maintenance for a path nobody public had. Experimental module; dropping it is cheaper.
  • Only PromptStrategy is tested with max_turns omitted (test_strategies.py:109-121); the other five changed signatures aren't. A parametrize over BUILTIN_STRATEGIES closes it.
  • bad_likert_judge/__init__.py:116 rebinds the parameter (max_turns = self._turn_cap(max_turns)) where the other five assign turn_cap; works, just the odd one out.
  • User subclasses declaring max_turns: int now fail mypy (Liskov) — the package ships no py.typed, so most users won't see it; worth one line in the PR's "Known limitation".

Comment thread src/strands_evals/experimental/redteam/strategies/target_session.py
Comment thread src/strands_evals/experimental/redteam/task.py Outdated
Comment thread src/strands_evals/experimental/redteam/strategies/target_session.py Outdated
@nhungbi nhungbi self-assigned this Oct 9, 2026
@nhungbi
nhungbi deployed to auto-approve October 9, 2026 17:45 — with GitHub Actions Active
@nhungbi
nhungbi marked this pull request as ready for review October 9, 2026 17:46
@nhungbi
nhungbi requested a review from a team as a code owner October 9, 2026 17:46
@nhungbi
nhungbi requested review from notowen333 and removed request for strandly-the-agent October 9, 2026 17:46
@nhungbi

nhungbi commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

@strandly-the-agent review the changes

@strandly-the-agent strandly-the-agent left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve — all three round-1 findings fixed at 20d153d, and the fix is simpler than what I suggested. Dropping the ClassVar and _turn_cap entirely (plain max_turns: int = MAX_ALLOWED_TURNS, runner passes the module constant) removes the cap escape, the silent-clamp behaviour change, and the private-method-as-contract question in one move. The as_target_session shared-target leak is handled by the docstring Note rather than a capture_baseline knob — fine for Part 3, and the README now points custom tasks at a fresh target per case.

Still no needs-api-review label for a public ABC signature change + two new root exports; not blocking from me, just flagging for whoever merges.

✅ Verified (head 20d153d, delta vs ecdfc89 is fixes-only)
  • Round-1 🟡 cap escape — fixed: subclass with MAX_ALLOWED_TURNS = 10_000 driven through task._run_attack now receives max_turns=50 (was 10 000).
  • Round-1 🟡 shared-target reset() — documented (target_session.py:352-353); behaviour unchanged by design, README/docstring example build a fresh target per case.
  • Round-1 ⚪ error message — fixed: TypeError: target must be …; test tightened to match="^target must be".
  • Appendix items also picked up: README.md:76/87/229, SKILL.md:421/429 updated; parametrized default test over all six strategies + the ABC (test_strategies.py); explicit max_turns=80 is now honoured and tested.
  • pytest tests/strands_evals/experimental/redteam → 365 passed; ruff check/format clean; mypy clean.
  • Mutation: changing one strategy's default to 49 fails the new parametrized test, so it discriminates.
  • Attacked the fix: removing the clamp restores pre-PR semantics for direct callers (no behaviour change left to deprecate); bad_likert_judge max_turns < 2 guard still reached. Nothing new found.

This branch is waiting to be deployed

1 active and 1 waiting deployments
manual-approval — 20d153df Waiting Oct 9, 2026 by nhungbi via Trigger Strands Review #604
auto-approve — 20d153df Deployed Oct 9, 2026 by nhungbi via Run integration tests #950
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-devx Developer experience: papercuts, confusing public APIs, error messages, ergonomics, usability area-redteam Red teaming: adversarial generation, attack strategies, attack success evaluation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants