fix(e2e): close gaps a review found in the paired harness

e2e/bub/README.md

@@ -139,7 +139,9 @@ Each selected workload writes the same layout:
 Shared runs write the same v1 files per source task under `batch-<name>/tasks/<workload-id>/`, plus one aggregate
 evaluation and report at `batch-<name>/`. `collect-all` reports every failed task; `fail-fast` stops only that shared
 Harbor trial at its first failed step. Runtime batch steps are flat and task-prefixed. Each agent invocation starts an
-independent ACP session and Bub tape.
+independent ACP session and Bub tape. Bub keeps every tape as a file in its home, which Harbor does not clear between
+steps, so before each invocation the harness removes the earlier tapes: Bub does not search another session's tape, but
+an agent could read the files.
 
 ## Compare PowerContext off and on
 
@@ -176,7 +178,9 @@ therefore runs only against a Server that requires authentication: it stops befo
 its Scopes to a client without a token. Start the Server with `POWERCONTEXT_SERVER_ACCESS_MODE=enforced` and a
 `POWERCONTEXT_SERVER_AUTH_TOKEN`, and give the harness the same value as `POWERCONTEXT_CLIENT_API_TOKEN`; the harness
 passes it to the ON arm's integration as `POWERCONTEXT_BUB_API_TOKEN` or `POWERCONTEXT_<HOST>_AUTHORIZATION`. The
-token still lets an ON agent read other Scopes on the same Server, including earlier trials'.
+harness holds that value in its own environment and gives Harbor a reference to it, so Harbor's job files record the
+reference and no part of the token. The token still lets an ON agent read other Scopes on the same Server, including
+earlier trials'.
 
 After each ON session the harness records the Scope's Server statistics. When another session follows, it first
 flushes the Scope, standing in for the time that passes between real sessions, and repeats the flush until the Scope
@@ -192,7 +196,10 @@ and that the integration asked PowerContext for context during it. Otherwise it
 flush creates Memory and whether recall returns content are PowerContext's own behavior, so the snapshots record them
 but a run that gets nothing useful still counts as an ON attempt.
 Integration failures and harness or infrastructure errors are reported but left out of success rates and paired
-differences. An agent timeout counts as a failed attempt in either arm.
+differences. A session whose model request failed is such an error on every host: Codex and Claude Code exit non-zero,
+Harbor reads OpenCode's error events, and the harness reads Pi's last message, because Pi exits 0 in the JSON mode
+Harbor uses. The harness reads Pi's output through the logs that Harbor's Docker environment mounts and stops with an
+error when the file is not there. An agent timeout counts as a failed attempt in either arm.
 
 The harness Client waits for each flush, which runs the Server's generation model, so raise its 10-second default
 timeout; the Bub plugin also flushes during a session.
@@ -356,8 +363,8 @@ forwards other native `BUB_*` values without translating them.
 
 If the agent task container requires an outbound proxy, set `POWERCONTEXT_E2E_AGENT_PROXY_URL` to a URL reachable
 from that container. In the fixed nested-container harness, `host-gateway` addresses the harness container, so a
-proxy exposed there can be passed as `http://host-gateway:<port>`. The typed setting is also treated as a secret when
-evidence is written.
+proxy exposed there can be passed as `http://host-gateway:<port>`. The URL can carry credentials, so the harness
+treats it as a secret when evidence is written and gives Harbor a reference to it rather than the value.
 
 The agent container sees only the repository files that installation needs: the `powercontext` package and the host
 integration, and none of them in a paired OFF arm. Workload files, answer keys, and benchmark data stay on the host,
@@ -397,8 +404,12 @@ The harness does not mirror PowerContext Server, PowerContext Client, Bub, Harbo
 loads its native parameters, and the adapter only forwards the native values needed across the nested-container
 boundary. The Bub plugin uses Bub's Pydantic settings extension and accepts the same fields in the `powercontext`
 section of `bub.yml`. Every `*_API_KEY`, `*_TOKEN`, `*_AUTHORIZATION`, and `*_SECRET_ACCESS_KEY` value in the harness
-environment, including Bub's API keys and the PowerContext Client token, is redacted at every final evidence sink,
-whatever its length. Only common placeholders for local model servers, such as `1` or `ollama`, are left in place,
-because they protect nothing and redacting them by substring would rewrite the evidence. CI scans evidence with
-TruffleHog before publishing it. Native ACP artifacts can contain arbitrary command output and should be reviewed before
-sharing.
+environment, including Bub's API keys and the PowerContext Client token, is redacted in every file the harness writes,
+whatever its length. A name with `_TOKEN_` in the middle, such as `AWS_BEARER_TOKEN_BEDROCK`, counts too, and names
+match in any case, but a name ending in `_FILE`, `_PATH`, or `_URL`, such as `AWS_WEB_IDENTITY_TOKEN_FILE`, says
+where a token is and is left alone. Only common placeholders for local model servers, such as `1` or `ollama`, are
+left in place, because they protect nothing and redacting them by substring would rewrite the evidence. CI scans
+evidence with TruffleHog before publishing it. Harbor writes the files under `harbor-jobs/` itself, and the harness
+does not redact them: the job configuration holds references to secrets rather than their values, but every host's
+own output there can contain arbitrary command output, such as an agent printing its environment, and should be
+reviewed before sharing.

e2e/bub/src/powercontext_e2e/harbor_agent.py

@@ -28,6 +28,8 @@
 AGENT_ID = "powercontext-bub-acp"
 REMOTE_BIN_DIR = "/installed-agent/bin"
 REMOTE_BUB_HOME = "/installed-agent/bub-home"
+# Bub keeps every session's messages as JSONL in its home, and nothing else clears them between the steps of a trial.
+BUB_TAPES = '"${BUB_HOME:?}/tapes"'
 REMOTE_BUB_PROJECT = "/installed-agent/bub-project"
 REMOTE_CODEX_AUTH = "/run/agent-auth/codex-auth.json"
 REMOTE_CODEX_HOME = "/installed-agent/codex"
@@ -41,7 +43,10 @@
 
 
 class PowerContextBubAcpAgent(harbor_acp.AcpAgent):
-    """Install Bub through its supported uv tool and plugin commands."""
+    """Install Bub through its supported uv tool and plugin commands, and start every session without Bub's tapes.
+
+    Bub does not search another session's tape, but an agent can read the files, so each session starts without them.
+    """
 
     def __init__(self, **kwargs: Any) -> None:
         self._invocation_scopes = tuple(kwargs.pop("invocation_scopes", ()))
@@ -64,7 +69,10 @@ def __init__(self, **kwargs: Any) -> None:
     @override
     async def run(self, instruction: str, environment: BaseEnvironment, context: AgentContext) -> None:
         try:
+            # The marker comes first: the verifier reads a missing marker as a passed step, so every failure
+            # after this line, including a failed removal, must leave it in place.
             await environment.exec(command=f"touch {STEP_FAILURE_MARKER}")
+            await self.exec_as_agent(environment, command=f"rm -rf {BUB_TAPES}")
             if not self._invocation_scopes:
                 await super().run(instruction, environment, context)
             else:

e2e/bub/src/powercontext_e2e/harbor_pi.py

@@ -16,8 +16,10 @@
 
 from __future__ import annotations
 
+import json
 from typing import Any, override
 
+from harbor.agents.installed.base import NonZeroAgentExitCodeError
 from harbor.agents.installed.pi import Pi
 from harbor.environments.base import BaseEnvironment
 from harbor.models.agent.context import AgentContext
@@ -40,7 +42,7 @@ class PowerContextPiAgent(Pi):
     As on the other hosts, the OFF arm runs without the package. The package reads its Server URL, Scope, consent, and
     Server token from its own environment, which only the ON arm receives. Harbor runs Pi without a saved session, and
     each session starts without the tool output an earlier session saved, so neither arm can read an earlier session
-    from Pi's own files.
+    from Pi's own files. A session whose model request failed is an error rather than an attempt at the task.
     """
 
     def __init__(
@@ -85,6 +87,43 @@ async def install(self, environment: BaseEnvironment) -> None:
     async def run(self, instruction: str, environment: BaseEnvironment, context: AgentContext) -> None:
         await self.exec_as_agent(environment, command=f"rm -f {PI_TOOL_OUTPUT}")
         await super().run(instruction, environment, context)
+        if (failure := self._model_failure()) is not None:
+            raise NonZeroAgentExitCodeError(f"Pi's model request failed: {failure}")  # noqa: TRY003
+
+    def _model_failure(self) -> str | None:
+        """Return why the session's last model request failed, or nothing when it completed.
+
+        Harbor runs Pi in JSON mode, where Pi exits 0 after a failed or aborted model request. Pi's text mode exits 1
+        on the same condition: the last message is an assistant message that stopped on an error. The output must be
+        on the host when this runs; see below.
+        """
+
+        output = self.logs_dir / self._OUTPUT_FILENAME
+        if not output.is_file():
+            # Harbor's Pi agent writes the file in the container and downloads the agent's logs only after this
+            # method runs, so the check reads it through the bind mount of Harbor's Docker environment. An
+            # environment without that mount would otherwise pass every failed session as an attempt.
+            raise RuntimeError(  # noqa: TRY003
+                f"Pi's output {output} is not on the host: the harness reads it before Harbor downloads the "
+                "agent's logs, which requires an environment that mounts /logs"
+            )
+        last: dict[str, Any] = {}
+        # Pi ends each record with LF. str.splitlines would also split at U+2028 and the other separators, which
+        # JSON leaves unescaped inside a string, and each half of a record split there is dropped as invalid JSON.
+        for line in output.read_text(encoding="utf-8", errors="replace").split("\n"):
+            try:
+                event = json.loads(line)
+            except json.JSONDecodeError:
+                continue
+            if (
+                isinstance(event, dict)
+                and event.get("type") == "message_end"
+                and isinstance(event.get("message"), dict)
+            ):
+                last = event["message"]
+        if last.get("role") == "assistant" and last.get("stopReason") in ("error", "aborted"):
+            return str(last.get("errorMessage") or f"request {last['stopReason']}")
+        return None
 
 
 def install_plugin_command() -> str:

e2e/bub/src/powercontext_e2e/hosts.py

@@ -31,6 +31,7 @@
 from .harbor_opencode import OPENCODE_VERSION
 from .harbor_pi import PI_VERSION
 from .settings import (
+    agent_secret,
     bub_environment,
     codex_auth_path,
     powercontext_bub_environment,
@@ -135,7 +136,7 @@ def agent_config(
                 "POWERCONTEXT_BUB_SCOPE_ID": scope_id,
             })
             if (token := server_api_token()) is not None:
-                env["POWERCONTEXT_BUB_API_TOKEN"] = token
+                env["POWERCONTEXT_BUB_API_TOKEN"] = agent_secret("POWERCONTEXT_BUB_API_TOKEN", token)
             if invocation_scopes is not None:
                 env.pop("POWERCONTEXT_BUB_SCOPE_ID")
                 kwargs["invocation_scopes"] = invocation_scopes
@@ -201,7 +202,8 @@ def agent_config(
             env = {**self._plugin_environment(), f"{self.plugin_prefix}SCOPE_ID": scope_id}
             if (token := server_api_token()) is not None:
                 # Each plugin sends this value as its Authorization header.
-                env[f"{self.plugin_prefix}AUTHORIZATION"] = f"Bearer {token}"
+                authorization = f"{self.plugin_prefix}AUTHORIZATION"
+                env[authorization] = agent_secret(authorization, f"Bearer {token}")
         return AgentConfig(
             import_path=self.agent_import_path,
             model_name=self.agent_model(),

e2e/bub/src/powercontext_e2e/runner.py

@@ -61,7 +61,7 @@
     TaskObservation,
 )
 from .report import render_evaluation_summary
-from .settings import HarnessSettings, ModelNotConfiguredError
+from .settings import HarnessSettings, ModelNotConfiguredError, agent_secret
 
 FailurePolicy = Literal["fail-fast", "collect-all"]
 TaskStatus = Literal["completed", "failed", "skipped"]
@@ -439,7 +439,8 @@ def _job_config(
         invocation_scopes=invocation_scopes if runtime is not None else None,
     )
     if settings.agent_proxy_url is not None:
-        proxy_url = settings.agent_proxy_url.get_secret_value()
+        # Harbor writes a literal under these names to its job files in full, and the URL can carry credentials.
+        proxy_url = agent_secret("PROXY_URL", settings.agent_proxy_url.get_secret_value())
         agent.env.update({
             "HTTP_PROXY": proxy_url,
             "HTTPS_PROXY": proxy_url,

e2e/bub/src/powercontext_e2e/settings.py

@@ -55,19 +55,55 @@ def server_api_token() -> str | None:
     return None if token is None else token.get_secret_value()
 
 
+# Values the harness derives for an agent are held under this prefix, which no integration reads as its own
+# setting, so reading an integration's native environment later never returns a derived value.
+_AGENT_SECRET_PREFIX = "POWERCONTEXT_E2E_AGENT_SECRET_"  # noqa: S105 - part of a variable name
+
+
+def agent_secret(name: str, value: str) -> str:
+    """Hold a value the harness derives for an agent in the harness's own environment, and return a reference to it.
+
+    ``name`` is the variable the agent reads, such as ``POWERCONTEXT_BUB_API_TOKEN``, or a short name for a value
+    that reaches the agent under several variables, such as ``PROXY_URL``.
+    Harbor writes each agent's environment to its job files. It keeps the first four and last three characters of a
+    sensitive literal, which is most of a short token, and writes a literal under any other name in full. It writes a
+    ``${NAME}`` reference as it is, whatever the name, and resolves the reference from this process's environment when
+    it starts the agent. A value held here is also an evidence secret.
+    """
+
+    held = f"{_AGENT_SECRET_PREFIX}{name.removeprefix('POWERCONTEXT_')}"
+    environ[held] = value
+    return f"${{{held}}}"
+
+
 def codex_auth_path() -> Path:
     """Resolve Codex's native authentication document location."""
 
     return Path(environ.get("CODEX_HOME", Path.home() / ".codex")).expanduser() / "auth.json"
 
 
 _SECRET_SUFFIXES = ("_API_KEY", "_AUTHORIZATION", "_TOKEN", "_SECRET_ACCESS_KEY")
+# A token can also be named in the middle, as in AWS_BEARER_TOKEN_BEDROCK. A plural, as in MAX_THINKING_TOKENS, is a
+# count, and a name ending in one of _LOCATION_SUFFIXES, as in AWS_WEB_IDENTITY_TOKEN_FILE, says where a token is:
+# its value is a path or an address, which redacting by substring would rewrite in the evidence.
+_SECRET_INFIX = "_TOKEN_"  # noqa: S105 - part of a variable name
+_LOCATION_SUFFIXES = ("_FILE", "_PATH", "_URL")
 # Local model servers accept any key, and the placeholders commonly passed to them are ordinary words and numbers.
 # They protect nothing, and redacting them by substring would rewrite the evidence. Any other value is redacted,
 # however short.
 _PLACEHOLDER_CREDENTIALS = frozenset({"1", "true", "none", "null", "empty", "dummy", "ollama", "lm-studio"})
 
 
+def _names_a_secret(name: str) -> bool:
+    # Settings read their variables in any case, so POWERCONTEXT_CLIENT_API_TOKEN may be set in lower case.
+    name = name.upper()
+    if name.startswith(_AGENT_SECRET_PREFIX):
+        return True
+    if name.endswith(_LOCATION_SUFFIXES):
+        return False
+    return name.endswith(_SECRET_SUFFIXES) or _SECRET_INFIX in name
+
+
 class ModelNotConfiguredError(RuntimeError):
     """Report model-backed workloads whose host lacks its runtime model or another required setting."""
 
@@ -118,7 +154,7 @@ def evidence_secrets(self) -> tuple[str, ...]:
         values = {
             value
             for name, value in environ.items()
-            if name.endswith(_SECRET_SUFFIXES) and value and value.lower() not in _PLACEHOLDER_CREDENTIALS
+            if _names_a_secret(name) and value and value.lower() not in _PLACEHOLDER_CREDENTIALS
         }
         if self.agent_proxy_url is not None and (proxy_url := self.agent_proxy_url.get_secret_value()):
             values.add(proxy_url)

e2e/bub/tests/conftest.py

@@ -0,0 +1,28 @@
+# Copyright (c) 2026 OceanBase.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import annotations
+
+import os
+from collections.abc import Iterator
+from unittest.mock import patch
+
+import pytest
+
+
+@pytest.fixture(autouse=True)
+def harness_environment() -> Iterator[None]:
+    # The harness holds the secrets it derives for agents in its own environment, so each test gets its own copy.
+    with patch.dict(os.environ):
+        yield

e2e/bub/tests/test_evidence_redaction.py

@@ -73,6 +73,10 @@ def test_resolved_instruction_evidence_matches_harbor_acp_summaries(
                 "HF_TOKEN",
                 "AWS_SECRET_ACCESS_KEY",
                 "POWERCONTEXT_PI_AUTHORIZATION",
+                # Harbor's Claude Code agent forwards this one, which names the token in the middle.
+                "AWS_BEARER_TOKEN_BEDROCK",
+                # Settings read their variables in any case.
+                "powercontext_client_api_token",
             )
         ),
         # The Client and the Server accept a token of any length.
@@ -152,8 +156,25 @@ def test_short_placeholder_credentials_do_not_corrupt_evidence(monkeypatch) -> N
     monkeypatch.setenv("LOCAL_API_KEY", "1")
     monkeypatch.setenv("OLLAMA_API_KEY", "ollama")
     monkeypatch.setenv("OPENROUTER_API_KEY", "sk-or-provider-secret")
-    evidence = json.dumps({"reward": 1, "provider": "ollama", "error": "rejected sk-or-provider-secret"})
+    # A plural names a count, which Harbor's Claude Code agent also forwards.
+    monkeypatch.setenv("MAX_THINKING_TOKENS", "8192")
+    # CI sets these to where a token is, not to the token: a path CI logs and an agent can print.
+    monkeypatch.setenv("AWS_WEB_IDENTITY_TOKEN_FILE", "/var/run/secrets/eks.amazonaws.com/serviceaccount/token")
+    monkeypatch.setenv("HF_TOKEN_PATH", "/home/runner/.cache/huggingface/token")
+    evidence = json.dumps({
+        "reward": 1,
+        "provider": "ollama",
+        "max_bytes": 8192,
+        "error": "rejected sk-or-provider-secret",
+        "stderr": "open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: no such file",
+    })
 
     redacted = json.loads(redact(evidence, HarnessSettings()))
 
-    assert redacted == {"reward": 1, "provider": "ollama", "error": "rejected [REDACTED]"}
+    assert redacted == {
+        "reward": 1,
+        "provider": "ollama",
+        "max_bytes": 8192,
+        "error": "rejected [REDACTED]",
+        "stderr": "open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: no such file",
+    }