Skip to content

[fix][evaluation] fix occasional missing top-level root_step in eval trajectory - #667

Merged
caijialin0626 merged 5 commits into
mainfrom
fix/trajectory-extract-budget
Sep 17, 2026
Merged

caijialin0626 merged 5 commits into
mainfrom
fix/trajectory-extract-budget

Conversation

@caijialin0626

@caijialin0626 caijialin0626 commented Sep 16, 2026 •

Copy link
Copy Markdown
Collaborator

What type of PR is this?

fix

Check the PR title

  • This PR title match the format: [<type>][<scope>] <description>. For example: [fix][backend] flaky fix
  • The description of this PR title is user-oriented and clear enough for others to understand.
  • Add documentation if the current PR requires user awareness at the usage level.
  • This PR is written in English. PRs not in English will not be reviewed.

(Optional) Translate the PR title into Chinese

[fix][evaluation] 修复评测轨迹偶发缺失顶层 root_step

(Optional) More detailed description for this PR(en: English/zh: Chinese)

en:

Problem. The persisted eval-target record occasionally lacks the top-level root_step in its trajectory: output_fields.trajectory shows only {id, agent_steps}, or the trajectory field is missing entirely. In a PPE lane this reproduced at roughly a 50% rate.

Root cause. On the observability side, the top-level root span (the only span with ParentID == "" || ParentID == "0", see BuildTrajectoryFromSpans) lands with eventual consistency on a different ingestion path than its child spans, and its landing latency has a long tail (observed 180s–540s after the record, with outliers beyond 9 minutes). While the root span is not yet visible, ListTrajectory returns RootStep == nil, so entity.Trajectory.IsValid() (which requires both ID != nil and RootStep != nil) is false. No agent step can substitute for the root because every agent step carries a real ParentID pointing at the not-yet-landed root — the extractor can only wait. The old retry window on the evaluation side was too short (3 x 1s, and even 6 x 10s ~= 60s), so extraction gave up before the root span landed and never re-extracted, permanently dropping the trajectory. The underlying data is never lost — re-querying the same trace minutes later returns the root span.

There were two layers, fixed across this branch:

  1. The background extraction budget was being consumed by the wait interval (extractInterval), leaving the final ListTrajectory with a near-zero deadline and failing with timeout=0s. Fixed by sleeping the wait interval on the background context and giving the work budget an independent timeout that covers all attempts (RPC x attempts + retry intervals + one persist).
  2. The retry window itself was too short to cover the root-span landing tail. Widened the retry window to 14 x 30s (~7 min); the background workBudget scales automatically with the attempt count.

Extraction runs entirely in a background goroutine (off the request path) and only defers the trajectory's UpdateEvalTargetRecord, so widening the window has no user-facing latency cost.

Verification.

  • Added/updated unit tests covering (a) the retry window persisting until the root span lands and (b) the background budget covering every attempt plus the persist; full evaluation/domain/service package passes.
  • PPE multi-sample E2E: root_step persistence rose from ~50% to ~93%.

Known limitation. The root-span landing tail is unbounded (an outlier exceeded 9 minutes), so a "sit and retry in a goroutine" design cannot guarantee 100% by widening the window alone — the residual ~7% is inherent to eventual consistency. Fully covering the tail would require an event-driven or deferred-queue re-extraction, which is a separate follow-up. No trajectory data is lost in any case.

zh(optional):

评测记录的 trajectory 偶发缺失顶层 root_step。根因是 observability 侧顶层 root span(唯一 ParentID 为空/"0" 的 span)落库最终一致、长尾无界(实测 record 后 180s~540s,个例 >9min),子 span 先可见时 ListTrajectory 返回 RootStep=nil、IsValid()=false,且无法用 agent step 顶替 root,只能等;旧重试窗口过短会在 root span 落库前放弃且不补抽,导致轨迹永久缺失(数据本身不丢)。本分支分两层修复:① 让后台抽取的等待期在 background ctx 上先睡完、工作预算独立覆盖全部 attempts;② 重试窗口加宽到 14×30s(约 7min,后台 workBudget 自动缩放)。抽取全程在后台 goroutine、仅延后 UpdateEvalTargetRecord,无用户侧延迟代价。PPE 多样本 E2E 成功率由 ~50% 升至 ~93%。受最终一致长尾无界限制,此架构无法靠加宽窗口保证 100%,彻底覆盖需事件驱动/延迟队列补抽(独立后续项)。


🤖 Generated with Claude Code

caijialin0626 and others added 5 commits September 16, 2026 12:22
…t work budget

历史修复(7f6e880)把等待期 extractInterval 与抽取+落库共用一个 extractCtx 超时预算,
且工作预算只算了 (attempts-1)*retryInterval + persistTimeout, 完全没给 attempts 次
ListTrajectory RPC 本身留时间。当 trace 尚未最终一致触发多次重试、每次 RPC 又各耗约 1.5s 时,
最后一次 RPC 拿到的 ctx 剩余会 <= 0, 以 "rpc timeout: timeout=0s ... method=ListTrajectory,
timeout by business" 立即失败 → trajectory 丢失(PPE 偶发, 约 50%)。

修复:
- 等待期 extractInterval 先在无 deadline 的 backgroundCtx 上睡完, 不占用工作预算;
- 工作预算独立且覆盖 attempts 次 RPC + (attempts-1) 次重试间隔 + 一次落库。

新增回归测试 TestEvalTargetServiceImpl_ReportInvokeRecords_TrajectoryBudgetCoversAllAttempts:
模拟前两次抽取不完整触发重试、每次 RPC 耗 800ms, 断言最后一次抽取时 ctx 剩余仍能容纳落库。

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…root span

observability aggregates trace spans with eventual consistency: the top-level
root span (parent_id empty/"0") often lands tens of seconds after extraction
starts. Until then ListTrajectory returns {id, agent_steps} with RootStep=nil,
so IsValid() is false. The old retry window (3 attempts x 1s) gave up before the
root span landed and never re-extracted, permanently dropping the trajectory.

Widen to 6 attempts x 10s (50s retry period) to cover the observed landing
delay. workBudget already scales with these constants.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
多样本 PPE E2E 暴露: 6x10s(~60s) 重试窗口仍有 ~29% 样本丢轨迹。
后端日志 + 重查证据表明 observability 侧顶层 root span(唯一 ParentID
空/"0") 与子 span 走不同落库路径, 其对 ListTrajectory 可见的最终一致
长尾实测达 record 后 180s~540s; 子 span 先可见时 ListTrajectory 返回
{id, agent_steps} 而 RootStep=nil, IsValid()=false, 无法用任一 agent
step 顶替 root, 只能等。窗口过短会在 root span 落库前判 incomplete 放弃
且不再补抽 -> 轨迹永久缺失。

抽取全程在后台 goroutine(脱离请求路径), 只延后 trajectory 的
UpdateEvalTargetRecord, 放宽窗口无用户侧延迟代价:
- trajectoryExtractAttempts 6 -> 14
- defaultTrajectoryRetryInterval 10s -> 30s
重试窗口约 7min, 后台 workBudget 自动缩放。

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@codecov

codecov Bot commented Sep 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.42857% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...d/modules/evaluation/domain/service/target_impl.go 91.42% 3 Missing ⚠️

Impacted file tree graph

@@           Coverage Diff           @@
##             main     #667   +/-   ##
=======================================
  Coverage   78.81%   78.82%           
=======================================
  Files         707      707           
  Lines       87804    87825   +21     
=======================================
+ Hits        69204    69224   +20     
- Misses      14609    14611    +2     
+ Partials     3991     3990    -1     
Flag Coverage Δ
unittests 78.82% <91.42%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...d/modules/evaluation/domain/service/target_impl.go 85.26% <91.42%> (+0.01%) ⬆️

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1133eda...232d7cd. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@caijialin0626 caijialin0626 changed the title fix(evaluation): 修复评测轨迹偶发缺失 root_step (抽取过早+重试窗口不足) [fix][evaluation] fix occasional missing top-level root_step in eval trajectory Sep 16, 2026
@xueyizheng
xueyizheng self-requested a review September 17, 2026 08:58
@caijialin0626
caijialin0626 merged commit 41c0896 into main Sep 17, 2026
18 of 19 checks passed
@caijialin0626
caijialin0626 deleted the fix/trajectory-extract-budget branch September 17, 2026 09:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants