[fix][evaluation] fix occasional missing top-level root_step in eval trajectory - #667
Merged
Merged
Conversation
…t work budget 历史修复(7f6e880)把等待期 extractInterval 与抽取+落库共用一个 extractCtx 超时预算, 且工作预算只算了 (attempts-1)*retryInterval + persistTimeout, 完全没给 attempts 次 ListTrajectory RPC 本身留时间。当 trace 尚未最终一致触发多次重试、每次 RPC 又各耗约 1.5s 时, 最后一次 RPC 拿到的 ctx 剩余会 <= 0, 以 "rpc timeout: timeout=0s ... method=ListTrajectory, timeout by business" 立即失败 → trajectory 丢失(PPE 偶发, 约 50%)。 修复: - 等待期 extractInterval 先在无 deadline 的 backgroundCtx 上睡完, 不占用工作预算; - 工作预算独立且覆盖 attempts 次 RPC + (attempts-1) 次重试间隔 + 一次落库。 新增回归测试 TestEvalTargetServiceImpl_ReportInvokeRecords_TrajectoryBudgetCoversAllAttempts: 模拟前两次抽取不完整触发重试、每次 RPC 耗 800ms, 断言最后一次抽取时 ctx 剩余仍能容纳落库。 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…root span
observability aggregates trace spans with eventual consistency: the top-level
root span (parent_id empty/"0") often lands tens of seconds after extraction
starts. Until then ListTrajectory returns {id, agent_steps} with RootStep=nil,
so IsValid() is false. The old retry window (3 attempts x 1s) gave up before the
root span landed and never re-extracted, permanently dropping the trajectory.
Widen to 6 attempts x 10s (50s retry period) to cover the observed landing
delay. workBudget already scales with these constants.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
多样本 PPE E2E 暴露: 6x10s(~60s) 重试窗口仍有 ~29% 样本丢轨迹。
后端日志 + 重查证据表明 observability 侧顶层 root span(唯一 ParentID
空/"0") 与子 span 走不同落库路径, 其对 ListTrajectory 可见的最终一致
长尾实测达 record 后 180s~540s; 子 span 先可见时 ListTrajectory 返回
{id, agent_steps} 而 RootStep=nil, IsValid()=false, 无法用任一 agent
step 顶替 root, 只能等。窗口过短会在 root span 落库前判 incomplete 放弃
且不再补抽 -> 轨迹永久缺失。
抽取全程在后台 goroutine(脱离请求路径), 只延后 trajectory 的
UpdateEvalTargetRecord, 放宽窗口无用户侧延迟代价:
- trajectoryExtractAttempts 6 -> 14
- defaultTrajectoryRetryInterval 10s -> 30s
重试窗口约 7min, 后台 workBudget 自动缩放。
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
@@ Coverage Diff @@
## main #667 +/- ##
=======================================
Coverage 78.81% 78.82%
=======================================
Files 707 707
Lines 87804 87825 +21
=======================================
+ Hits 69204 69224 +20
- Misses 14609 14611 +2
+ Partials 3991 3990 -1
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 1 file with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
ming845378603
approved these changes
Sep 17, 2026
xueyizheng
self-requested a review
September 17, 2026 08:58
xueyizheng
approved these changes
Sep 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
fix
Check the PR title
(Optional) Translate the PR title into Chinese
[fix][evaluation] 修复评测轨迹偶发缺失顶层 root_step
(Optional) More detailed description for this PR(en: English/zh: Chinese)
en:
Problem. The persisted eval-target record occasionally lacks the top-level
root_stepin its trajectory:output_fields.trajectoryshows only{id, agent_steps}, or the trajectory field is missing entirely. In a PPE lane this reproduced at roughly a 50% rate.Root cause. On the observability side, the top-level root span (the only span with
ParentID == "" || ParentID == "0", seeBuildTrajectoryFromSpans) lands with eventual consistency on a different ingestion path than its child spans, and its landing latency has a long tail (observed 180s–540s after the record, with outliers beyond 9 minutes). While the root span is not yet visible,ListTrajectoryreturnsRootStep == nil, soentity.Trajectory.IsValid()(which requires bothID != nilandRootStep != nil) is false. No agent step can substitute for the root because every agent step carries a realParentIDpointing at the not-yet-landed root — the extractor can only wait. The old retry window on the evaluation side was too short (3 x 1s, and even 6 x 10s ~= 60s), so extraction gave up before the root span landed and never re-extracted, permanently dropping the trajectory. The underlying data is never lost — re-querying the same trace minutes later returns the root span.There were two layers, fixed across this branch:
extractInterval), leaving the finalListTrajectorywith a near-zero deadline and failing withtimeout=0s. Fixed by sleeping the wait interval on the background context and giving the work budget an independent timeout that covers all attempts (RPC x attempts + retry intervals + one persist).14 x 30s(~7 min); the backgroundworkBudgetscales automatically with the attempt count.Extraction runs entirely in a background goroutine (off the request path) and only defers the trajectory's
UpdateEvalTargetRecord, so widening the window has no user-facing latency cost.Verification.
evaluation/domain/servicepackage passes.Known limitation. The root-span landing tail is unbounded (an outlier exceeded 9 minutes), so a "sit and retry in a goroutine" design cannot guarantee 100% by widening the window alone — the residual ~7% is inherent to eventual consistency. Fully covering the tail would require an event-driven or deferred-queue re-extraction, which is a separate follow-up. No trajectory data is lost in any case.
zh(optional):
评测记录的 trajectory 偶发缺失顶层
root_step。根因是 observability 侧顶层 root span(唯一ParentID为空/"0" 的 span)落库最终一致、长尾无界(实测 record 后 180s~540s,个例 >9min),子 span 先可见时ListTrajectory返回RootStep=nil、IsValid()=false,且无法用 agent step 顶替 root,只能等;旧重试窗口过短会在 root span 落库前放弃且不补抽,导致轨迹永久缺失(数据本身不丢)。本分支分两层修复:① 让后台抽取的等待期在 background ctx 上先睡完、工作预算独立覆盖全部 attempts;② 重试窗口加宽到 14×30s(约 7min,后台 workBudget 自动缩放)。抽取全程在后台 goroutine、仅延后 UpdateEvalTargetRecord,无用户侧延迟代价。PPE 多样本 E2E 成功率由 ~50% 升至 ~93%。受最终一致长尾无界限制,此架构无法靠加宽窗口保证 100%,彻底覆盖需事件驱动/延迟队列补抽(独立后续项)。🤖 Generated with Claude Code