Skip to content

[feat][evaluation] add failed evaluator record creation and handling - #563

Merged
HearyShen merged 4 commits into
mainfrom
fix/evaluator-record-precheck-failure
Jun 25, 2026
Merged

HearyShen merged 4 commits into
mainfrom
fix/evaluator-record-precheck-failure

Conversation

@HearyShen

Copy link
Copy Markdown
Collaborator
  1. add CreateEvaluatorRunFailRecord method to generate failure records before normal run persists
  2. update sync/async evaluator calls to create failure records when run fails
  3. add validation for missing evaluator results and update turn run status
  4. add corresponding mocks and test cases

What type of PR is this?

Check the PR title

  • This PR title match the format: [<type>][<scope>] <description>. For example: [fix][backend] flaky fix
  • The description of this PR title is user-oriented and clear enough for others to understand.
  • Add documentation if the current PR requires user awareness at the usage level.
  • This PR is written in English. PRs not in English will not be reviewed.

(Optional) Translate the PR title into Chinese

(Optional) More detailed description for this PR(en: English/zh: Chinese)

en:
zh(optional):

(Optional) Which issue(s) this PR fixes

1. add CreateEvaluatorRunFailRecord method to generate failure records before normal run persists
2. update sync/async evaluator calls to create failure records when run fails
3. add validation for missing evaluator results and update turn run status
4. add corresponding mocks and test cases
…reation

Rewrite the error handling logic for CreateEvaluatorRunFailRecord to:
1. Use proper errorx status error parsing consistently
2. Correct the conditional check for custom RPC error skip logic
3. Add comprehensive unit test cases covering multiple error scenarios
4. Standardize error message stripping and code extraction
@codecov

codecov Bot commented Jun 25, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.67347% with 16 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...es/evaluation/domain/service/expt_run_item_impl.go 59.25% 6 Missing and 5 partials ⚠️
...aluation/domain/service/expt_run_item_turn_impl.go 80.76% 3 Missing and 2 partials ⚠️

Impacted file tree graph

@@           Coverage Diff           @@
##             main     #563   +/-   ##
=======================================
  Coverage   77.82%   77.83%           
=======================================
  Files         670      670           
  Lines       76151    76234   +83     
=======================================
+ Hits        59268    59338   +70     
- Misses      13421    13428    +7     
- Partials     3462     3468    +6     
Flag Coverage Δ
unittests 77.83% <83.67%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...odules/evaluation/domain/service/evaluator_impl.go 84.65% <100.00%> (+0.95%) ⬆️
...aluation/domain/service/expt_run_item_turn_impl.go 85.82% <80.76%> (-0.64%) ⬇️
...es/evaluation/domain/service/expt_run_item_impl.go 76.84% <59.25%> (-1.25%) ⬇️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8485ee8...7248317. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

lsy357
lsy357 previously approved these changes Jun 25, 2026
@HearyShen
HearyShen merged commit b98db10 into main Jun 25, 2026
16 checks passed
@HearyShen
HearyShen deleted the fix/evaluator-record-precheck-failure branch June 25, 2026 13:51
Colin4k1024 pushed a commit to Colin4k1024/coze-loop that referenced this pull request Aug 28, 2026
…oze-dev#563)

* feat(evaluation): add failed evaluator record creation and handling

1. add CreateEvaluatorRunFailRecord method to generate failure records before normal run persists
2. update sync/async evaluator calls to create failure records when run fails
3. add validation for missing evaluator results and update turn run status
4. add corresponding mocks and test cases

* refactor(evaluator): simplify and fix error handling in fail record creation

Rewrite the error handling logic for CreateEvaluatorRunFailRecord to:
1. Use proper errorx status error parsing consistently
2. Correct the conditional check for custom RPC error skip logic
3. Add comprehensive unit test cases covering multiple error scenarios
4. Standardize error message stripping and code extraction

* Add error path coverage for evaluator failure records

* Refactor evaluator error assertions in turn impl test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants