Skip to content

docs: add DSV4 agentic benchmark + ARM64 SWE-bench builder + Approved artifacts list - #494

Merged
hvagadia merged 4 commits into
mlcommons:mainfrom
hvagadia:docs/dsv4-agentic-model
Sep 10, 2026
Merged

hvagadia merged 4 commits into
mlcommons:mainfrom
hvagadia:docs/dsv4-agentic-model

Conversation

@hvagadia

@hvagadia hvagadia commented Sep 4, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • add DeepSeek-V4-Pro to the supported agentic models and provide a minimal SGLang reference command
  • add a DSV4 benchmark YAML with the approved sampling and thinking parameters
  • document approved checkpoints and speculative-decoding heads for Kimi K3, DSV4, and Qwen3.6-35B-A3B
  • add a configurable build-and-push script and usage instructions for the first 200 pinned ARM64 SWE-bench Verified images

Validation

  • parsed dsv4_agentic_benchmark.yaml with BenchmarkConfig.from_yaml_file
  • compiled build_and_push.py and verified that it requires an explicit destination registry
  • ran file-scoped pre-commit checks for the README, DSV4 YAML, and ARM64 build script

@hvagadia
hvagadia requested a review from a team September 4, 2026 16:51
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@github-actions github-actions Bot added the size/normal PR Review Policy: <=500 non-test lines & <=20 files label Sep 4, 2026
@tianmu-li

tianmu-li commented Sep 4, 2026 •

Copy link
Copy Markdown
Collaborator

The changes bundled in nvfp4 checkpoint acceptance. That's fine by itself but complicates the scope of the PR. Recommend separating the nvfp4-related changes out

@hvagadia hvagadia changed the title docs: add DSV4 agentic benchmark example docs: add DSV4 agentic benchmark + approved checkpoint list Sep 4, 2026
@hvagadia

hvagadia commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

Hi Tianmu, I am going to send out an email to TF explaining the approved artifacts list. Should make it clearer.

@codecov-commenter

codecov-commenter commented Sep 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@3614f36). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #494   +/-   ##
=======================================
  Coverage        ?   80.83%           
=======================================
  Files           ?      153           
  Lines           ?    21066           
  Branches        ?        0           
=======================================
  Hits            ?    17028           
  Misses          ?     4038           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@hvagadia hvagadia changed the title docs: add DSV4 agentic benchmark + approved checkpoint list docs: add DSV4 agentic benchmark and ARM64 SWE-bench builder Sep 4, 2026
@hvagadia hvagadia changed the title docs: add DSV4 agentic benchmark and ARM64 SWE-bench builder docs: add DSV4 agentic benchmark + ARM64 SWE-bench builder + Approved artifacts list Sep 4, 2026
Comment thread examples/10_Agentic_Inference/README.md Outdated
Comment thread examples/10_Agentic_Inference/build_and_push.py

@arekay-nv arekay-nv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please include the SHA for the checkpoints so we have pinned versions of the model.

Comment thread examples/10_Agentic_Inference/README.md Outdated
Comment thread examples/10_Agentic_Inference/README.md Outdated
Comment thread examples/10_Agentic_Inference/README.md Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/normal PR Review Policy: <=500 non-test lines & <=20 files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants