From a742f5593a48cd0cd0b9c3e2f9e7b6d781d23d2c Mon Sep 17 00:00:00 2001 From: Harshil Vagadia Date: Fri, 4 Sep 2026 09:50:51 -0700 Subject: [PATCH 1/4] docs: add DSV4 agentic benchmark example --- examples/10_Agentic_Inference/README.md | 94 +++++++++++++++---- .../dsv4_agentic_benchmark.yaml | 62 ++++++++++++ 2 files changed, 136 insertions(+), 20 deletions(-) create mode 100644 examples/10_Agentic_Inference/dsv4_agentic_benchmark.yaml diff --git a/examples/10_Agentic_Inference/README.md b/examples/10_Agentic_Inference/README.md index 7f0f84b6c..fabce3fea 100644 --- a/examples/10_Agentic_Inference/README.md +++ b/examples/10_Agentic_Inference/README.md @@ -21,14 +21,15 @@ Place the dataset under `examples/10_Agentic_Inference/datasets/` or point the Y ## Supported Models -The Agentic Inference benchmark can support any model. The current iteration of the MLPerf Inference benchmark accepts official submissions for the following two models: +The Agentic Inference benchmark can support any model. The current iteration of the MLPerf Inference benchmark accepts official submissions for the following three models: -| Model | Architecture | Parameters | Context | -| --------------- | ------------------------------ | ------------------------ | ---------------- | -| Kimi K3 | MoE + KDA/Gated MLA | 2.8T total / 104B active | 1,048,576 tokens | -| Qwen3.6-35B-A3B | MoE + Gated DeltaNet/Attention | 35B total / 3B active | 262,144 tokens | +| Model | Architecture | Parameters | Context | +| ---------------------- | ------------------------------ | ------------------------ | ---------------- | +| Kimi K3 | MoE + KDA/Gated MLA | 2.8T total / 104B active | 1,048,576 tokens | +| Qwen3.6-35B-A3B | MoE + Gated DeltaNet/Attention | 35B total / 3B active | 262,144 tokens | +| DeepSeek-V4-Pro (DSV4) | MoE + hybrid CSA/HCA + mHC | 1.6T total / 49B active | 1,048,576 tokens | -Reference implementations and runnable examples for both models are provided below. +Reference implementations and runnable examples for all three models are provided below. ## Start A Server @@ -68,11 +69,27 @@ sglang serve \ --mem-fraction-static 0.95 ``` +### DSV4 + +See the [SGLang DeepSeek-V4 recipe](https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4) for hardware-specific deployment guidance. + +```bash +sglang serve \ + --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \ + --served-model-name deepseek-ai/DeepSeek-V4-Pro-0813 \ + --tp-size 8 \ + --trust-remote-code \ + --reasoning-parser deepseek-v4 \ + --tool-call-parser deepseekv4 \ + --host 0.0.0.0 \ + --port 30000 +``` + `--model-path` is the checkpoint loaded by the server. It can be a local path visible to the server container or a Hugging Face model ID, depending on your SGLang environment. `--served-model-name` is the OpenAI model name exposed to clients; set `model_params.name` in the YAML to the same value. ## Client YAML -Runnable example configs are provided for [Kimi K3](kimi_agentic_benchmark.yaml) and [Qwen3.6-35B-A3B](qwen_agentic_benchmark.yaml). +Runnable example configs are provided for [Kimi K3](kimi_agentic_benchmark.yaml), [Qwen3.6-35B-A3B](qwen_agentic_benchmark.yaml), and [DSV4](dsv4_agentic_benchmark.yaml). Some key client features specific to the Agentic Inference benchmark are described below. @@ -100,7 +117,7 @@ For official submissions, submitters must set `agentic_inference.stop_issuing_on ### SWE-bench Accuracy -Submitters must enable SWE-bench accuracy for official submissions. Both `qwen_agentic_benchmark.yaml` and `kimi_agentic_benchmark.yaml` include the required SWE-bench accuracy dataset. The benchmark framework skips its built-in endpoint phase for the SWE-bench dataset. Instead, `SWEBenchScorer` submits the run to a native SWE-bench service. The service host owns Docker, `mini-swe-agent`, and the `swebench` evaluation harness, and it drives requests to the configured endpoint. +Submitters must enable SWE-bench accuracy for official submissions. The Kimi K3, Qwen3.6-35B-A3B, and DSV4 example YAML files include the required SWE-bench accuracy dataset. The benchmark framework skips its built-in endpoint phase for the SWE-bench dataset. Instead, `SWEBenchScorer` submits the run to a native SWE-bench service. The service host owns Docker, `mini-swe-agent`, and the `swebench` evaluation harness, and it drives requests to the configured endpoint. Keep `accuracy_config.num_repeats: 1`: the scorer performs one external evaluation run per benchmark. Optional `accuracy_config.extras.subset` and `split` are used consistently for dataset loading, preflight, and scoring. @@ -125,6 +142,7 @@ Update the first `datasets` entry (`name` and `path`), `model_params.name`, and ```bash CONFIG=examples/10_Agentic_Inference/qwen_agentic_benchmark.yaml # For Kimi, use examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml. +# For DSV4, use examples/10_Agentic_Inference/dsv4_agentic_benchmark.yaml. # PERF (default): agentic performance and inline scoring; skips SWE-bench. uv run inference-endpoint benchmark from-config --config "$CONFIG" @@ -164,6 +182,17 @@ For Qwen3.6-35B-A3B: - `max_new_tokens: 8192` - `chat_template_kwargs.preserve_thinking: true` +For DSV4: + +- `temperature: 1.0` +- `top_k: 0` +- `top_p: 0.95` +- `max_new_tokens: 8192` +- `streaming: "on"` +- `chat_template_kwargs.thinking: true` +- `chat_template_kwargs.reasoning_effort: max` +- `chat_template_kwargs.preserve_thinking: true` + Any sampling parameter or thinking flag not listed above for the selected model must be omitted. Submitters must not introduce additional sampling parameters or thinking flags. `preserve_thinking` ensures that reasoning tokens from previous turns are not stripped by the chat template before the input is sent to the inference engine. Because chat-template processing is a server-side property, the client can only request this behavior by sending the flag. Popular serving frameworks, including SGLang, vLLM, and TensorRT-LLM, honor this flag; however, each submitter is responsible for verifying that their server is compliant. The chat template must not omit reasoning tokens from any previous turn. @@ -183,21 +212,46 @@ For official submissions, `agentic_inference.stop_issuing_on_first_user_complete Official submissions must enable both inline accuracy and SWE-bench accuracy. Configure the performance dataset with `accuracy_config.eval_method: agentic_inference_inline`, and configure the SWE-bench accuracy dataset with `accuracy_config.eval_method: swe_bench_scorer`. For SWE-bench, `accuracy_config.extras.num_instances` must be set to `200`. When using the example `online` configs, run with `--mode both` so performance, inline accuracy, and SWE-bench accuracy are all executed. -Qwen3.6-35B-A3B submissions must set `accuracy_config.extras.swebench_template: qwen_tools`. Kimi K3 submissions must omit `accuracy_config.extras.swebench_template`. +Qwen3.6-35B-A3B submissions must set `accuracy_config.extras.swebench_template: qwen_tools`. Kimi K3 submissions must omit `accuracy_config.extras.swebench_template`. The DSV4 SWE-bench template requirement is TBD. + +Every Kimi K3 and Qwen3.6-35B-A3B submitted Pareto point must satisfy all of the model-specific accuracy thresholds below. For these models, SWE-bench accuracy is evaluated using mean-of-N: average one SWE-bench accuracy result from each of the [four mandatory regions](https://github.com/mlcommons/endpoints_policies/blob/main/endpoints_rules.md#54-regions-of-interest) (`N = 4`), then compare that mean with the model-specific SWE-bench threshold below. The DSV4 accuracy thresholds and SWE-bench evaluation policy are TBD. + +| Metric | Kimi K3 | Qwen3.6-35B-A3B | DSV4 | +| ------------------ | ---------------: | ---------------: | ---: | +| Inline accuracy | `>= 58.32%` | `>= 55.86%` | TBD | +| OSL per-turn mean | `390-475` tokens | `355-434` tokens | TBD | +| SWE-bench accuracy | `>= 93.5%` | `>= 69%` | TBD | + +### Approved Checkpoints and Speculative-Decoding Heads + +Under the MLPerf Endpoint Benchmark rules, submitters must use only the approved model checkpoints and speculative-decoding heads listed below in their submissions. New artifacts may be added to this list only after approval from the Agentic Inference taskforce. + +#### Kimi K3 + +Approved model checkpoints: + +- [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3) +- [nvidia/Kimi-K3-NVFP4](https://huggingface.co/nvidia/Kimi-K3-NVFP4) + +Approved speculative-decoding heads: + +- [RadixArk/Kimi-K3-DSpark](https://huggingface.co/RadixArk/Kimi-K3-DSpark) +- [Inferact/Kimi-K3-DSpark](https://huggingface.co/Inferact/Kimi-K3-DSpark) + +#### DSV4 + +Approved model checkpoints: -Every submitted Pareto point must satisfy all of the following accuracy thresholds, except that SWE-bench accuracy is evaluated using mean-of-N for both models. For each model, average one SWE-bench accuracy result from each of the [four mandatory regions](https://github.com/mlcommons/endpoints_policies/blob/main/endpoints_rules.md#54-regions-of-interest) (`N = 4`), then compare that mean with the model-specific SWE-bench threshold below. +- [deepseek-ai/DeepSeek-V4-Pro-0813](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813) +- [nvidia/DeepSeek-V4-Pro-0813-NVFP4](https://huggingface.co/nvidia/DeepSeek-V4-Pro-0813-NVFP4) -| Metric | Kimi K3 | Qwen3.6-35B-A3B | -| ------------------ | ---------------: | ---------------: | -| Inline accuracy | `>= 58.32%` | `>= 55.86%` | -| OSL per-turn mean | `390-475` tokens | `355-434` tokens | -| SWE-bench accuracy | `>= 93.5%` | `>= 69%` | +Both approved DSV4 checkpoints include bundled DSpark speculative-decoding heads. -### Speculative Decoding +#### Qwen3.6-35B-A3B -Speculative decoding may be used in accordance with the MLPerf Endpoint Benchmark rules. The list of allowed speculative-decoding heads for this benchmark will be maintained below. Only heads included in this list may be used for official submissions. +Approved model checkpoints: -Allowed speculative-decoding heads: +- [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) +- [nvidia/Qwen3.6-35B-A3B-NVFP4](https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4) (contains both W4A16 and W4A4 scales) -- Qwen3.6-35B-A3B: native MTP head included with the model -- Kimi K3: DSPARK with [RadixArk/Kimi-K3-DSpark](https://huggingface.co/RadixArk/Kimi-K3-DSpark) on SGLang or [Inferact/Kimi-K3-DSpark](https://huggingface.co/Inferact/Kimi-K3-DSpark) on vLLM +Both approved Qwen checkpoints include their native MTP speculative-decoding heads. diff --git a/examples/10_Agentic_Inference/dsv4_agentic_benchmark.yaml b/examples/10_Agentic_Inference/dsv4_agentic_benchmark.yaml new file mode 100644 index 000000000..a6aee8db1 --- /dev/null +++ b/examples/10_Agentic_Inference/dsv4_agentic_benchmark.yaml @@ -0,0 +1,62 @@ +name: "dsv4-agentic-benchmark" +version: "1.0" +type: "online" + +model_params: + name: "deepseek-ai/DeepSeek-V4-Pro-0813" + tokenizer_name: "${MODEL_DIR}" + temperature: 1.0 + top_k: 0 + top_p: 0.95 + max_new_tokens: 8192 + streaming: "on" + chat_template_kwargs: + thinking: true + reasoning_effort: max + preserve_thinking: true + +datasets: + - name: agentic_combined + type: performance + path: /path/to/agentic_combined.jsonl + accuracy_config: + eval_method: agentic_inference_inline # required benchmark default. + num_repeats: 1 + agentic_inference: + turn_timeout_s: 86400.0 + enable_salt: true # required benchmark default. + inject_tool_delay: true + num_trajectories_to_issue: 613 # Should be integer multiple of 613. + # Required benchmark default; set to true only for faster optimization/debug runs. + stop_issuing_on_first_user_complete: false + - name: swe_bench # Default PERF skips external evaluation; use ACC or BOTH. + type: "accuracy" + accuracy_config: + eval_method: "swe_bench_scorer" + num_repeats: 1 + extras: + swebench_service_url: "http://localhost:18080" + swebench_service_auth_token: "${SWEBENCH_SERVICE_AUTH_TOKEN}" + num_instances: 200 + workers: 100 + # DSV4 SWE-bench template requirement: TBD. + +settings: + runtime: + scheduler_random_seed: 42 + dataloader_random_seed: 42 + + load_pattern: + type: agentic_inference + target_concurrency: 8 # Submission-specific concurrency. + + client: + warmup_connections: 0 + max_idle_time: 0.5 + +endpoint_config: + endpoints: + - "http://localhost:30000" + api_type: openai + +report_dir: logs/dsv4_agentic From ad137a52804b8ea8d2e4b00a0fed8cdcf4c4d33d Mon Sep 17 00:00:00 2001 From: Harshil Vagadia Date: Fri, 4 Sep 2026 11:01:50 -0700 Subject: [PATCH 2/4] docs: add ARM64 SWE-bench image builder --- examples/10_Agentic_Inference/README.md | 18 ++ .../10_Agentic_Inference/build_and_push.py | 172 ++++++++++++++++++ 2 files changed, 190 insertions(+) create mode 100644 examples/10_Agentic_Inference/build_and_push.py diff --git a/examples/10_Agentic_Inference/README.md b/examples/10_Agentic_Inference/README.md index fabce3fea..240269ab3 100644 --- a/examples/10_Agentic_Inference/README.md +++ b/examples/10_Agentic_Inference/README.md @@ -135,6 +135,24 @@ uv run --project src/inference_endpoint/evaluation/swebench_service \ --auth-token "$SWEBENCH_SERVICE_AUTH_TOKEN" ``` +#### Build ARM64 SWE-bench Images + +[`build_and_push.py`](build_and_push.py) builds and pushes the native ARM64 images for the first 200 pinned SWE-bench Verified tasks. The script validates the task list, applies the required ARM compatibility fixes, and skips images that already exist in the destination registry. + +Run it on a native ARM64 machine with Docker after logging in to a registry where you have push access: + +```bash +python3 -m venv /tmp/swebench-arm64-venv +/tmp/swebench-arm64-venv/bin/pip install "swebench==4.1.0" +docker login registry.example.com + +REGISTRY=registry.example.com/group/project \ + /tmp/swebench-arm64-venv/bin/python \ + examples/10_Agentic_Inference/build_and_push.py +``` + +Set `WORKERS` to change the default build and push concurrency of `16`. Images are tagged `v4.1.0-arm64`, and interrupted runs can be resumed with the same command. + ## Run The Client Update the first `datasets` entry (`name` and `path`), `model_params.name`, and `endpoint_config.endpoints` as needed. Then select the matching model config and run it from the repo root: diff --git a/examples/10_Agentic_Inference/build_and_push.py b/examples/10_Agentic_Inference/build_and_push.py new file mode 100644 index 000000000..f3d427fa8 --- /dev/null +++ b/examples/10_Agentic_Inference/build_and_push.py @@ -0,0 +1,172 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Build and push the benchmark's first 200 ARM64 SWE-bench images.""" + +import hashlib +import os +import subprocess +from concurrent.futures import ThreadPoolExecutor, as_completed + +import docker +from datasets import load_dataset +from swebench.harness.docker_build import build_env_images, build_instance_image +from swebench.harness.test_spec.test_spec import make_test_spec + +DATASET = "princeton-nlp/SWE-bench_Verified" +REVISION = "c104f840cc67f8b6eec6f759ebc8b2693d585d4a" +IDS_SHA256 = "4b562a390c5bfe39a9b00d503ebce1d8b7d1c60f6b761617317a0c507534f40f" +REGISTRY = os.getenv("REGISTRY", "").rstrip("/") +TAG = "v4.1.0-arm64" +WORKERS = int(os.getenv("WORKERS", "16")) +COMPAT_IDS = {"django__django-10097", "django__django-15103"} +# This old Django runner consumes unsorted os.listdir("/testbed/tests"). The ARM +# image enumerates it differently from x86, exposing leaked test state in +# generic_inline_admin. Replay the recorded x86 order; unlisted entries still run. +DJANGO_10097_TEST_ORDER = """\ +template_tests delete shortcuts fixtures admin_default_site delete_regress distinct_on_fields from_db_value +mutually_referential empty admin_autodiscover model_inheritance_regress null_fk middleware_exceptions context_processors bash_completion +files queries datatypes app_loading one_to_one view_tests responses httpwrappers +extra_regress model_regress select_related many_to_one file_uploads migrations2 auth_tests proxy_models +base transactions builtin_server update_only_fields lookup custom_columns model_formsets_regress proxy_model_inheritance +decorators contenttypes_tests prefetch_related m2m_through_regress transaction_hooks deprecation order_with_respect_to utils_tests +modeladmin redirects_tests get_object_or_404 admin_widgets basic managers_regress humanize_tests custom_managers +migration_test_data_persistence defer string_lookup many_to_one_null datetimes syndication_tests model_options m2m_and_m2o +settings_tests known_related_objects model_indexes field_deconstruction test_exceptions shell filtered_relation csrf_tests +handlers admin_filters dispatch sites_tests model_package conditional_processing null_queries test_client +admin_ordering custom_migration_operations annotations properties messages_tests absolute_url_overrides get_earliest_or_latest expressions +test_client_regress str generic_views aggregation_regress choices or_lookups m2m_signals i18n +update admin_custom_urls user_commands generic_inline_admin admin_changelist logging_tests field_subclassing urlpatterns_reverse +invalid_models_tests m2m_recursive signed_cookies_tests generic_relations_regress m2m_through m2m_intermediary db_typecasts select_related_regress +raw_query mail custom_methods fixtures_regress aggregation postgres_tests admin_views sessions_tests +custom_pk ordering nested_foreign_keys max_lengths null_fk_ordering sites_framework inspectdb expressions_window +no_models template_loader defer_regress custom_lookups many_to_many introspection swappable_models model_meta +get_or_create force_insert_update schema expressions_case generic_relations field_defaults unmanaged_models admin_docs +pagination admin_scripts m2o_recursive servers project_template template_backends dates reserved_names +timezones indexes model_fields admin_checks staticfiles_tests foreign_object check_framework forms_tests +bulk_create file_storage multiple_database urlpatterns reverse_lookup select_related_onetoone admin_inlines sitemaps_tests +serializers test_utils model_inheritance m2m_regress apps dbshell resolve_url model_forms +queryset_pickle select_for_update admin_utils flatpages_tests test_runner validators signing inline_formsets +cache fixtures_model_package signals model_formsets middleware wsgi admin_registration requests +version m2m_multiple migrate_signals validation save_delete_hooks db_functions backends migrations +""".split() + + +def arm_compat(spec): + script = "\n".join(spec.env_script_list) + script = script.replace("python=3.5 -y", "python=3.6 -y") + script = script.replace( + "python=3.6 setuptools==38.2.4 -y\nconda activate testbed", + "python=3.6 -y\nconda activate testbed\npython -m pip install setuptools==38.2.4", + ) + spec.env_script_list = script.splitlines() + spec.repo_script_list = [ + command.replace( + "python -m pip install -e .[test] --verbose", + "python -m pip install jinja2==3.1.6 cython==0.29.36\n" + "python -m pip install -e .[test] --no-build-isolation --verbose", + ) + for command in spec.repo_script_list + ] + if spec.instance_id == "django__django-15103": + spec.repo_script_list.append( + "conda install python=3.9.20 -y && conda clean -afy" + ) + if spec.instance_id == "django__django-10097": + spec.repo_script_list.append( + "cat > /opt/miniconda3/envs/testbed/lib/python3.6/site-packages/" + "sitecustomize.py <<'PY'\n" + "import os\n\n" + "_listdir = os.listdir\n" + f"_order = {DJANGO_10097_TEST_ORDER!r}\n" + "_order_set = set(_order)\n\n" + 'def listdir(path="."):\n' + " entries = _listdir(path)\n" + ' if os.path.abspath(path) != "/testbed/tests":\n' + " return entries\n" + " present = set(entries)\n" + " ordered = [entry for entry in _order if entry in present]\n" + " ordered.extend(entry for entry in entries if entry not in _order_set)\n" + " return ordered\n\n" + "os.listdir = listdir\n" + "PY" + ) + return spec + + +def image_exists(name: str) -> bool: + return ( + subprocess.run( + ["docker", "manifest", "inspect", name], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ).returncode + == 0 + ) + + +def main() -> None: + if not REGISTRY: + raise SystemExit("REGISTRY must identify the destination container registry") + + dataset = load_dataset(DATASET, split="test", revision=REVISION) + rows = [dict(row) for row in dataset.select(range(200))] + ids = [row["instance_id"] for row in rows] + digest = hashlib.sha256(("\n".join(ids) + "\n").encode()).hexdigest() + if len(ids) != 200 or digest != IDS_SHA256: + raise SystemExit( + "The pinned dataset no longer matches the benchmark's first 200 tasks" + ) + + client = docker.from_env() + specs = [ + arm_compat( + make_test_spec( + row, + arch="arm64", + base_image_tag=TAG, + env_image_tag=TAG, + instance_image_tag=TAG, + ) + ) + for row in rows + ] + pending = [ + s + for s in specs + if s.instance_id in COMPAT_IDS + or not image_exists(f"{REGISTRY}/{s.instance_image_key}") + ] + if not pending: + print("All 200 images are already in the registry") + return + + _, failed = build_env_images(client, pending, max_workers=WORKERS) + if failed: + raise SystemExit(f"Failed to build {len(failed)} environment images") + + def publish(spec): + remote = f"{REGISTRY}/{spec.instance_image_key}" + build_instance_image(spec, client, None, spec.instance_id in COMPAT_IDS) + image = client.images.get(spec.instance_image_key) + if image.attrs["Architecture"] not in {"arm64", "aarch64"}: + raise RuntimeError(f"Non-ARM image built for {spec.instance_id}") + subprocess.run(["docker", "tag", spec.instance_image_key, remote], check=True) + subprocess.run(["docker", "push", remote], check=True) + return remote + + errors = [] + with ThreadPoolExecutor(max_workers=WORKERS) as pool: + futures = {pool.submit(publish, spec): spec.instance_id for spec in pending} + for future in as_completed(futures): + try: + print(f"PUSHED {future.result()}", flush=True) + except Exception as exc: + errors.append(futures[future]) + print(f"FAILED {futures[future]}: {exc}", flush=True) + if errors: + raise SystemExit(f"Failed images ({len(errors)}): {' '.join(errors)}") + + +if __name__ == "__main__": + main() From 5d355de66d396aa9ebb394a276cbe33db607dbe3 Mon Sep 17 00:00:00 2001 From: Harshil Vagadia Date: Thu, 10 Sep 2026 12:01:20 -0700 Subject: [PATCH 3/4] docs(agentic): pin artifacts and explain ARM64 --- examples/10_Agentic_Inference/README.md | 32 +++++++++++++------ .../10_Agentic_Inference/build_and_push.py | 14 ++++++++ 2 files changed, 37 insertions(+), 9 deletions(-) diff --git a/examples/10_Agentic_Inference/README.md b/examples/10_Agentic_Inference/README.md index 240269ab3..91d94cd57 100644 --- a/examples/10_Agentic_Inference/README.md +++ b/examples/10_Agentic_Inference/README.md @@ -153,6 +153,19 @@ REGISTRY=registry.example.com/group/project \ Set `WORKERS` to change the default build and push concurrency of `16`. Images are tagged `v4.1.0-arm64`, and interrupted runs can be resumed with the same command. +##### ARM64 Compatibility and x86 Comparability + +The builder keeps the benchmark workload fixed by pinning the SWE-bench Verified dataset revision and validating the ordered first 200 instance IDs. It does not change the dataset rows, repository or test patches, evaluation commands, or set of tests. The generated images are nevertheless architecture-specific compatibility builds: `arm_compat` adjusts only the environment and repository setup commands needed to build and run the same instances on ARM64. + +The compatibility adjustments are: + +- Replace legacy Python 3.5 environment requests with Python 3.6, and install `setuptools==38.2.4` with pip after activating the environment when that exact legacy setup command is present. +- Preinstall `jinja2==3.1.6` and `cython==0.29.36`, then disable build isolation for the matching editable test-dependency installation. +- Pin `django__django-15103` to Python 3.9.20. +- For `django__django-10097`, replay the recorded x86 `/testbed/tests` directory enumeration order. The old Django runner consumes unsorted directory entries and leaks test state through `generic_inline_admin`; ARM64 exposes a different filesystem order. Tests absent from the recorded order are appended, so this changes only discovery order, not the test set. + +These substitutions intentionally preserve the benchmark instances and evaluation logic, but the resulting software environment is not byte-for-byte identical to the x86 image. The script rebuilds the two instance-specific compatibility images on every run and verifies that every image is ARM64 before pushing it. + ## Run The Client Update the first `datasets` entry (`name` and `path`), `model_params.name`, and `endpoint_config.endpoints` as needed. Then select the matching model config and run it from the repo root: @@ -248,20 +261,20 @@ Under the MLPerf Endpoint Benchmark rules, submitters must use only the approved Approved model checkpoints: -- [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3) -- [nvidia/Kimi-K3-NVFP4](https://huggingface.co/nvidia/Kimi-K3-NVFP4) +- [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3/tree/f831ab66814297da540d832a5235f8e904f29d06) (`f831ab66814297da540d832a5235f8e904f29d06`) +- [nvidia/Kimi-K3-NVFP4](https://huggingface.co/nvidia/Kimi-K3-NVFP4/tree/b2428a0b83a8b712ff2e1a8448a103d4175341f1) (`b2428a0b83a8b712ff2e1a8448a103d4175341f1`) Approved speculative-decoding heads: -- [RadixArk/Kimi-K3-DSpark](https://huggingface.co/RadixArk/Kimi-K3-DSpark) -- [Inferact/Kimi-K3-DSpark](https://huggingface.co/Inferact/Kimi-K3-DSpark) +- [RadixArk/Kimi-K3-DSpark](https://huggingface.co/RadixArk/Kimi-K3-DSpark/tree/3c5bac301d9cf392706189d82ed947feca6c2f0f) (`3c5bac301d9cf392706189d82ed947feca6c2f0f`) +- [Inferact/Kimi-K3-DSpark](https://huggingface.co/Inferact/Kimi-K3-DSpark/tree/cf6b8244620e7ea4b0651d214f28e89eac75bed6) (`cf6b8244620e7ea4b0651d214f28e89eac75bed6`) #### DSV4 Approved model checkpoints: -- [deepseek-ai/DeepSeek-V4-Pro-0813](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813) -- [nvidia/DeepSeek-V4-Pro-0813-NVFP4](https://huggingface.co/nvidia/DeepSeek-V4-Pro-0813-NVFP4) +- [deepseek-ai/DeepSeek-V4-Pro-0813](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813/tree/72e1d3230f6c080a530b0a1d46f8eb4602340597) (`72e1d3230f6c080a530b0a1d46f8eb4602340597`) +- [nvidia/DeepSeek-V4-Pro-0813-NVFP4](https://huggingface.co/nvidia/DeepSeek-V4-Pro-0813-NVFP4/tree/949138637e8e8fe190335be2396af62187dc8a83) (`949138637e8e8fe190335be2396af62187dc8a83`) Both approved DSV4 checkpoints include bundled DSpark speculative-decoding heads. @@ -269,7 +282,8 @@ Both approved DSV4 checkpoints include bundled DSpark speculative-decoding heads Approved model checkpoints: -- [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) -- [nvidia/Qwen3.6-35B-A3B-NVFP4](https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4) (contains both W4A16 and W4A4 scales) +- [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B/tree/995ad96eacd98c81ed38be0c5b274b04031597b0) (`995ad96eacd98c81ed38be0c5b274b04031597b0`) +- [Qwen/Qwen3.6-35B-A3B-FP8](https://huggingface.co/Qwen/Qwen3.6-35B-A3B-FP8/tree/95a723d08a9490559dae23d0cff1d9466213d989) (`95a723d08a9490559dae23d0cff1d9466213d989`) +- [nvidia/Qwen3.6-35B-A3B-NVFP4](https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4/tree/1355db6a052410cfd62085d94b58866fd0f2c3c5) (`1355db6a052410cfd62085d94b58866fd0f2c3c5`; contains both W4A16 and W4A4 scales) -Both approved Qwen checkpoints include their native MTP speculative-decoding heads. +The approved Qwen checkpoints include their native MTP speculative-decoding heads. diff --git a/examples/10_Agentic_Inference/build_and_push.py b/examples/10_Agentic_Inference/build_and_push.py index f3d427fa8..2d9b67c81 100644 --- a/examples/10_Agentic_Inference/build_and_push.py +++ b/examples/10_Agentic_Inference/build_and_push.py @@ -53,13 +53,25 @@ def arm_compat(spec): + """Adjust generated setup commands without changing benchmark semantics. + + The dataset row, repository and test patches, evaluation commands, and test + set remain unchanged. These substitutions only make the generated x86 + environment and repository setup commands buildable on native ARM64. The + Django 10097 exception reproduces x86 test discovery order because its old + test runner leaks state between tests. + """ script = "\n".join(spec.env_script_list) + # Replace legacy interpreter/setup commands only when their exact generated + # forms occur; leave all other environment commands untouched. script = script.replace("python=3.5 -y", "python=3.6 -y") script = script.replace( "python=3.6 setuptools==38.2.4 -y\nconda activate testbed", "python=3.6 -y\nconda activate testbed\npython -m pip install setuptools==38.2.4", ) spec.env_script_list = script.splitlines() + # Prepare the ARM source build while preserving the repository checkout, + # requested test extras, and evaluation commands generated by SWE-bench. spec.repo_script_list = [ command.replace( "python -m pip install -e .[test] --verbose", @@ -69,10 +81,12 @@ def arm_compat(spec): for command in spec.repo_script_list ] if spec.instance_id == "django__django-15103": + # This instance requires the pinned ARM-compatible Python environment. spec.repo_script_list.append( "conda install python=3.9.20 -y && conda clean -afy" ) if spec.instance_id == "django__django-10097": + # Reproduce x86 directory enumeration without adding or removing tests. spec.repo_script_list.append( "cat > /opt/miniconda3/envs/testbed/lib/python3.6/site-packages/" "sitecustomize.py <<'PY'\n" From 54e8c6ada8a72610764fab42175c0792323f689e Mon Sep 17 00:00:00 2001 From: Harshil Vagadia Date: Thu, 10 Sep 2026 12:14:20 -0700 Subject: [PATCH 4/4] docs(agentic): add reference accuracy means --- examples/10_Agentic_Inference/README.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/examples/10_Agentic_Inference/README.md b/examples/10_Agentic_Inference/README.md index 91d94cd57..29a5074e0 100644 --- a/examples/10_Agentic_Inference/README.md +++ b/examples/10_Agentic_Inference/README.md @@ -247,11 +247,13 @@ Qwen3.6-35B-A3B submissions must set `accuracy_config.extras.swebench_template: Every Kimi K3 and Qwen3.6-35B-A3B submitted Pareto point must satisfy all of the model-specific accuracy thresholds below. For these models, SWE-bench accuracy is evaluated using mean-of-N: average one SWE-bench accuracy result from each of the [four mandatory regions](https://github.com/mlcommons/endpoints_policies/blob/main/endpoints_rules.md#54-regions-of-interest) (`N = 4`), then compare that mean with the model-specific SWE-bench threshold below. The DSV4 accuracy thresholds and SWE-bench evaluation policy are TBD. -| Metric | Kimi K3 | Qwen3.6-35B-A3B | DSV4 | -| ------------------ | ---------------: | ---------------: | ---: | -| Inline accuracy | `>= 58.32%` | `>= 55.86%` | TBD | -| OSL per-turn mean | `390-475` tokens | `355-434` tokens | TBD | -| SWE-bench accuracy | `>= 93.5%` | `>= 69%` | TBD | +Reference mean values are shown in parentheses. + +| Metric | Kimi K3 | Qwen3.6-35B-A3B | DSV4 | +| ------------------ | --------------------: | ---------------------: | ---: | +| Inline accuracy | `>= 58.32%` (`58.9%`) | `>= 55.86%` (`56.43%`) | TBD | +| OSL per-turn mean | `390-475` tokens | `355-434` tokens | TBD | +| SWE-bench accuracy | `>= 93.5%` (`94.83%`) | `>= 69%` (`71.7%`) | TBD | ### Approved Checkpoints and Speculative-Decoding Heads