Bootstrap official Qwen3 training evidence

Author: sanbuphy

README.md

@@ -1389,6 +1389,10 @@ natural-language prompt claim.
 tokenizer-generated prompts (English, Chinese, C++ and Qwen chat): 32/32 workers, 14 direct rows,
 two FP32-attributed splits, 4/4 strict B1/B2 oracles and 8/8 exact KV rows. It extends the explicit
 policy to these short prompts without making a language-quality or default-policy claim.
+[Experiment 381](docs/optimization-log/experiments/381-qwen3-training-smoke.md) moves official
+Qwen3 into training: FP32 B1/T32 completes forward/backward/AdamW with a `2.38e-7` loss difference
+and a `2.57e-10` observed-parameter difference. BF16 executes but is slower and has a `0.00996`
+loss gap, so complete gradients and multi-step trajectories remain required.
 [Experiment 122](docs/optimization-log/experiments/122-official-fp8-static-scale.md) runs official
 Qwen/DeepSeek with single-representation FP8 Linear weights. Residency drops sharply, but every
 static-scale precision gate fails, so FP8 remains experimental and opt-in.

benchmarks/results/2026-08-27-qwen3-training-smoke/README.md

@@ -0,0 +1,32 @@
+# Official Qwen3-0.6B training-step smoke
+
+This is the first official Qwen3 training execution evidence. A temporary local manifest adds only
+`training.learning_rate=1e-5`; model revision, weights and token seed remain unchanged.
+
+Both runs use B1/T32, zero warm-up, one measured step and fresh microLLM/PyTorch processes.
+
+## FP32
+
+- both frameworks execute forward, loss, backward and AdamW update;
+- loss: `2.37676549 / 2.376765728`, absolute difference `2.38e-7`;
+- observed final-norm parameter after update differs by `2.57e-10`;
+- peak: `9.778 / 12.102 GB` microLLM/PyTorch;
+- single-process throughput ratio: `1.119x` (smoke only, not a performance claim).
+
+## BF16 forward with FP32 masters
+
+- both frameworks execute and update a parameter;
+- microLLM retains 196 BF16 training mirrors over 880,803,840 bytes;
+- loss: `2.377071381 / 2.367111206`, absolute difference `0.009960`;
+- microLLM reaches only `0.5969x` the matched PyTorch throughput and `0.7900x` its own FP32 smoke;
+- microLLM peak is `1.0901x` its FP32 smoke because FP32 masters, gradients and AdamW state remain.
+
+The FP32 row is an execution/alignment smoke. The BF16 row is an execution smoke with an explicit
+loss and performance gap. Neither row proves complete parameter/gradient alignment, multi-step loss
+trajectory or repeated performance. Those require the next parameter-signature and trajectory gate.
+
+Files:
+
+- `fp32-raw.jsonl` / `fp32-summary.json`;
+- `bf16-raw.jsonl` / `bf16-summary.json`;
+- `summary.json`: compact decision.

benchmarks/results/2026-08-27-qwen3-training-smoke/bf16-raw.jsonl

@@ -0,0 +1,2 @@
+{"adamw_bf16_multi_tensor_elements": 0, "adamw_bf16_multi_tensor_tensors": 0, "adamw_bf16_multi_tensor_threshold": 0, "adamw_implementation": "auto", "adamw_moment_precision": "fp32", "adamw_moment_state_bytes": 4768399360, "adamw_multi_tensor_update": false, "architecture": "gfx942:sramecc+:xnack-", "attention_context_layout_fusion": true, "attention_gemm_scale_fusion": false, "attention_gqa_forward_value_broadcast": false, "attention_gqa_value_broadcast": false, "attention_layout_plan_cache": false, "attention_layout_plan_cache_entries": 0, "attention_layout_plan_cache_hits": 0, "attention_layout_plan_cache_misses": 0, "attention_paired_gqa_repeat": false, "attention_rope_layout_fusion": true, "batch": 1, "bf16_algorithm_registrations": 0, "bf16_algorithm_spec": "", "bf16_training_mirror_bytes": 880803840, "bf16_training_mirror_tensors": 196, "bf16_weight_mirrors_enabled": true, "compute_dtype": "bf16_linear_fp32_master", "context": 32, "device": "hip:0", "device_name": "AMD Instinct MI300X VF", "diagnostics_enabled": false, "engine_allocation_calls": 2254, "engine_backend_allocation_calls": 1280, "engine_backend_deallocation_calls": 0, "engine_cache_reuse_calls": 974, "engine_cached_bytes": 241995148, "engine_current_bytes": 10417602816, "engine_deallocation_calls": 1944, "engine_peak_bytes": 10659004680, "engine_reserved_bytes": 10659597964, "engine_total_allocated_bytes": 2750344844, "final_loss": 2.377071381, "first_loss": 2.377071381, "fp32_gate_up_weight_gradient_solution_index": -1, "fp32_solution_dispatches": 0, "fp32_solution_registered_entries": 0, "fp32_solution_registry_hits": 0, "fp32_solution_registry_misses": 0, "fp32_weight_bytes": 2384199680, "framework": "microllm", "gate_up_parameter_elements": 0, "gate_up_parameter_tensors": 0, "gate_up_parameters_output_written": false, "hip_driver_version": 71399004, "hip_runtime_version": 71399004, "load_current_engine_bytes": 2384199680, "load_device_to_device_calls": 0, "load_device_to_host_calls": 0, "load_host_to_device_bytes": 1192099840, "load_host_to_device_calls": 310, "load_ms": 2323.569196, "load_peak_engine_bytes": 2695364608, "loaded_tensors": 310, "loss": 2.377071381, "loss_trajectory_output_written": false, "loss_trajectory_steps": 1, "mean_optimizer_ms": 6.344704, "mean_step_ms": 7188.488699, "measured_ms": 7188.488699, "measurement_profile": "smoke", "milliseconds_per_token": 224.640271844, "model": "qwen3-0.6b", "observed_parameter_after": 3.937509537, "observed_parameter_before": 3.9375, "optimizer_device_to_host_bytes": 0, "optimizer_device_to_host_calls": 0, "optimizer_host_to_device_bytes": 0, "optimizer_host_to_device_calls": 0, "optimizer_ms": 6.344704, "optimizer_timing_boundary": "post_backward_sync", "pair_order": ["microllm", "pytorch"], "parameter_changed": true, "parameter_count": 596049920, "peak_bytes": 10659004680, "precision": "bf16", "process_run": 1, "record_type": "official_training_shape_measurement", "revision": "c916fa4defd319b7d4e4da17604ca7338f4d99f5", "schema_version": 1, "status": "pass", "step_ms": 7188.488699, "steps": 1, "tied_embedding_sparse_add": true, "tokens_per_second": 4.451561565, "trained_tokens": 32, "unique_gradient_inplace_add": false, "warmup": 0, "warmup_ms": 0.0156}
+{"architecture": "gfx942:sramecc+:xnack-", "batch": 1, "compute_dtype": "bf16_amp", "context": 32, "device": "cuda", "device_current_allocated_bytes": 9699042816, "device_current_reserved_bytes": 12241076224, "device_discovery_workaround": "amdsmi_zero_fallback_to_hip_runtime", "device_name": "AMD Instinct MI300X VF", "device_peak_allocated_bytes": 12093491200, "device_peak_reserved_bytes": 12241076224, "final_loss": 2.3671112060546875, "first_loss": 2.3671112060546875, "fp32_weight_bytes": 2384199680, "framework": "pytorch", "load_ms": 3258.2290559075773, "loaded_tensors": 311, "loss": 2.3671112060546875, "mean_optimizer_ms": 79.949168022722, "mean_step_ms": 4290.810153353959, "measured_ms": 4290.810153353959, "measurement_profile": "smoke", "milliseconds_per_token": 134.08781729231123, "mode": "train", "model": "qwen3-0.6b", "observed_parameter_after": 3.937509536743164, "observed_parameter_before": 3.9375, "optimizer_ms": 79.949168022722, "pair_order": ["microllm", "pytorch"], "parameter_changed": true, "parameter_count": 596049920, "peak_bytes": 12093491200, "precision": "bf16", "process_run": 1, "pytorch_state_tensors": 311, "record_type": "official_training_shape_measurement", "resident_weight_bytes": 2384199680, "revision": "c916fa4defd319b7d4e4da17604ca7338f4d99f5", "schema_version": 1, "status": "pass", "step_ms": 4290.810153353959, "steps": 1, "tokens_per_second": 7.4577990767050935, "torch_hip_version": "7.13.99004", "torch_version": "2.11.0+rocm7.13.0rc2", "trained_tokens": 32, "transformers_version": "4.55.4", "warmup": 0, "warmup_ms": 0.05065323784947395}

benchmarks/results/2026-08-27-qwen3-training-smoke/bf16-summary.json

@@ -0,0 +1,25 @@
+{
+  "pairing": "fresh processes; order alternates each run",
+  "precision": "bf16",
+  "rows": [
+    {
+      "batch": 1,
+      "context": 32,
+      "microllm_final_loss": 2.377071381,
+      "microllm_peak_bytes": 10659004680.0,
+      "microllm_tokens_per_second": 4.451561565,
+      "model": "qwen3-0.6b",
+      "peak_memory_ratio": 0.8813835892153293,
+      "pytorch_final_loss": 2.3671112060546875,
+      "pytorch_peak_allocated_bytes": 12093491200.0,
+      "pytorch_tokens_per_second": 7.4577990767050935,
+      "revision": "c916fa4defd319b7d4e4da17604ca7338f4d99f5",
+      "status": "pass",
+      "throughput_ratio_microllm_over_pytorch": 0.5969001737931952
+    }
+  ],
+  "runs_per_framework": 1,
+  "schema_version": 1,
+  "status": "pass",
+  "track": "official_training_shape_matrix"
+}

benchmarks/results/2026-08-27-qwen3-training-smoke/fp32-raw.jsonl

@@ -0,0 +1,2 @@
+{"adamw_bf16_multi_tensor_elements": 0, "adamw_bf16_multi_tensor_tensors": 0, "adamw_bf16_multi_tensor_threshold": 0, "adamw_implementation": "auto", "adamw_moment_precision": "fp32", "adamw_moment_state_bytes": 4768399360, "adamw_multi_tensor_update": false, "architecture": "gfx942:sramecc+:xnack-", "attention_context_layout_fusion": true, "attention_gemm_scale_fusion": false, "attention_gqa_forward_value_broadcast": false, "attention_gqa_value_broadcast": false, "attention_layout_plan_cache": false, "attention_layout_plan_cache_entries": 0, "attention_layout_plan_cache_hits": 0, "attention_layout_plan_cache_misses": 0, "attention_paired_gqa_repeat": false, "attention_rope_layout_fusion": true, "batch": 1, "bf16_algorithm_registrations": 0, "bf16_algorithm_spec": "", "bf16_training_mirror_bytes": 0, "bf16_training_mirror_tensors": 0, "bf16_weight_mirrors_enabled": false, "compute_dtype": "float32", "context": 32, "device": "hip:0", "device_name": "AMD Instinct MI300X VF", "diagnostics_enabled": false, "engine_allocation_calls": 2030, "engine_backend_allocation_calls": 1279, "engine_backend_deallocation_calls": 0, "engine_cache_reuse_calls": 751, "engine_cached_bytes": 241798540, "engine_current_bytes": 9536798976, "engine_deallocation_calls": 1720, "engine_peak_bytes": 9778200840, "engine_reserved_bytes": 9778597516, "engine_total_allocated_bytes": 2730159756, "final_loss": 2.37676549, "first_loss": 2.37676549, "fp32_gate_up_weight_gradient_solution_index": -1, "fp32_solution_dispatches": 0, "fp32_solution_registered_entries": 0, "fp32_solution_registry_hits": 0, "fp32_solution_registry_misses": 0, "fp32_weight_bytes": 2384199680, "framework": "microllm", "gate_up_parameter_elements": 0, "gate_up_parameter_tensors": 0, "gate_up_parameters_output_written": false, "hip_driver_version": 71399004, "hip_runtime_version": 71399004, "load_current_engine_bytes": 2384199680, "load_device_to_device_calls": 0, "load_device_to_host_calls": 0, "load_host_to_device_bytes": 1192099840, "load_host_to_device_calls": 310, "load_ms": 2248.083205, "load_peak_engine_bytes": 2695364608, "loaded_tensors": 310, "loss": 2.37676549, "loss_trajectory_output_written": false, "loss_trajectory_steps": 1, "mean_optimizer_ms": 6.192707, "mean_step_ms": 5678.741314, "measured_ms": 5678.741314, "measurement_profile": "smoke", "milliseconds_per_token": 177.460666062, "model": "qwen3-0.6b", "observed_parameter_after": 3.937509537, "observed_parameter_before": 3.9375, "optimizer_device_to_host_bytes": 0, "optimizer_device_to_host_calls": 0, "optimizer_host_to_device_bytes": 0, "optimizer_host_to_device_calls": 0, "optimizer_ms": 6.192707, "optimizer_timing_boundary": "post_backward_sync", "pair_order": ["microllm", "pytorch"], "parameter_changed": true, "parameter_count": 596049920, "peak_bytes": 9778200840, "precision": "fp32", "process_run": 1, "record_type": "official_training_shape_measurement", "revision": "c916fa4defd319b7d4e4da17604ca7338f4d99f5", "schema_version": 1, "status": "pass", "step_ms": 5678.741314, "steps": 1, "tied_embedding_sparse_add": true, "tokens_per_second": 5.635051542, "trained_tokens": 32, "unique_gradient_inplace_add": false, "warmup": 0, "warmup_ms": 0.01414}
+{"architecture": "gfx942:sramecc+:xnack-", "batch": 1, "compute_dtype": "fp32", "context": 32, "device": "cuda", "device_current_allocated_bytes": 9698280960, "device_current_reserved_bytes": 12222201856, "device_discovery_workaround": "amdsmi_zero_fallback_to_hip_runtime", "device_name": "AMD Instinct MI300X VF", "device_peak_allocated_bytes": 12102453248, "device_peak_reserved_bytes": 12222201856, "final_loss": 2.376765727996826, "first_loss": 2.376765727996826, "fp32_weight_bytes": 2384199680, "framework": "pytorch", "load_ms": 3382.7795470133424, "loaded_tensors": 311, "loss": 2.376765727996826, "mean_optimizer_ms": 79.70799598842859, "mean_step_ms": 6356.529301963747, "measured_ms": 6356.529301963747, "measurement_profile": "smoke", "milliseconds_per_token": 198.64154068636708, "mode": "train", "model": "qwen3-0.6b", "observed_parameter_after": 3.937509536743164, "observed_parameter_before": 3.9375, "optimizer_ms": 79.70799598842859, "pair_order": ["microllm", "pytorch"], "parameter_changed": true, "parameter_count": 596049920, "peak_bytes": 12102453248, "precision": "fp32", "process_run": 1, "pytorch_state_tensors": 311, "record_type": "official_training_shape_measurement", "resident_weight_bytes": 2384199680, "revision": "c916fa4defd319b7d4e4da17604ca7338f4d99f5", "schema_version": 1, "status": "pass", "step_ms": 6356.529301963747, "steps": 1, "tokens_per_second": 5.034193736842229, "torch_hip_version": "7.13.99004", "torch_version": "2.11.0+rocm7.13.0rc2", "trained_tokens": 32, "transformers_version": "4.55.4", "warmup": 0, "warmup_ms": 0.05153613165020943}

benchmarks/results/2026-08-27-qwen3-training-smoke/fp32-summary.json

@@ -0,0 +1,25 @@
+{
+  "pairing": "fresh processes; order alternates each run",
+  "precision": "fp32",
+  "rows": [
+    {
+      "batch": 1,
+      "context": 32,
+      "microllm_final_loss": 2.37676549,
+      "microllm_peak_bytes": 9778200840.0,
+      "microllm_tokens_per_second": 5.635051542,
+      "model": "qwen3-0.6b",
+      "peak_memory_ratio": 0.8079519614435118,
+      "pytorch_final_loss": 2.376765727996826,
+      "pytorch_peak_allocated_bytes": 12102453248.0,
+      "pytorch_tokens_per_second": 5.034193736842229,
+      "revision": "c916fa4defd319b7d4e4da17604ca7338f4d99f5",
+      "status": "pass",
+      "throughput_ratio_microllm_over_pytorch": 1.1193553201499686
+    }
+  ],
+  "runs_per_framework": 1,
+  "schema_version": 1,
+  "status": "pass",
+  "track": "official_training_shape_matrix"
+}

benchmarks/results/2026-08-27-qwen3-training-smoke/summary.json

@@ -0,0 +1,35 @@
+{
+  "schema_version": 1,
+  "status": "pass_fp32_alignment_bf16_execution_smoke",
+  "model": "Qwen/Qwen3-0.6B",
+  "shape": {"batch": 1, "context": 32, "warmup": 0, "steps": 1},
+  "learning_rate": 0.00001,
+  "fp32": {
+    "microllm_loss": 2.37676549,
+    "pytorch_loss": 2.376765727996826,
+    "absolute_loss_difference": 2.3799682624670027e-7,
+    "observed_parameter_after_difference": 2.5683588589231476e-10,
+    "microllm_tokens_per_second": 5.635051542,
+    "pytorch_tokens_per_second": 5.034193736842229,
+    "single_process_throughput_ratio": 1.1193553201499686,
+    "microllm_peak_bytes": 9778200840,
+    "pytorch_peak_bytes": 12102453248
+  },
+  "bf16": {
+    "microllm_loss": 2.377071381,
+    "pytorch_loss": 2.3671112060546875,
+    "absolute_loss_difference": 0.009960174945312428,
+    "observed_parameter_after_difference": 2.5683588589231476e-10,
+    "microllm_tokens_per_second": 4.451561565,
+    "pytorch_tokens_per_second": 7.4577990767050935,
+    "single_process_throughput_ratio": 0.5969001737931952,
+    "microllm_over_microllm_fp32_throughput": 0.7899770803906517,
+    "microllm_peak_bytes": 10659004680,
+    "pytorch_peak_bytes": 12093491200,
+    "microllm_peak_over_fp32": 1.0900783134251923,
+    "bf16_training_mirror_tensors": 196,
+    "bf16_training_mirror_bytes": 880803840
+  },
+  "decision": "FP32 official training path is ready for complete-gradient and multi-step gates; BF16 remains execution-only and is not a speed policy",
+  "boundary": "one step, one shape and one process per framework; no complete parameter/gradient, trajectory or repeated-performance claim"
+}

docs/HUGGINGFACE.md

@@ -71,6 +71,9 @@ exact自然prompt使用`make_qwen3_natural_prompt_manifest.py`从同一tokenizer
 代码18和chat24 token，再由`qwen3_exact_prompt_matrix.py`逐条按原长度运行。32/32 worker、
 14/16直接一致，两处B1/B2分叉的四份strict oracle均支持候选。
 
+官方Qwen3训练step也已进入smoke：FP32 B1/T32 loss差2.38e-7，观测参数更新差2.57e-10；BF16
+执行成功但loss差0.00996且没有加速。完整参数/梯度与多步轨迹仍是单独门，不能由一个参数代替。
+
 ### 一条命令准备固定 fixture
 
 仓库不会提交数GB权重，但会固定来源、revision、许可和结构预期：

docs/development/2026-08-27-qwen3-training-smoke.md

@@ -0,0 +1,12 @@
+# Qwen3官方训练step smoke
+
+日期：2026-08-27
+状态：FP32执行/局部对齐，BF16仅执行
+
+B1/T32、1 step、lr1e-5真实运行官方Qwen3。FP32 micro/PyTorch loss差2.38e-7，观测final norm
+参数更新后差2.57e-10；峰值9.78/12.10GB。单样本吞吐不作排名。
+
+BF16-forward+FP32 master/gradient/AdamW也跑通，但loss差0.00996；196份mirror占880,803,840字节，
+micro吞吐只有PyTorch的0.5969和自身FP32的0.7900，峰值反而1.0901倍。BF16不是当前优化路径。
+
+下一步必须增加全参数/梯度签名输出和多步轨迹。只看一个final norm参数不能证明完整backward。

docs/development/NEXT_STEPS.md

@@ -87,6 +87,11 @@ First target: one pinned small dense checkpoint, not every Qwen release.
   complete32/32, direct14/16, strict B1/B2 oracles4/4, KV8/8 and B2 rows4/4;
 - [ ] expand beyond four short prompts to a versioned prompt suite, longer documents,
   sampling and quality evaluation; Radeon/backend and memory-constrained devices remain open;
+- [x] execute official Qwen3 B1/T32 training once in FP32 and BF16: FP32 loss/one
+  parameter align at2.38e-7/2.57e-10; BF16 executes but has0.00996 loss gap,
+  0.5969x PyTorch throughput and no memory win;
+- [ ] add complete Qwen3 parameter/gradient signatures, then run multi-step FP32/BF16
+  trajectories before any official training or SFT readiness claim;
 - [x] preserve identical-input batch rows instead of throwing a generic PyTorch worker
   failure; classify T1024/B2/N8 as `batch_invariance_mismatch`, with microLLM 2/2,
   Transformers BF16 474/2 and common FP32 oracle 2;