Awq - #10
Open
skt0725 wants to merge 6 commits into
Open
Conversation
- pseudo_quantize_tensor: per-group clip ratio search (--clip-search) - DESIGN_NOTES: kowikitext 41.61% (best), calibration language effect
wikitext2 calibration produced huge act_scales for layers with activation outliers (e.g. layers.2.mlp.down_proj). Computing w * act_scales^alpha in fp16 overflowed to inf, propagating NaN into scales and corrupting the model (KMMLU 9.87%, below random). - Quantization math now runs in fp32; scales/weights stored back in the original dtype (fp16) - w_int and w_dequant are derived from the stored fp16 scale so the INT4 export and in-model weights match exactly
…ctive)
Scale/clip search can now minimize output error ||Q(W·s)·(X/s) − W·X||
instead of weight reconstruction error, matching the official AWQ
objective. Calibration hooks cache a per-layer input subsample
(512 rows, fp16/CPU, ~0.7GB total) to make this possible without
block-sequential execution infrastructure.
- calibration.py: ActivationCollector optionally caches input rows,
spread evenly across calibration samples; run_calibration returns
(stats, input_cache)
- quantize.py: search_best_scale and pseudo_quantize_tensor accept x
for output-error objective; clip selection uses per-group output
error (same approximation as official AutoAWQ); search runs on GPU
when x is provided (21 large matmuls per layer)
- run_awq.py: --search-mode {weight,output} flag
Motivation: weight-error clip search hurt KMMLU by 3.2%p (38.51% vs
41.73% baseline) because it clips exactly the salient weights AWQ is
meant to protect. Unit test confirms: with activation outliers,
weight+clip worsens output MSE while output+clip improves it.
…earch) Output-error search (roadmap Pseudo-Lab#1) reaches 42.98% KMMLU on kowikitext, closing the gap to official AutoAWQ from -4.08%p to -1.13%p (+1.25%p over weight-error search). Clip experiments confirm objective dependence: weight-error+clip -3.22%p (harmful) vs output-error+clip -1.45%p; best config is output-error search without clip. - Reworked results table: split by search objective and clip - Added VRAM compression table (8.04GB -> 2.67GB, 3.0x) - Marked roadmaps Pseudo-Lab#1/Pseudo-Lab#2 done; Pseudo-Lab#3 (LayerNorm fold) is the remaining item - Added experiment log: env migration, fp16 overflow fix, roadmap Pseudo-Lab#1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implementation - float16 overflow fixed
Experiment- weight clipping (done), objective function (on-progress)
Documentation - Roadmap & Quantitative results in DESIGN_NOTES.md