add ablation study for FOEM - #9
Open
minje-KIM wants to merge 9 commits into
Open
Conversation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…puts
- quantize_mistral.py: 저장 경로를 {model}_{method}_{bits}bit 구조로 개편,
양자화 완료 후 WikiText-2 PPL 자동 평가 및 README 자동 생성 추가.
write_readme()에 압축률 분석·알고리즘 원리·loss 해석·모듈 민감도·
레이어 깊이·하이퍼파라미터 의미 6개 심층 분석 섹션 추가.
sm_120(Blackwell) 환경에서 Marlin JIT 실패 시 gptq_triton 백엔드 폴백 처리.
- eval_ppl.py: WikiText-2 sliding-window PPL 평가 스크립트 업데이트.
- Ministral-3-3B-Instruct-2512-BF16_foem_3bit: FOEM 3-bit 양자화 결과
(PPL 10.8654, 2.7 GB).
- Ministral-3-3B-Instruct-2512-BF16_gptq_4bit: GPTQ 4-bit 양자화 결과
(PPL 8.7238, 3.0 GB).
- logs/: 양자화 실행 로그.
- pseudo/ 하위 구 실험 파일 정리(삭제).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds log-likelihood-based 5-shot KMMLU scoring (eval_kmmlu.py), wires it into quantize_mistral.py's post-quantization pipeline, and records results (45 subjects, full test set) for both the GPTQ 4-bit and FOEM 3-bit Ministral-3-3B models in their READMEs and a comparison report. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…s FOEM) - eval_kmmlu.py: fix tie_word_embeddings override for BF16 base model load (forcing False caused lm_head random init since checkpoint uses tied weights); add -u flag to shebang for unbuffered stdout in redirected nohup runs - KMMLU_REPORT.md: expand to 3-way comparison including BF16 baseline (micro 46.88% / macro 46.02%, 47.1분, 35,030 questions) GPTQ 4-bit loses -2.69%p, FOEM 3-bit loses -11.30%p vs BF16 - update_report_with_base.py: utility script that parsed base_kmmlu.log and regenerated the report - logs/base_kmmlu.log: raw evaluation log for BF16 base model Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three-way comparison (BF16 / GPTQ 4-bit / FOEM 3-bit) now complete for all 4 example questions. Key finding: overconfidence on wrong answers worsens with quantization depth (BF16→GPTQ→FOEM), while factoid knowledge (Computer-Science) degrades gracefully. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Education: FOEM-only correct despite lowest overall accuracy (60% on C vs BF16/GPTQ wrong at ~66% D) — shows per-question quantization anomaly. Patent: All correct, confidence degrades BF16(84.8%)→GPTQ(64.5%)→FOEM(45.8%). Law: All wrong but each picks a different answer; GPTQ less overconfident than BF16. Information-Technology: FOEM-only wrong due to D→C probability rank inversion. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…beta/group_size/mixed-precision), and FastAPI serving - Add K-DTCBench VQA benchmark eval (pseudo/eval_kdtcbench.py) alongside existing KMMLU, with report - Run FOEM 3bit ablation across 5 axes vs baseline (group_size=128, alpha=0, beta=0.2): act-order, calibration language, alpha/beta grid, group_size, mixed-precision layer skip (11 configs total, documented in pseudo/ABLATION_REPORT.md). group_size=32 is the only config beating baseline on both PPL and K-DTCBench. - Quantize FOEM 4bit baseline (vLLM-compatible bit width) after discovering vLLM's GPTQ backend rejects 3-bit checkpoints - Add FastAPI server (pseudo/serve_foem.py) serving the FOEM 3bit baseline with a simple web chat UI, replacing vLLM after hitting unresolved Mistral3/Pixtral multimodal processor bugs in vLLM 0.24.0 - Reorganize all quantization outputs under FOEM/ subdirectory; exclude tokenizer.json/tekken.json (duplicated ~17MB per model dir) from git alongside existing *.safetensors/*.log exclusions Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… work Remote main independently reorganized the repo into FOEM/ (same intent as the local commit). Resolved rename/rename and rename/delete conflicts: - Kept our .gitignore (superset: excludes tokenizer.json/tekken.json too) - Kept tokenizer.json untracked per our size-reduction decision - Preserved both FOEM_3bit_alpha0.3's own config files and gptq_4bit's config files (git mis-detected a false rename between them due to byte-identical tokenizer content)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ablation study 추가로 진행되어 해당 내용 포함하여 풀리퀘합니다.