diff --git a/.gitignore b/.gitignore index 348ce05..44b6bf8 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,11 @@ +__pycache__ +__pruning +__experiment +__pycache__/ +*.pyc +.DS_Store +outputs/ +tmp/ # Byte-compiled / optimized / DLL files __pycache__/ *.py[codz] @@ -227,4 +235,4 @@ __experiment outputs/ *.safetensors *.jinja -*.log \ No newline at end of file +*.log diff --git a/llm-streamline/README.md b/llm-streamline/README.md index 45b983d..199a72b 100644 --- a/llm-streamline/README.md +++ b/llm-streamline/README.md @@ -5,18 +5,28 @@ ICLR 2025 에서 Spotlight 되었던 Pruning 기법: **LLM Streamline** 을 적 해당 기법에 대한 자세한 설명은 논문 ([Link](https://arxiv.org/abs/2403.19135)) 혹은 Pseudolab 세미나 발표자료 ([Link](https://github.com/khyeongkyun/plab12-llm-on-jetson/blob/main/hkim-LLM-STREAMLINE-ICLR25-spotlight.pdf)) 를 참고하시기 바랍니다. --- -✅: Worked / ⬜: Not-worked +🔔 UPDATE 🔔 + +**2026-07-19:** Pruning 된 OPT 모델의 Benchmark 결과 -**Model & Replacement** +**2026-06-24:** OPT/Llama 테스트 코드 업로드 + +## Model & Replacement: Implementation | LLM / Replacement | None | FFN(MLP) | Transformer | |---|---|---|---| -| LLama-2 ([Link](https://huggingface.co/meta-llama/models?search=llama2)] | ⬜ | ✅ | ✅ | -| LLama-3 ([Link](https://huggingface.co/meta-llama/models?search=llama3)] | ⬜ | ✅ | ✅ | -| OPT ([Link](https://huggingface.co/facebook/models?search=opt)) | ⬜ | ✅ | ✅ | -| GPT-OSS ([Link](https://huggingface.co/openai/models?search=gpt-oss)) | ⬜ | ⬜ | ⬜ | +| OPT ([Link](https://huggingface.co/facebook/models?search=opt)) | ✅ | ✅ | ✅ | +| LLama-3 ([Link](https://huggingface.co/meta-llama/models?search=llama3)] | ✅ | ✅ | ✅ | +| LLama-2 ([Link](https://huggingface.co/meta-llama/models?search=llama2)] | ✅ | ✅ | ✅ | +| GPT-OSS ([Link](https://huggingface.co/openai/models?search=gpt-oss)): `MoE` | ❌ | ❌ | ❌ | + +* 각 모델은 Pruning 된 기존 Layer 의 Input/Output Token에 대한 MSE Loss를 사용하여 Replacement Layer (MLP/TF)의 재학습을 진행하였습니다. +* LLama 모델은 +* LLama-2는 Full MHA을 사용하기 때문에 GQA를 사용하는 LLama-3에 비해 trainin/inference 과정에서 out-of-memory 현상이 발생할 수 있습니다. +* LLM-Streamline 기법은 MOE 기반의 LLM 모델 (e.g., GTP-OSS)에 적용되지 않습니다. -**Dataset - Pruning/Retraining** + +## Dataset for Pruning/Retraining - [EN] SlimPajama-6B ([Link](https://huggingface.co/datasets/DKYoon/SlimPajama-6B)): ✅ @@ -24,10 +34,15 @@ ICLR 2025 에서 Spotlight 되었던 Pruning 기법: **LLM Streamline** 을 적 - [KR] kowikitext([Link](https://huggingface.co/datasets/heegyu/kowikitext)): ⬜ ---- -🔔 UPDATE 🔔 +## Deployment Test + +⬜ TensorRT-LLM Library 사용, Quantization 진행 (e.g., SmoothQuant-INT8 or AWQ-INT4) + +- Device: NVIDIA Jetson Orin Nano + +- Runtime: TensorRT-LLM Runtime + -**2026-00-00:** TBDs # Environment Setup @@ -38,14 +53,14 @@ conda activate plab-llm-streamline pip install -r requirements.txt ``` -**System Resource** +아래 Resource를 사용하여 실험이 진행되었습니다. - CPU: Intel Xeon Gold 6336Y 24C 185W 2.4GHz - GPU: NVIDIA HGX A100 80GB 500W # Workflow -1. Pruning Layer Searching : Layer Group 별 In/Out hidden state의 Cosine similarity score 기반 +1. Pruning Layer Searching : Layer Group 별 In/Out hidden state의 Cosine similarity score 기반 - MSE Loss ``` python search_pruning_layer.py \ @@ -68,31 +83,49 @@ pip install -r requirements.txt --pruning_start_layer 2 \ --pruning_end_layer 9 \ --output_dir /path/to/directory/ + --patience 5 \ ``` -# Evaluation - -LLM : [KMMLU Benchmark](https://huggingface.co/datasets/HAERAE-HUB/KMMLU) + - Best model만 HuggingFace Checkpoint 형태로 저장 + - 5회 연속으로 Best model이 저장되지 않은 경우 Early stopping + - 단, `--replace mlp`의 경우 대체 레이어가 표준 decoder layer 형태가 아니므로 `modeling_pruned_.py`가 함께 저장 (`trust_remote_code=True`) + - 추가로, `opt_prune_from2to9_mlp_eval_log.csv` 파일을 통해 (`global_step`, `eval_loss`, `saved`) 정보 확인가능 +3. Model benchmark : Pruning 된 모델을 KMMLU Benchmark를 활용하여 평가 -**Replacement: None** -| Model | STEM | Applied Science | HUMSS | Other | **Average** | -|:---:|:---:|:---:|:---:|:---:|:---:| -| LLama-2-7b: 8 layer | - (-%) | - (-%) | - (-%) | - (-%) | **- (-%)** | -| LLama-3-8b: 8 layer | - (-%) | - (-%) | - (-%) | - (-%) | **- (-%)** | -| OPT-6.7b: 8 layer | - (-%) | - (-%) | - (-%) | - (-%) | **- (-%)** | + ``` + python benchmark.py \ + --num_fewshot 1 \ + --model_name "/path/to/directory/opt_prune_from2to9_mlp" \ + --model_label "opt_prune_from2to9_mlp" \ + --replace_type "mlp" \ + --output_dir "/path/to/directory/benchmark_results" + ``` -**Replacement: FFN** -| Model | STEM | Applied Science | HUMSS | Other | **Average** | -|:---:|:---:|:---:|:---:|:---:|:---:| -| LLama-2-7b: 8 layer | - (-%) | - (-%) | - (-%) | - (-%) | **- (-%)** | -| LLama-3-8b: 8 layer | - (-%) | - (-%) | - (-%) | - (-%) | **- (-%)** | -| OPT-6.7b: 8 layer | - (-%) | - (-%) | - (-%) | - (-%) | **- (-%)** | + *OPT 모델의 한국어에 대한 Tokenizer Overflow 이슈로 인해 1-shot setting으로 바꾸어 평가 진행. -**Replacement: Transformer** -| Model | STEM | Applied Science | HUMSS | Other | **Average** | -|:---:|:---:|:---:|:---:|:---:|:---:| -| LLama-2-7b: 8 layer | - (-%) | - (-%) | - (-%) | - (-%) | **- (-%)** | -| LLama-3-8b: 8 layer | - (-%) | - (-%) | - (-%) | - (-%) | **- (-%)** | -| OPT-6.7b: 8 layer | - (-%) | - (-%) | - (-%) | - (-%) | **- (-%)** | +# Evaluation +**Benchmark dataset:** [KMMLU Benchmark](https://huggingface.co/datasets/HAERAE-HUB/KMMLU) + + +| Model (Layers) | NParams | Size(GB) | VRAM(GB) | STEM | App.Sci | HUMSS | Other | Avg | +|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| +| **OPT-6.7b (32):**
**1-shot**| | | | | | | | | +| Dense (-) | 6.66B | 12.4 | 18.71 | 30.11 | 30.13 | 24.16 | 25.69 | 27.58 +| 🔁 None (24) | 5.05B
($\color{red}{\blacktriangledown}$ 24.20%) | 9.4
($\color{red}{\blacktriangledown}$ 24.19%) | 14.72
($\color{red}{\blacktriangledown}$ 21.33%) | 20.76
($\color{red}{\blacktriangledown}$ 31.05%) | 18.38
($\color{red}{\blacktriangledown}$ 39.00%) | 23.19
($\color{red}{\blacktriangledown}$ 4.01%) | 14.25
($\color{red}{\blacktriangledown}$ 44.53%) | 19.13
($\color{red}{\blacktriangledown}$ 30.64%) +| 🔁 MLP (24+@) | 5.18B
($\color{red}{\blacktriangledown}$ 22.18%) | 9.65
($\color{red}{\blacktriangledown}$ 22.18%) | 14.97
($\color{red}{\blacktriangledown}$ 19.99%) | 28.37
($\color{red}{\blacktriangledown}$ 5.78%) | 24.64
($\color{red}{\blacktriangledown}$ 18.22%) | 25.34
($\color{green}{\blacktriangle}$ 4.88%) | 23.84
($\color{red}{\blacktriangledown}$ 7.20%) | 25.53
($\color{red}{\blacktriangledown}$ 7.43%) +| 🔁 TF (24+1) | 5.25B
($\color{red}{\blacktriangledown}$ 21.17%) | 9.78
($\color{red}{\blacktriangledown}$ 21.13%) | 15.22
($\color{red}{\blacktriangledown}$ 18.65%) | 29.50
($\color{red}{\blacktriangledown}$ 2.03%) | 27.33
($\color{red}{\blacktriangledown}$ 9.29%) | 25.00
($\color{green}{\blacktriangle}$ 3.48%) | 23.72
($\color{red}{\blacktriangledown}$ 7.67%) | 26.41
($\color{red}{\blacktriangledown}$ 4.24%) +|
||||||||| +| **Llama-3.1-8B (32):**
**5-shot**| | | | | | | | +| Dense (-) | 8.03B | 14.96 | 22.26 | 42.81 | 38.63 | 41.09 | 41.23 | 40.89 +| 🔁 TF | 5.41B
($\color{red}{\blacktriangledown}$ 32.59%) | 10.08
($\color{red}{\blacktriangledown}$ 32.62%) | 17.38
($\color{red}{\blacktriangledown}$ 21.92%) | 44.86
($\color{green}{\blacktriangle}$ 4.79%) | 41.65
($\color{green}{\blacktriangle}$ 7.82%) | 39.01
($\color{red}{\blacktriangledown}$ 5.06%) | 44.21
($\color{green}{\blacktriangle}$ 7.23%) | 42.41
($\color{green}{\blacktriangle}$ 3.72%) +|
||||||||| +| **Llama-2-7b-hf (32):**
**5-shot**| | | | | | | | | +| Dense (-) | 6.74B | 12.55 | 23.61 | 18.79 | 16.67 | 25.50 | 18.47 | 19.79 +| 🔁 TF | 4.71B
($\color{red}{\blacktriangledown}$ 30.03%) | 8.78
($\color{red}{\blacktriangledown}$ 30.04%) | 17.34
($\color{red}{\blacktriangledown}$ 26.56%) | 25.29
($\color{green}{\blacktriangle}$ 34.59%) | 23.37
($\color{green}{\blacktriangle}$ 40.19%) | 24.48
($\color{red}{\blacktriangledown}$ 4.00%) | 23.05
($\color{green}{\blacktriangle}$ 24.80%) | 24.03
($\color{green}{\blacktriangle}$ 21.42%) +|
||||||||| + + +* Llama-2/3 모델의 Benchmark 결과는 저자가 공유한 HuggingFace 모델 ([Llama-2](https://huggingface.co/XiaodongChen/Llama-2-4.7B) / [Llama-3](https://huggingface.co/XiaodongChen/Llama-3.1-5.4B))을 재사용하여 진행하였습니다. 해당 모델은 MSE loss가 아닌, LLM loss (다음 토큰 예측 loss)가 사용되었습니다. +* VRAM은 Benchmark dataset의 각 Subject 별 5개의 샘플을 사용하여 Peak Allocated 값을 `torch.cuda.max_memory_allocated` 함수를 활용하여 측정하였습니다. \ No newline at end of file diff --git a/llm-streamline/args.py b/llm-streamline/args.py index 28adbd1..992a933 100755 --- a/llm-streamline/args.py +++ b/llm-streamline/args.py @@ -89,6 +89,60 @@ class TrainingArguments: metadata={"help": "Evaluate every N gradient steps."}, ) + patience: Optional[int] = field( + default=5, + metadata={"help": "Stop retraining after this many consecutive evals with no new best eval loss. Set 0 to disable."}, + ) + lr: float = field(default=2e-4) min_lr: float = field(default=5e-5) weight_decay: float = field(default=1e-3) + + +@dataclass +class BenchmarkArguments: + """ + Arguments for benchmark.py — 1-shot log-likelihood evaluation on KMMLU + (https://huggingface.co/datasets/HAERAE-HUB/KMMLU), reused by all model/replace + combinations. `ModelArguments.model_name` doubles as the checkpoint path here, + same as in search_pruning_layer.py / replace_and_retrain.py. + + num_fewshot defaults to 1, not the paper's 5: OPT's English-trained BPE + tokenizes Korean far more densely than English, so measured against + facebook/opt-6.7b's own tokenizer, a 5-shot prompt overflows its 2048-token + max_position_embeddings on ~76% of KMMLU questions (silently discarding + most/all of the fewshot exemplars for those), vs. ~6% at 1-shot. + """ + + output_dir: str = field( + default="./benchmark_results", + metadata={"help": "Directory for kmmlu_summary.csv and _per_subject.csv."}, + ) + + model_label: Optional[str] = field( + default=None, + metadata={"help": "Value written to the CSV 'Model' column. Defaults to model_name's basename."}, + ) + + replace_type: Optional[str] = field( + default=None, + metadata={"help": "Value written to the CSV 'Replacement' column (e.g. none/mlp/tf); informational only."}, + ) + + num_fewshot: int = field( + default=1, + metadata={"help": "Few-shot exemplars drawn from each subject's KMMLU 'dev' split (up to 5 available per subject)."}, + ) + + max_examples: Optional[int] = field( + default=None, + metadata={"help": "Cap test examples scored per subject. Peak VRAM is set by one score_choices " + "batch (4 sequences, bounded by max_position_embeddings), not by dataset size, " + "so a small cap (e.g. 5-10) still yields a representative Peak VRAM reading " + "while running in a fraction of the time of a full accuracy pass."}, + ) + + dtype: Literal["bf16", "fp16", "fp32"] = field( + default="bf16", + metadata={"help": "torch_dtype used to load the model for inference."}, + ) diff --git a/llm-streamline/benchmark.py b/llm-streamline/benchmark.py new file mode 100644 index 0000000..c62899c --- /dev/null +++ b/llm-streamline/benchmark.py @@ -0,0 +1,287 @@ +""" +1-shot, log-likelihood evaluation of a pruned checkpoint on KMMLU +(https://huggingface.co/datasets/HAERAE-HUB/KMMLU): 45 subjects, each scored by +comparing the model's log-probability of continuing a "정답：" prompt with "A"/ +"B"/"C"/"D" and taking the argmax (same methodology as lm-evaluation-harness's +default "kmmlu" task group). Subjects are macro-averaged into the paper's four +reporting categories (STEM / Applied Science / HUMSS / Other) plus an overall +Average across all 45 subjects — see modeling/prune_utils.py-saved checkpoints, +loadable via AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True). + +num_fewshot defaults to 1, not the paper's 5: OPT's English-trained BPE +tokenizes Korean far more densely than English, so a 5-shot prompt overflows +facebook/opt-6.7b's 2048-token max_position_embeddings on ~76% of KMMLU +questions (silently discarding most/all fewshot exemplars for those), vs. ~6% +at 1-shot — see args.py's BenchmarkArguments for the measurement this is based +on. The remaining ~6% (mostly long scenario-style questions in subjects like +Criminal-Law/Patent/Accounting) still gets truncated via score_choices' left- +truncation below, regardless of num_fewshot. + +Per-subject results are appended incrementally to +/_kmmlu_per_subject.csv, so a killed/requeued slurm job +can resume without re-scoring already-finished subjects. The final row (one per +model) is appended to the shared /kmmlu_summary.csv. + +Usage: + python benchmark.py \ + --model_name /path/to/opt_prune_from2to9_mlp \ + --model_label "OPT-6.7b: 8 layer pruned" \ + --replace_type mlp \ + --output_dir ./benchmark_results +""" + +import csv +import os + +import torch +from datasets import load_dataset +from tqdm import tqdm +from transformers import AutoModelForCausalLM, AutoTokenizer, HfArgumentParser + +from args import BenchmarkArguments, ModelArguments + +KMMLU_CATEGORIES = { + "Accounting": "HUMSS", + "Agricultural-Sciences": "Other", + "Aviation-Engineering-and-Maintenance": "Applied Science", + "Biology": "STEM", + "Chemical-Engineering": "STEM", + "Chemistry": "STEM", + "Civil-Engineering": "STEM", + "Computer-Science": "STEM", + "Construction": "Other", + "Criminal-Law": "HUMSS", + "Ecology": "STEM", + "Economics": "HUMSS", + "Education": "HUMSS", + "Electrical-Engineering": "STEM", + "Electronics-Engineering": "Applied Science", + "Energy-Management": "Applied Science", + "Environmental-Science": "Applied Science", + "Fashion": "Other", + "Food-Processing": "Other", + "Gas-Technology-and-Engineering": "Applied Science", + "Geomatics": "Applied Science", + "Health": "Other", + "Industrial-Engineer": "Applied Science", + "Information-Technology": "STEM", + "Interior-Architecture-and-Design": "Other", + "Korean-History": "HUMSS", + "Law": "HUMSS", + "Machine-Design-and-Manufacturing": "Applied Science", + "Management": "HUMSS", + "Maritime-Engineering": "Applied Science", + "Marketing": "Other", + "Materials-Engineering": "STEM", + "Math": "STEM", + "Mechanical-Engineering": "STEM", + "Nondestructive-Testing": "Applied Science", + "Patent": "Other", + "Political-Science-and-Sociology": "HUMSS", + "Psychology": "HUMSS", + "Public-Safety": "Other", + "Railway-and-Automotive-Engineering": "Applied Science", + "Real-Estate": "Other", + "Refrigerating-Machinery": "Other", + "Social-Welfare": "HUMSS", + "Taxation": "HUMSS", + "Telecommunications-and-Wireless-Technology": "Applied Science", +} +CHOICES = ["A", "B", "C", "D"] +DTYPE_MAP = {"bf16": torch.bfloat16, "fp16": torch.float16, "fp32": torch.float32} + + +def format_question(row): + return ( + f"{row['question'].strip()}\n" + f"A. {row['A']}\nB. {row['B']}\nC. {row['C']}\nD. {row['D']}\n정답：" + ) + + +def build_fewshot_prefix(dev_rows, num_fewshot): + examples = [] + for row in dev_rows.select(range(min(num_fewshot, len(dev_rows)))): + examples.append(format_question(row) + CHOICES[row["answer"] - 1]) + return ("\n\n".join(examples) + "\n\n") if examples else "" + + +@torch.no_grad() +def score_choices(model, tokenizer, context, device, max_position_embeddings): + """ + Returns (scores, truncated): scores is a length-4 list of log-probability + sums, one per candidate in CHOICES, each the model's log-likelihood of that + choice as a continuation of `context`; truncated flags whether context had + to be shortened to fit the model's position-embedding table. + + OPT's tokenizer is an English-trained byte-level BPE with no Korean-aware + merges, so Korean text tokenizes far more densely than English does on it — + a 5-shot Korean prompt can overflow max_position_embeddings (2048 for + OPT-6.7b) even though it looks short. Overflowing it feeds an out-of-range + index into the learned position embedding and crashes with a CUDA + device-side assert, so context is truncated from the left (dropping the + earliest few-shot exemplars first) to leave room for the longest choice. + """ + context_ids = tokenizer(context)["input_ids"] + choice_ids = [tokenizer(c, add_special_tokens=False)["input_ids"] for c in CHOICES] + reserve = max(len(ids) for ids in choice_ids) + + truncated = len(context_ids) > max_position_embeddings - reserve + if truncated: + context_ids = context_ids[-(max_position_embeddings - reserve):] + + sequences = [context_ids + ids for ids in choice_ids] + cont_lens = [len(ids) for ids in choice_ids] + + max_len = max(len(s) for s in sequences) + pad_id = tokenizer.pad_token_id + input_ids = torch.full((len(CHOICES), max_len), pad_id, dtype=torch.long) + attention_mask = torch.zeros((len(CHOICES), max_len), dtype=torch.long) + for i, s in enumerate(sequences): + input_ids[i, max_len - len(s):] = torch.tensor(s, dtype=torch.long) + attention_mask[i, max_len - len(s):] = 1 + + logits = model( + input_ids=input_ids.to(device), attention_mask=attention_mask.to(device) + ).logits + logprobs = torch.log_softmax(logits.float(), dim=-1) + + scores = [] + for i, (s, cont_len) in enumerate(zip(sequences, cont_lens)): + pad_offset = max_len - len(s) + cont_start = pad_offset + (len(s) - cont_len) + score = sum( + logprobs[i, pos - 1, input_ids[i, pos]].item() + for pos in range(cont_start, pad_offset + len(s)) + ) + scores.append(score) + return scores, truncated + + +def load_finished_subjects(per_subject_csv): + finished = {} + if os.path.exists(per_subject_csv): + with open(per_subject_csv, newline="") as f: + for row in csv.DictReader(f): + finished[row["subject"]] = { + "accuracy": float(row["accuracy"]), + "num_examples": int(row["num_examples"]), + } + return finished + + +if __name__ == "__main__": + parser = HfArgumentParser((ModelArguments, BenchmarkArguments)) + model_args, args, _ = parser.parse_args_into_dataclasses(return_remaining_strings=True) + + os.makedirs(args.output_dir, exist_ok=True) + model_label = args.model_label or os.path.basename(model_args.model_name.rstrip("/")) + per_subject_csv = os.path.join(args.output_dir, f"{model_label}_kmmlu_per_subject.csv") + summary_csv = os.path.join(args.output_dir, "kmmlu_summary.csv") + + device = "cuda" if torch.cuda.is_available() else "cpu" + + print(f"Loading {model_args.model_name} ...") + tokenizer = AutoTokenizer.from_pretrained( + model_args.model_name, use_fast=model_args.use_fast, trust_remote_code=True + ) + if tokenizer.pad_token is None: + tokenizer.pad_token = tokenizer.eos_token + + model = AutoModelForCausalLM.from_pretrained( + model_args.model_name, torch_dtype=DTYPE_MAP[args.dtype], trust_remote_code=True + ).to(device) + model.eval() + max_position_embeddings = getattr(model.config, "max_position_embeddings", 2048) + + # Peak VRAM is set by weight loading plus one score_choices batch (4 sequences, + # bounded by max_position_embeddings) — it doesn't grow with dataset size, so this + # reading stays representative even under --max_examples. + if device == "cuda": + torch.cuda.reset_peak_memory_stats(device) + + num_params = sum(p.numel() for p in model.parameters()) + bytes_per_param = torch.finfo(DTYPE_MAP[args.dtype]).bits // 8 + model_size_gb = num_params * bytes_per_param / (1024 ** 3) + print(f"Model parameters: {num_params:,} ({num_params / 1e9:.2f}B) ~{model_size_gb:.2f} GB in {args.dtype}") + + results = load_finished_subjects(per_subject_csv) + write_header = not os.path.exists(per_subject_csv) + with open(per_subject_csv, "a", newline="") as f: + writer = csv.writer(f) + if write_header: + writer.writerow(["subject", "category", "num_examples", "accuracy"]) + + for subject in KMMLU_CATEGORIES: + if subject in results: + print(f"[{subject}] already scored — accuracy {results[subject]['accuracy']:.4f} (skipped)") + continue + + dataset = load_dataset("HAERAE-HUB/KMMLU", subject) + fewshot_prefix = build_fewshot_prefix(dataset["dev"], args.num_fewshot) + test_rows = dataset["test"] + if args.max_examples is not None: + test_rows = test_rows.select(range(min(args.max_examples, len(test_rows)))) + + correct = 0 + truncated_count = 0 + for row in tqdm(test_rows, desc=subject): + context = fewshot_prefix + format_question(row) + scores, truncated = score_choices( + model, tokenizer, context, device, max_position_embeddings + ) + pred = max(range(len(CHOICES)), key=lambda i: scores[i]) + correct += int(pred == row["answer"] - 1) + truncated_count += int(truncated) + + accuracy = correct / len(test_rows) + results[subject] = {"accuracy": accuracy, "num_examples": len(test_rows)} + writer.writerow([subject, KMMLU_CATEGORIES[subject], len(test_rows), accuracy]) + f.flush() + print( + f"[{subject}] accuracy: {accuracy:.4f} ({correct}/{len(test_rows)})" + + (f" [truncated context: {truncated_count}/{len(test_rows)}]" if truncated_count else "") + ) + + # ───────────────────────────────────────────────────────────────────────── + # Macro-average per category + overall macro-average across all 45 subjects + # ───────────────────────────────────────────────────────────────────────── + per_category = {"STEM": [], "Applied Science": [], "HUMSS": [], "Other": []} + for subject, info in results.items(): + per_category[KMMLU_CATEGORIES[subject]].append(info["accuracy"]) + + category_avg = {cat: sum(scores) / len(scores) for cat, scores in per_category.items()} + overall_avg = sum(info["accuracy"] for info in results.values()) / len(results) + + peak_vram_allocated_gb = torch.cuda.max_memory_allocated(device) / (1024 ** 3) if device == "cuda" else 0.0 + peak_vram_reserved_gb = torch.cuda.max_memory_reserved(device) / (1024 ** 3) if device == "cuda" else 0.0 + + write_header = not os.path.exists(summary_csv) + with open(summary_csv, "a", newline="") as f: + writer = csv.writer(f) + if write_header: + writer.writerow( + ["Model", "Replacement", "Params", "Size (GB)", + "STEM", "Applied Science", "HUMSS", "Other", "Average", + "Peak VRAM Allocated (GB)", "Peak VRAM Reserved (GB)"] + ) + writer.writerow( + [ + model_label, + args.replace_type or "", + num_params, + f"{model_size_gb:.2f}", + f"{category_avg['STEM']:.4f}", + f"{category_avg['Applied Science']:.4f}", + f"{category_avg['HUMSS']:.4f}", + f"{category_avg['Other']:.4f}", + f"{overall_avg:.4f}", + f"{peak_vram_allocated_gb:.2f}", + f"{peak_vram_reserved_gb:.2f}", + ] + ) + + print(f"\nParams: {num_params:,} ({num_params / 1e9:.2f}B) Size: {model_size_gb:.2f} GB\n" + f"STEM: {category_avg['STEM']:.4f} Applied Science: {category_avg['Applied Science']:.4f} " + f"HUMSS: {category_avg['HUMSS']:.4f} Other: {category_avg['Other']:.4f} Average: {overall_avg:.4f}\n" + f"Peak VRAM: {peak_vram_allocated_gb:.2f} GB allocated / {peak_vram_reserved_gb:.2f} GB reserved") + print(f"Summary appended to {summary_csv}") diff --git a/llm-streamline/modeling/prune_utils.py b/llm-streamline/modeling/prune_utils.py new file mode 100644 index 0000000..b74ae62 --- /dev/null +++ b/llm-streamline/modeling/prune_utils.py @@ -0,0 +1,245 @@ +""" +Assembles and saves the full, ready-to-use pruned model (as opposed to just +the trained replace_layer) for all three --replace modes (none / mlp / tf). + +For --replace none/tf, the result is a fully standard OPT/Llama checkpoint — +loadable with plain `AutoModelForCausalLM.from_pretrained(dir)`. + +For --replace mlp, the replacement is a bare 2-layer MLP with no +attention/layernorm, which doesn't fit the standard OPTDecoderLayer / +LlamaDecoderLayer shape. To keep the saved checkpoint genuinely load-able by +anyone (e.g. after pushing to the Hub), a small self-contained modeling file +is written alongside it and wired up via config.json's `auto_map`, so +`AutoModelForCausalLM.from_pretrained(dir, trust_remote_code=True)` works +out of the box. +""" + +import os +import re +import shutil +import time + +import torch.nn as nn +from safetensors import SafetensorError +from transformers import AutoConfig, AutoModelForCausalLM + + +class MLP(nn.Module): + def __init__(self, hidden_size): + super().__init__() + self.fc1 = nn.Linear(hidden_size, 4 * hidden_size) + self.fc2 = nn.Linear(4 * hidden_size, hidden_size) + self.activation = nn.ReLU() + + def forward(self, x): + return self.fc2(self.activation(self.fc1(x))) + + +class MLPReplaceLayer(nn.Module): + """ + Drop-in replacement for a standard decoder layer in a stock HF layer + stack. Trained as a bare MLP(hidden_states) with no residual/attention/ + layernorm, so forward must reproduce that exactly — the decoder-layer + kwargs (attention_mask, position_ids, past_key_value, ...) are ignored. + Still returns the (hidden_states, [attn_weights], [present_key_value]) + tuple shape the parent decoder loop expects, so output_attentions/ + use_cache toggles used during .generate() don't break indexing. + """ + + def __init__(self, hidden_size): + super().__init__() + self.mlp = MLP(hidden_size) + + def forward(self, hidden_states, *args, output_attentions=False, use_cache=False, **kwargs): + outputs = (self.mlp(hidden_states),) + if output_attentions: + outputs += (None,) + if use_cache: + outputs += (None,) + return outputs + + +_FAMILY_INFO = { + "llama": { + "layer_re": re.compile(r"^model\.layers\.(\d+)\.(.*)"), + "layer_fmt": "model.layers.{j}.{rest}", + "hf_module": "llama", + "base_class": "LlamaForCausalLM", + "layers_path": "self.model.layers", + }, + "opt": { + "layer_re": re.compile(r"^model\.decoder\.layers\.(\d+)\.(.*)"), + "layer_fmt": "model.decoder.layers.{j}.{rest}", + "hf_module": "opt", + "base_class": "OPTForCausalLM", + "layers_path": "self.model.decoder.layers", + }, +} + + +def _get_layers(model, model_family): + return model.model.layers if model_family == "llama" else model.model.decoder.layers + + +def assemble_pruned_model(pretrained_dict, model_name, model_family, replace, pruning_start_layer, pruning_end_layer, replace_layer_state_dict=None): + """ + Build the full-size deployable pruned model: the original model with + layers [pruning_start_layer, pruning_end_layer] collapsed into either + nothing ("none") or a single replacement layer ("mlp"/"tf"); every other + layer is copied unchanged and re-indexed. + + `pretrained_dict` should be `AutoModelForCausalLM.from_pretrained(model_name).state_dict()`, + loaded once by the caller and reused across calls (this function does not + re-download/re-load the pretrained checkpoint). + + Returns (model, replace_layer_index_or_None). + """ + info = _FAMILY_INFO[model_family] + n_pruned = pruning_end_layer - pruning_start_layer + 1 + insert_replacement = replace != "none" + + config = AutoConfig.from_pretrained(model_name) + config.num_hidden_layers = config.num_hidden_layers - n_pruned + (1 if insert_replacement else 0) + + pruned_model = AutoModelForCausalLM.from_config(config) + + replace_layer_index = pruning_start_layer if insert_replacement else None + + if replace == "mlp": + _get_layers(pruned_model, model_family)[replace_layer_index] = MLPReplaceLayer(config.hidden_size) + + pruned_dict = pruned_model.state_dict() + layer_re, layer_fmt = info["layer_re"], info["layer_fmt"] + + # Layers removed outright vs. collapsed into one new layer shift tail + # layers back by a different amount. + shift = n_pruned - (1 if insert_replacement else 0) + for key, value in pretrained_dict.items(): + m = layer_re.match(key) + if m: + i, rest = int(m.group(1)), m.group(2) + if pruning_start_layer <= i <= pruning_end_layer: + continue # part of the pruned block — handled separately below + j = i if i < pruning_start_layer else i - shift + new_key = layer_fmt.format(j=j, rest=rest) + else: + new_key = key + if new_key in pruned_dict: + pruned_dict[new_key] = value + + if replace == "tf" and replace_layer_state_dict is not None: + prefix = layer_fmt.format(j=replace_layer_index, rest="") + for k, v in replace_layer_state_dict.items(): + pruned_dict[prefix + k] = v + + if replace == "mlp" and replace_layer_state_dict is not None: + prefix = layer_fmt.format(j=replace_layer_index, rest="mlp.") + for k, v in replace_layer_state_dict.items(): + pruned_dict[prefix + k] = v + + pruned_model.load_state_dict(pruned_dict) + return pruned_model, replace_layer_index + + +_REMOTE_CODE_TEMPLATE = '''\ +""" +Auto-generated by replace_and_retrain.py — do not hand-edit. + +Loads this checkpoint with layer {layer_index} replaced by a lightweight +2-layer MLP (trained to approximate the block of layers this model +originally pruned). Requires trust_remote_code=True. +""" +import torch.nn as nn +from transformers.models.{hf_module}.modeling_{hf_module} import {base_class} + + +class MLP(nn.Module): + def __init__(self, hidden_size): + super().__init__() + self.fc1 = nn.Linear(hidden_size, 4 * hidden_size) + self.fc2 = nn.Linear(4 * hidden_size, hidden_size) + self.activation = nn.ReLU() + + def forward(self, x): + return self.fc2(self.activation(self.fc1(x))) + + +class MLPReplaceLayer(nn.Module): + def __init__(self, hidden_size): + super().__init__() + self.mlp = MLP(hidden_size) + + def forward(self, hidden_states, *args, output_attentions=False, use_cache=False, **kwargs): + outputs = (self.mlp(hidden_states),) + if output_attentions: + outputs += (None,) + if use_cache: + outputs += (None,) + return outputs + + +class {model_class}({base_class}): + def __init__(self, config): + super().__init__(config) + {layers_path}[{layer_index}] = MLPReplaceLayer(config.hidden_size) +''' + + +def _write_remote_code(output_dir, model_family, replace_layer_index): + info = _FAMILY_INFO[model_family] + model_class = f"Pruned{info['base_class']}" + module_name = f"modeling_pruned_{model_family}" + source = _REMOTE_CODE_TEMPLATE.format( + layer_index=replace_layer_index, + hf_module=info["hf_module"], + base_class=info["base_class"], + layers_path=info["layers_path"], + model_class=model_class, + ) + with open(os.path.join(output_dir, f"{module_name}.py"), "w") as f: + f.write(source) + return f"{module_name}.{model_class}" + + +def save_pruned_model(model, tokenizer, output_dir, model_family, replace, replace_layer_index): + """ + Saves `model` as a ready-to-use HF checkpoint at `output_dir`. For + --replace mlp, also ships the custom layer's modeling code and wires up + config.json's auto_map so AutoModelForCausalLM.from_pretrained(..., + trust_remote_code=True) works for anyone who downloads it — plain + from_pretrained (no trust_remote_code) is NOT enough for this mode, + since the MLP replacement layer isn't a stock decoder layer. + """ + max_attempts = 3 + for attempt in range(1, max_attempts + 1): + shutil.rmtree(output_dir, ignore_errors=True) + os.makedirs(output_dir, exist_ok=True) + + if replace == "mlp": + target = _write_remote_code(output_dir, model_family, replace_layer_index) + model.config.auto_map = {"AutoModelForCausalLM": target} + + try: + model.save_pretrained(output_dir) + break + except SafetensorError: + # Transient I/O errors (e.g. ENOENT from a network filesystem + # hiccup mid-write) have been observed here; retry from a freshly + # recreated output_dir rather than losing hours of training. + if attempt == max_attempts: + raise + time.sleep(5) + + if tokenizer is not None: + tokenizer.save_pretrained(output_dir) + + +def replace_best_checkpoint(tmp_dir, final_dir): + """ + Atomically (best-effort) swap `tmp_dir` in as the new `final_dir`, + removing whatever previous best checkpoint was there. Used to keep only + the single best-eval-loss checkpoint on disk. + """ + if os.path.exists(final_dir): + shutil.rmtree(final_dir) + os.rename(tmp_dir, final_dir) diff --git a/llm-streamline/replace_and_retrain.py b/llm-streamline/replace_and_retrain.py index 05e8d72..7c80fd2 100644 --- a/llm-streamline/replace_and_retrain.py +++ b/llm-streamline/replace_and_retrain.py @@ -13,9 +13,19 @@ from which `modeling_*` module they imported) by picking the right modeling class from ./modeling/ at runtime based on --model/--replace. ---replace none skips training entirely and produces a pruned model by directly -wiring layer (pruning_start_layer - 1) to layer (pruning_end_layer + 1), then -saving the result with save_pretrained. +Regardless of --replace, the output is a ready-to-use, full-size HF checkpoint +(save_pretrained + tokenizer) at args.output_dir/_prune_fromto_/, +loadable directly with AutoModelForCausalLM.from_pretrained — for --replace mlp +specifically, the replacement layer is a bare MLP that doesn't fit the stock +decoder-layer shape, so a small modeling_pruned_.py + config.json +auto_map are shipped alongside it, requiring trust_remote_code=True to load. + +--replace none skips training entirely and produces that model immediately by +directly wiring layer (pruning_start_layer - 1) to layer (pruning_end_layer + 1). +For --replace mlp/tf, only the single best-eval-loss checkpoint is kept on +disk, overwritten in place on every improvement; every eval (not just +improvements) is additionally logged to _eval_log.csv (global_step, +eval_loss, saved) for later analysis. Usage: python retrain_pruned_layer.py --model --replace \ @@ -23,9 +33,9 @@ class from ./modeling/ at runtime based on --model/--replace. [--model_name ] [--data_path ] """ +import csv import importlib import os -import re from itertools import chain import torch @@ -44,6 +54,7 @@ class from ./modeling/ at runtime based on --model/--replace. from args import ModelArguments, TrainingArguments from LLM_Streamline.scheduler import get_cosine_schedule_with_warmup +from modeling.prune_utils import assemble_pruned_model, replace_best_checkpoint, save_pruned_model if __name__ == "__main__": @@ -101,50 +112,30 @@ class from ./modeling/ at runtime based on --model/--replace. if args.replace == "none": # ───────────────────────────────────────────────────────────────────── - # "none" replace: pure layer pruning, no training + # "none" replace: pure layer pruning, no training — one ready-to-use + # model, saved immediately. # ───────────────────────────────────────────────────────────────────── - n_pruned = args.pruning_end_layer - args.pruning_start_layer + 1 - print(f"Loading pretrained weights from {MODEL_NAME} ...") - pretrained = AutoModelForCausalLM.from_pretrained(MODEL_NAME) - pretrained_dict = pretrained.state_dict() - del pretrained - torch.cuda.empty_cache() + pretrained_dict = AutoModelForCausalLM.from_pretrained(MODEL_NAME).state_dict() + + pruned_model, _ = assemble_pruned_model( + pretrained_dict=pretrained_dict, + model_name=MODEL_NAME, + model_family=args.model, + replace=args.replace, + pruning_start_layer=args.pruning_start_layer, + pruning_end_layer=args.pruning_end_layer, + ) - config = AutoConfig.from_pretrained(MODEL_NAME) - original_n_layers = config.num_hidden_layers - config.num_hidden_layers -= n_pruned - - pruned_model = AutoModelForCausalLM.from_config(config) - pruned_dict = pruned_model.state_dict() - - if args.model == "llama": - layer_re = re.compile(r"^model\.layers\.(\d+)\.(.*)") - layer_fmt = "model.layers.{j}.{rest}" - else: # opt - layer_re = re.compile(r"^model\.decoder\.layers\.(\d+)\.(.*)") - layer_fmt = "model.decoder.layers.{j}.{rest}" - - for key, value in pretrained_dict.items(): - m = layer_re.match(key) - if m: - i, rest = int(m.group(1)), m.group(2) - if args.pruning_start_layer <= i <= args.pruning_end_layer: - continue # drop pruned layers - j = i if i < args.pruning_start_layer else i - n_pruned - new_key = layer_fmt.format(j=j, rest=rest) - else: - new_key = key - if new_key in pruned_dict: - pruned_dict[new_key] = value - - save_path = os.path.join(args.output_dir, f"{args.model}_prune_from{args.pruning_start_layer}to{args.pruning_end_layer}_{args.replace}.pt") - pruned_model.load_state_dict(pruned_dict) - torch.save(pruned_model.state_dict(), save_path) + tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME) + save_dir = os.path.join( + args.output_dir, + f"{args.model}_prune_from{args.pruning_start_layer}to{args.pruning_end_layer}_{args.replace}", + ) + save_pruned_model(pruned_model, tokenizer, save_dir, args.model, args.replace, replace_layer_index=None) print( - f"Pruned model saved to {save_path} " - f"(layers {args.pruning_start_layer}–{args.pruning_end_layer} removed, " - f"{original_n_layers} → {config.num_hidden_layers} layers)" + f"Pruned model saved to {save_dir} " + f"(layers {args.pruning_start_layer}–{args.pruning_end_layer} removed)" ) else: @@ -239,7 +230,7 @@ def group(examples): dataset, shuffle=True, collate_fn=data_collator, batch_size=args.batch_size ) eval_dataloader = DataLoader( - eval_dataset, shuffle=False, collate_fn=data_collator, batch_size=args.batch_size * 2 + eval_dataset, shuffle=False, collate_fn=data_collator, batch_size=args.batch_size * 1 ) # ───────────────────────────────────────────────────────────────────── @@ -266,11 +257,15 @@ def group(examples): mse_loss = nn.MSELoss() best_eval_loss = float("inf") global_step = 0 + no_improve_evals = 0 + should_stop = False # ───────────────────────────────────────────────────────────────────── # Training loop # ───────────────────────────────────────────────────────────────────── for epoch in range(args.epochs): + if should_stop: + break model.train() for step, batch in tqdm(enumerate(train_dataloader), desc=f"Epoch {epoch}"): with accelerator.accumulate(model): @@ -309,21 +304,64 @@ def group(examples): ) eval_loss = torch.cat(eval_losses).mean().item() + is_new_best = eval_loss < best_eval_loss + if is_new_best: + best_eval_loss = eval_loss + no_improve_evals = 0 + else: + no_improve_evals += 1 + if accelerator.is_main_process: print(f"Step {global_step} — eval MSE loss: {eval_loss:.6f}") - if eval_loss < best_eval_loss: - best_eval_loss = eval_loss - unwrapped = accelerator.unwrap_model(model) - replace_layer = ( - unwrapped.replace_layer if args.model == "llama" - else unwrapped.decoder.replace_layer + # R2b: log every eval (not just improvements) for later analysis. + csv_path = os.path.join( + args.output_dir, + f"{args.model}_prune_from{args.pruning_start_layer}to{args.pruning_end_layer}_{args.replace}_eval_log.csv", ) - torch.save( - replace_layer.state_dict(), - os.path.join( + write_header = not os.path.exists(csv_path) + with open(csv_path, "a", newline="") as f: + writer = csv.writer(f) + if write_header: + writer.writerow(["global_step", "eval_loss", "saved"]) + writer.writerow([global_step, eval_loss, int(is_new_best)]) + + # R2a: only the single best checkpoint is kept on disk — + # assembling and overwriting it on every improvement. + if is_new_best: + unwrapped = accelerator.unwrap_model(model) + replace_layer = ( + unwrapped.replace_layer if args.model == "llama" + else unwrapped.decoder.replace_layer + ) + pruned_model, replace_layer_index = assemble_pruned_model( + pretrained_dict=pretrained_dict, + model_name=MODEL_NAME, + model_family=args.model, + replace=args.replace, + pruning_start_layer=args.pruning_start_layer, + pruning_end_layer=args.pruning_end_layer, + replace_layer_state_dict=replace_layer.state_dict(), + ) + tmp_dir = os.path.join(args.output_dir, ".tmp_best_checkpoint") + save_pruned_model( + pruned_model, tokenizer, tmp_dir, args.model, args.replace, replace_layer_index + ) + final_dir = os.path.join( args.output_dir, - f"{args.model}_prune_from{args.pruning_start_layer}to{args.pruning_end_layer}_{args.replace}_step{global_step}.pt", - ), - ) + f"{args.model}_prune_from{args.pruning_start_layer}to{args.pruning_end_layer}_{args.replace}", + ) + replace_best_checkpoint(tmp_dir, final_dir) + print(f"New best (eval MSE {eval_loss:.6f}) saved to {final_dir}") + del pruned_model + + if args.patience and no_improve_evals >= args.patience: + if accelerator.is_main_process: + print( + f"No new best eval loss for {no_improve_evals} consecutive evals " + f"— stopping early at step {global_step}." + ) + should_stop = True + break + model.train() diff --git a/llm-streamline/slurm_output/benchmark-kmmlu_llama-2-7b.out b/llm-streamline/slurm_output/benchmark-kmmlu_llama-2-7b.out new file mode 100644 index 0000000..a167c5c --- /dev/null +++ b/llm-streamline/slurm_output/benchmark-kmmlu_llama-2-7b.out @@ -0,0 +1,104 @@ +=== Benchmarking llama_2_7b ( baseline) : meta-llama/Llama-2-7b-hf === +Loading meta-llama/Llama-2-7b-hf ... +Model parameters: 6,738,415,616 (6.74B) ~12.55 GB in bf16 +[Accounting] accuracy: 0.3200 (32/100) +[Agricultural-Sciences] accuracy: 0.1270 (127/1000) +[Aviation-Engineering-and-Maintenance] accuracy: 0.1590 (159/1000) +[Biology] accuracy: 0.2200 (220/1000) +[Chemical-Engineering] accuracy: 0.2560 (256/1000) +[Chemistry] accuracy: 0.2767 (166/600) +[Civil-Engineering] accuracy: 0.1520 (152/1000) +[Computer-Science] accuracy: 0.1760 (176/1000) +[Construction] accuracy: 0.1220 (122/1000) +[Criminal-Law] accuracy: 0.3000 (60/200) [truncated context: 131/200] +[Ecology] accuracy: 0.0990 (99/1000) +[Economics] accuracy: 0.2615 (34/130) +[Education] accuracy: 0.1900 (19/100) +[Electrical-Engineering] accuracy: 0.1720 (172/1000) +[Electronics-Engineering] accuracy: 0.3110 (311/1000) +[Energy-Management] accuracy: 0.2010 (201/1000) +[Environmental-Science] accuracy: 0.0930 (93/1000) +[Fashion] accuracy: 0.1730 (173/1000) +[Food-Processing] accuracy: 0.1500 (150/1000) +[Gas-Technology-and-Engineering] accuracy: 0.1090 (109/1000) +[Geomatics] accuracy: 0.1870 (187/1000) +[Health] accuracy: 0.2300 (23/100) +[Industrial-Engineer] accuracy: 0.0830 (83/1000) +[Information-Technology] accuracy: 0.1580 (158/1000) +[Interior-Architecture-and-Design] accuracy: 0.1950 (195/1000) +[Korean-History] accuracy: 0.2400 (24/100) +[Law] accuracy: 0.2390 (239/1000) +[Machine-Design-and-Manufacturing] accuracy: 0.1000 (100/1000) +[Management] accuracy: 0.2770 (277/1000) +[Maritime-Engineering] accuracy: 0.2183 (131/600) +[Marketing] accuracy: 0.2460 (246/1000) +[Materials-Engineering] accuracy: 0.1630 (163/1000) +[Math] accuracy: 0.2767 (83/300) +[Mechanical-Engineering] accuracy: 0.1180 (118/1000) +[Nondestructive-Testing] accuracy: 0.2010 (201/1000) +[Patent] accuracy: 0.2500 (25/100) [truncated context: 100/100] +[Political-Science-and-Sociology] accuracy: 0.1967 (59/300) +[Psychology] accuracy: 0.2630 (263/1000) +[Public-Safety] accuracy: 0.1350 (135/1000) +[Railway-and-Automotive-Engineering] accuracy: 0.1730 (173/1000) +[Real-Estate] accuracy: 0.2100 (42/200) +[Refrigerating-Machinery] accuracy: 0.1940 (194/1000) +[Social-Welfare] accuracy: 0.2680 (268/1000) +[Taxation] accuracy: 0.2500 (50/200) +[Telecommunications-and-Wireless-Technology] accuracy: 0.1650 (165/1000) + +Params: 6,738,415,616 (6.74B) Size: 12.55 GB +STEM: 0.1879 Applied Science: 0.1667 HUMSS: 0.2550 Other: 0.1847 Average: 0.1979 +Summary appended to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/benchmark_results/kmmlu_summary.csv +=== Benchmarking llama_2_prune_(none)_tf ( tf) : XiaodongChen/Llama-2-4.7B === +Loading XiaodongChen/Llama-2-4.7B ... +Model parameters: 4,714,582,016 (4.71B) ~8.78 GB in bf16 +[Accounting] accuracy: 0.2200 (22/100) +[Agricultural-Sciences] accuracy: 0.2230 (223/1000) +[Aviation-Engineering-and-Maintenance] accuracy: 0.2500 (250/1000) +[Biology] accuracy: 0.2300 (230/1000) +[Chemical-Engineering] accuracy: 0.2790 (279/1000) +[Chemistry] accuracy: 0.2700 (162/600) +[Civil-Engineering] accuracy: 0.3090 (309/1000) +[Computer-Science] accuracy: 0.2570 (257/1000) +[Construction] accuracy: 0.2840 (284/1000) +[Criminal-Law] accuracy: 0.2600 (52/200) [truncated context: 131/200] +[Ecology] accuracy: 0.1510 (151/1000) +[Economics] accuracy: 0.2692 (35/130) +[Education] accuracy: 0.1900 (19/100) +[Electrical-Engineering] accuracy: 0.2980 (298/1000) +[Electronics-Engineering] accuracy: 0.3200 (320/1000) +[Energy-Management] accuracy: 0.2100 (210/1000) +[Environmental-Science] accuracy: 0.2910 (291/1000) +[Fashion] accuracy: 0.2260 (226/1000) +[Food-Processing] accuracy: 0.1890 (189/1000) +[Gas-Technology-and-Engineering] accuracy: 0.2130 (213/1000) +[Geomatics] accuracy: 0.2240 (224/1000) +[Health] accuracy: 0.2400 (24/100) +[Industrial-Engineer] accuracy: 0.1340 (134/1000) +[Information-Technology] accuracy: 0.2210 (221/1000) +[Interior-Architecture-and-Design] accuracy: 0.2870 (287/1000) +[Korean-History] accuracy: 0.2700 (27/100) +[Law] accuracy: 0.2410 (241/1000) +[Machine-Design-and-Manufacturing] accuracy: 0.1920 (192/1000) +[Management] accuracy: 0.2570 (257/1000) +[Maritime-Engineering] accuracy: 0.2500 (150/600) +[Marketing] accuracy: 0.2420 (242/1000) +[Materials-Engineering] accuracy: 0.2600 (260/1000) +[Math] accuracy: 0.2700 (81/300) +[Mechanical-Engineering] accuracy: 0.2370 (237/1000) +[Nondestructive-Testing] accuracy: 0.2600 (260/1000) +[Patent] accuracy: 0.2200 (22/100) [truncated context: 100/100] +[Political-Science-and-Sociology] accuracy: 0.2367 (71/300) +[Psychology] accuracy: 0.2330 (233/1000) +[Public-Safety] accuracy: 0.1530 (153/1000) +[Railway-and-Automotive-Engineering] accuracy: 0.2560 (256/1000) +[Real-Estate] accuracy: 0.2200 (44/200) +[Refrigerating-Machinery] accuracy: 0.2510 (251/1000) +[Social-Welfare] accuracy: 0.2610 (261/1000) +[Taxation] accuracy: 0.2550 (51/200) +[Telecommunications-and-Wireless-Technology] accuracy: 0.2040 (204/1000) + +Params: 4,714,582,016 (4.71B) Size: 8.78 GB +STEM: 0.2529 Applied Science: 0.2337 HUMSS: 0.2448 Other: 0.2305 Average: 0.2403 +Summary appended to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/benchmark_results/kmmlu_summary.csv diff --git a/llm-streamline/slurm_output/benchmark-kmmlu_llama-3.1-8b.out b/llm-streamline/slurm_output/benchmark-kmmlu_llama-3.1-8b.out new file mode 100644 index 0000000..36e911f --- /dev/null +++ b/llm-streamline/slurm_output/benchmark-kmmlu_llama-3.1-8b.out @@ -0,0 +1,104 @@ +=== Benchmarking llama_3_8b ( baseline) : meta-llama/Llama-3.1-8B === +Loading meta-llama/Llama-3.1-8B ... +Model parameters: 8,030,261,248 (8.03B) ~14.96 GB in bf16 +[Accounting] accuracy: 0.4200 (42/100) +[Agricultural-Sciences] accuracy: 0.3040 (304/1000) +[Aviation-Engineering-and-Maintenance] accuracy: 0.3530 (353/1000) +[Biology] accuracy: 0.3560 (356/1000) +[Chemical-Engineering] accuracy: 0.4170 (417/1000) +[Chemistry] accuracy: 0.4350 (261/600) +[Civil-Engineering] accuracy: 0.4220 (422/1000) +[Computer-Science] accuracy: 0.6760 (676/1000) +[Construction] accuracy: 0.3150 (315/1000) +[Criminal-Law] accuracy: 0.3350 (67/200) +[Ecology] accuracy: 0.4050 (405/1000) +[Economics] accuracy: 0.4385 (57/130) +[Education] accuracy: 0.5300 (53/100) +[Electrical-Engineering] accuracy: 0.3070 (307/1000) +[Electronics-Engineering] accuracy: 0.5100 (510/1000) +[Energy-Management] accuracy: 0.3110 (311/1000) +[Environmental-Science] accuracy: 0.2910 (291/1000) +[Fashion] accuracy: 0.4050 (405/1000) +[Food-Processing] accuracy: 0.3520 (352/1000) +[Gas-Technology-and-Engineering] accuracy: 0.3250 (325/1000) +[Geomatics] accuracy: 0.3890 (389/1000) +[Health] accuracy: 0.5600 (56/100) +[Industrial-Engineer] accuracy: 0.3850 (385/1000) +[Information-Technology] accuracy: 0.6350 (635/1000) +[Interior-Architecture-and-Design] accuracy: 0.4580 (458/1000) +[Korean-History] accuracy: 0.3100 (31/100) +[Law] accuracy: 0.4080 (408/1000) +[Machine-Design-and-Manufacturing] accuracy: 0.3970 (397/1000) +[Management] accuracy: 0.4760 (476/1000) +[Maritime-Engineering] accuracy: 0.4250 (255/600) +[Marketing] accuracy: 0.7020 (702/1000) +[Materials-Engineering] accuracy: 0.4280 (428/1000) +[Math] accuracy: 0.2933 (88/300) +[Mechanical-Engineering] accuracy: 0.3350 (335/1000) +[Nondestructive-Testing] accuracy: 0.3960 (396/1000) +[Patent] accuracy: 0.3600 (36/100) +[Political-Science-and-Sociology] accuracy: 0.4633 (139/300) +[Psychology] accuracy: 0.3540 (354/1000) +[Public-Safety] accuracy: 0.3430 (343/1000) +[Railway-and-Automotive-Engineering] accuracy: 0.3440 (344/1000) +[Real-Estate] accuracy: 0.4300 (86/200) +[Refrigerating-Machinery] accuracy: 0.3060 (306/1000) +[Social-Welfare] accuracy: 0.4650 (465/1000) +[Taxation] accuracy: 0.3200 (64/200) +[Telecommunications-and-Wireless-Technology] accuracy: 0.5100 (510/1000) + +Params: 8,030,261,248 (8.03B) Size: 14.96 GB +STEM: 0.4281 Applied Science: 0.3863 HUMSS: 0.4109 Other: 0.4123 Average: 0.4089 +Summary appended to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/benchmark_results/kmmlu_summary.csv +=== Benchmarking llama_3_prune_(none)_tf ( tf) : XiaodongChen/Llama-3.1-5.4B === +Loading XiaodongChen/Llama-3.1-5.4B ... +Model parameters: 5,412,917,248 (5.41B) ~10.08 GB in bf16 +[Accounting] accuracy: 0.4000 (40/100) +[Agricultural-Sciences] accuracy: 0.3600 (360/1000) +[Aviation-Engineering-and-Maintenance] accuracy: 0.4050 (405/1000) +[Biology] accuracy: 0.3570 (357/1000) +[Chemical-Engineering] accuracy: 0.3890 (389/1000) +[Chemistry] accuracy: 0.4217 (253/600) +[Civil-Engineering] accuracy: 0.4300 (430/1000) +[Computer-Science] accuracy: 0.7070 (707/1000) +[Construction] accuracy: 0.3910 (391/1000) +[Criminal-Law] accuracy: 0.3250 (65/200) +[Ecology] accuracy: 0.4750 (475/1000) +[Economics] accuracy: 0.3538 (46/130) +[Education] accuracy: 0.5200 (52/100) +[Electrical-Engineering] accuracy: 0.3650 (365/1000) +[Electronics-Engineering] accuracy: 0.5480 (548/1000) +[Energy-Management] accuracy: 0.3120 (312/1000) +[Environmental-Science] accuracy: 0.3360 (336/1000) +[Fashion] accuracy: 0.4450 (445/1000) +[Food-Processing] accuracy: 0.4140 (414/1000) +[Gas-Technology-and-Engineering] accuracy: 0.3460 (346/1000) +[Geomatics] accuracy: 0.4060 (406/1000) +[Health] accuracy: 0.5300 (53/100) +[Industrial-Engineer] accuracy: 0.4590 (459/1000) +[Information-Technology] accuracy: 0.6700 (670/1000) +[Interior-Architecture-and-Design] accuracy: 0.5100 (510/1000) +[Korean-History] accuracy: 0.3200 (32/100) +[Law] accuracy: 0.3710 (371/1000) +[Machine-Design-and-Manufacturing] accuracy: 0.4110 (411/1000) +[Management] accuracy: 0.4470 (447/1000) +[Maritime-Engineering] accuracy: 0.4383 (263/600) +[Marketing] accuracy: 0.6980 (698/1000) +[Materials-Engineering] accuracy: 0.4470 (447/1000) +[Math] accuracy: 0.2767 (83/300) +[Mechanical-Engineering] accuracy: 0.3960 (396/1000) +[Nondestructive-Testing] accuracy: 0.4360 (436/1000) +[Patent] accuracy: 0.3400 (34/100) +[Political-Science-and-Sociology] accuracy: 0.4400 (132/300) +[Psychology] accuracy: 0.3490 (349/1000) +[Public-Safety] accuracy: 0.4030 (403/1000) +[Railway-and-Automotive-Engineering] accuracy: 0.3480 (348/1000) +[Real-Estate] accuracy: 0.4250 (85/200) +[Refrigerating-Machinery] accuracy: 0.3470 (347/1000) +[Social-Welfare] accuracy: 0.4650 (465/1000) +[Taxation] accuracy: 0.3000 (60/200) +[Telecommunications-and-Wireless-Technology] accuracy: 0.5530 (553/1000) + +Params: 5,412,917,248 (5.41B) Size: 10.08 GB +STEM: 0.4486 Applied Science: 0.4165 HUMSS: 0.3901 Other: 0.4421 Average: 0.4241 +Summary appended to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/benchmark_results/kmmlu_summary.csv diff --git a/llm-streamline/slurm_output/benchmark-kmmlu_opt-6.7b.out b/llm-streamline/slurm_output/benchmark-kmmlu_opt-6.7b.out new file mode 100644 index 0000000..3bbf9ce --- /dev/null +++ b/llm-streamline/slurm_output/benchmark-kmmlu_opt-6.7b.out @@ -0,0 +1,208 @@ +=== Benchmarking opt_6.7b ( baseline) : facebook/opt-6.7b === +Loading facebook/opt-6.7b ... +Model parameters: 6,658,473,984 (6.66B) ~12.40 GB in bf16 +[Accounting] accuracy: 0.2000 (20/100) +[Agricultural-Sciences] accuracy: 0.2530 (253/1000) +[Aviation-Engineering-and-Maintenance] accuracy: 0.3160 (316/1000) +[Biology] accuracy: 0.2780 (278/1000) +[Chemical-Engineering] accuracy: 0.2600 (260/1000) +[Chemistry] accuracy: 0.2933 (176/600) +[Civil-Engineering] accuracy: 0.3170 (317/1000) +[Computer-Science] accuracy: 0.3310 (331/1000) +[Construction] accuracy: 0.3420 (342/1000) +[Criminal-Law] accuracy: 0.2400 (48/200) [truncated context: 30/200] +[Ecology] accuracy: 0.2960 (296/1000) +[Economics] accuracy: 0.2462 (32/130) +[Education] accuracy: 0.2200 (22/100) +[Electrical-Engineering] accuracy: 0.3270 (327/1000) +[Electronics-Engineering] accuracy: 0.3220 (322/1000) +[Energy-Management] accuracy: 0.2760 (276/1000) +[Environmental-Science] accuracy: 0.3060 (306/1000) +[Fashion] accuracy: 0.2800 (280/1000) +[Food-Processing] accuracy: 0.2760 (276/1000) +[Gas-Technology-and-Engineering] accuracy: 0.3260 (326/1000) +[Geomatics] accuracy: 0.3260 (326/1000) +[Health] accuracy: 0.2000 (20/100) +[Industrial-Engineer] accuracy: 0.3100 (310/1000) +[Information-Technology] accuracy: 0.3150 (315/1000) +[Interior-Architecture-and-Design] accuracy: 0.3060 (306/1000) +[Korean-History] accuracy: 0.1700 (17/100) +[Law] accuracy: 0.2470 (247/1000) [truncated context: 3/1000] +[Machine-Design-and-Manufacturing] accuracy: 0.2550 (255/1000) +[Management] accuracy: 0.2520 (252/1000) +[Maritime-Engineering] accuracy: 0.2750 (165/600) +[Marketing] accuracy: 0.2430 (243/1000) +[Materials-Engineering] accuracy: 0.3150 (315/1000) +[Math] accuracy: 0.2633 (79/300) +[Mechanical-Engineering] accuracy: 0.3160 (316/1000) +[Nondestructive-Testing] accuracy: 0.2670 (267/1000) +[Patent] accuracy: 0.2100 (21/100) [truncated context: 29/100] +[Political-Science-and-Sociology] accuracy: 0.2533 (76/300) +[Psychology] accuracy: 0.2520 (252/1000) +[Public-Safety] accuracy: 0.3180 (318/1000) +[Railway-and-Automotive-Engineering] accuracy: 0.2960 (296/1000) +[Real-Estate] accuracy: 0.2000 (40/200) +[Refrigerating-Machinery] accuracy: 0.1980 (198/1000) +[Social-Welfare] accuracy: 0.2720 (272/1000) +[Taxation] accuracy: 0.3050 (61/200) +[Telecommunications-and-Wireless-Technology] accuracy: 0.3410 (341/1000) + +Params: 6,658,473,984 (6.66B) Size: 12.40 GB +STEM: 0.3011 Applied Science: 0.3013 HUMSS: 0.2416 Other: 0.2569 Average: 0.2758 +Summary appended to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/benchmark_results/kmmlu_summary.csv +=== Benchmarking opt_prune_from2to9_none ( none) : /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_none === +Loading /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_none ... +Model parameters: 5,047,435,264 (5.05B) ~9.40 GB in bf16 +[Accounting] accuracy: 0.1800 (18/100) +[Agricultural-Sciences] accuracy: 0.1010 (101/1000) +[Aviation-Engineering-and-Maintenance] accuracy: 0.1090 (109/1000) +[Biology] accuracy: 0.2700 (270/1000) +[Chemical-Engineering] accuracy: 0.2070 (207/1000) +[Chemistry] accuracy: 0.2600 (156/600) +[Civil-Engineering] accuracy: 0.2830 (283/1000) +[Computer-Science] accuracy: 0.2370 (237/1000) +[Construction] accuracy: 0.0920 (92/1000) +[Criminal-Law] accuracy: 0.2150 (43/200) [truncated context: 30/200] +[Ecology] accuracy: 0.0720 (72/1000) +[Economics] accuracy: 0.3000 (39/130) +[Education] accuracy: 0.2200 (22/100) +[Electrical-Engineering] accuracy: 0.0610 (61/1000) +[Electronics-Engineering] accuracy: 0.2450 (245/1000) +[Energy-Management] accuracy: 0.2390 (239/1000) +[Environmental-Science] accuracy: 0.1530 (153/1000) +[Fashion] accuracy: 0.1380 (138/1000) +[Food-Processing] accuracy: 0.1140 (114/1000) +[Gas-Technology-and-Engineering] accuracy: 0.1460 (146/1000) +[Geomatics] accuracy: 0.2990 (299/1000) +[Health] accuracy: 0.2400 (24/100) +[Industrial-Engineer] accuracy: 0.1600 (160/1000) +[Information-Technology] accuracy: 0.2240 (224/1000) +[Interior-Architecture-and-Design] accuracy: 0.0610 (61/1000) +[Korean-History] accuracy: 0.2000 (20/100) +[Law] accuracy: 0.2290 (229/1000) [truncated context: 3/1000] +[Machine-Design-and-Manufacturing] accuracy: 0.0830 (83/1000) +[Management] accuracy: 0.2400 (240/1000) +[Maritime-Engineering] accuracy: 0.1600 (96/600) +[Marketing] accuracy: 0.1390 (139/1000) +[Materials-Engineering] accuracy: 0.1850 (185/1000) +[Math] accuracy: 0.2500 (75/300) +[Mechanical-Engineering] accuracy: 0.2350 (235/1000) +[Nondestructive-Testing] accuracy: 0.2430 (243/1000) +[Patent] accuracy: 0.2800 (28/100) [truncated context: 29/100] +[Political-Science-and-Sociology] accuracy: 0.2233 (67/300) +[Psychology] accuracy: 0.2640 (264/1000) +[Public-Safety] accuracy: 0.0390 (39/1000) +[Railway-and-Automotive-Engineering] accuracy: 0.1500 (150/1000) +[Real-Estate] accuracy: 0.1800 (36/200) +[Refrigerating-Machinery] accuracy: 0.1830 (183/1000) +[Social-Welfare] accuracy: 0.2650 (265/1000) +[Taxation] accuracy: 0.2150 (43/200) +[Telecommunications-and-Wireless-Technology] accuracy: 0.2190 (219/1000) + +Params: 5,047,435,264 (5.05B) Size: 9.40 GB +STEM: 0.2076 Applied Science: 0.1838 HUMSS: 0.2319 Other: 0.1425 Average: 0.1913 +Summary appended to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/benchmark_results/kmmlu_summary.csv +=== Benchmarking opt_prune_from2to9_mlp ( mlp) : /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp === +Loading /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp ... +Model parameters: 5,181,673,472 (5.18B) ~9.65 GB in bf16 +[Accounting] accuracy: 0.3000 (30/100) +[Agricultural-Sciences] accuracy: 0.0880 (88/1000) +[Aviation-Engineering-and-Maintenance] accuracy: 0.3160 (316/1000) +[Biology] accuracy: 0.2870 (287/1000) +[Chemical-Engineering] accuracy: 0.1950 (195/1000) +[Chemistry] accuracy: 0.2933 (176/600) +[Civil-Engineering] accuracy: 0.3240 (324/1000) +[Computer-Science] accuracy: 0.3220 (322/1000) +[Construction] accuracy: 0.3370 (337/1000) +[Criminal-Law] accuracy: 0.2600 (52/200) [truncated context: 30/200] +[Ecology] accuracy: 0.1730 (173/1000) +[Economics] accuracy: 0.2462 (32/130) +[Education] accuracy: 0.2300 (23/100) +[Electrical-Engineering] accuracy: 0.3280 (328/1000) +[Electronics-Engineering] accuracy: 0.3170 (317/1000) +[Energy-Management] accuracy: 0.2440 (244/1000) +[Environmental-Science] accuracy: 0.0180 (18/1000) +[Fashion] accuracy: 0.1350 (135/1000) +[Food-Processing] accuracy: 0.2680 (268/1000) +[Gas-Technology-and-Engineering] accuracy: 0.1950 (195/1000) +[Geomatics] accuracy: 0.3020 (302/1000) +[Health] accuracy: 0.2200 (22/100) +[Industrial-Engineer] accuracy: 0.3300 (330/1000) +[Information-Technology] accuracy: 0.3060 (306/1000) +[Interior-Architecture-and-Design] accuracy: 0.2950 (295/1000) +[Korean-History] accuracy: 0.2000 (20/100) +[Law] accuracy: 0.2470 (247/1000) [truncated context: 3/1000] +[Machine-Design-and-Manufacturing] accuracy: 0.0700 (70/1000) +[Management] accuracy: 0.2090 (209/1000) +[Maritime-Engineering] accuracy: 0.1833 (110/600) +[Marketing] accuracy: 0.2430 (243/1000) +[Materials-Engineering] accuracy: 0.3010 (301/1000) +[Math] accuracy: 0.2700 (81/300) +[Mechanical-Engineering] accuracy: 0.3210 (321/1000) +[Nondestructive-Testing] accuracy: 0.3450 (345/1000) +[Patent] accuracy: 0.2400 (24/100) [truncated context: 29/100] +[Political-Science-and-Sociology] accuracy: 0.2733 (82/300) +[Psychology] accuracy: 0.2590 (259/1000) +[Public-Safety] accuracy: 0.3220 (322/1000) +[Railway-and-Automotive-Engineering] accuracy: 0.2960 (296/1000) +[Real-Estate] accuracy: 0.2550 (51/200) +[Refrigerating-Machinery] accuracy: 0.2190 (219/1000) +[Social-Welfare] accuracy: 0.2730 (273/1000) +[Taxation] accuracy: 0.2900 (58/200) +[Telecommunications-and-Wireless-Technology] accuracy: 0.3410 (341/1000) + +Params: 5,181,673,472 (5.18B) Size: 9.65 GB +STEM: 0.2837 Applied Science: 0.2464 HUMSS: 0.2534 Other: 0.2384 Average: 0.2553 +Summary appended to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/benchmark_results/kmmlu_summary.csv +=== Benchmarking opt_prune_from2to9_tf ( tf) : /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf === +Loading /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf ... +Model parameters: 5,248,815,104 (5.25B) ~9.78 GB in bf16 +[Accounting] accuracy: 0.3000 (30/100) +[Agricultural-Sciences] accuracy: 0.0880 (88/1000) +[Aviation-Engineering-and-Maintenance] accuracy: 0.3110 (311/1000) +[Biology] accuracy: 0.2870 (287/1000) +[Chemical-Engineering] accuracy: 0.2300 (230/1000) +[Chemistry] accuracy: 0.2933 (176/600) +[Civil-Engineering] accuracy: 0.3240 (324/1000) +[Computer-Science] accuracy: 0.3270 (327/1000) +[Construction] accuracy: 0.3330 (333/1000) +[Criminal-Law] accuracy: 0.2300 (46/200) [truncated context: 30/200] +[Ecology] accuracy: 0.2660 (266/1000) +[Economics] accuracy: 0.2462 (32/130) +[Education] accuracy: 0.2300 (23/100) +[Electrical-Engineering] accuracy: 0.3260 (326/1000) +[Electronics-Engineering] accuracy: 0.3160 (316/1000) +[Energy-Management] accuracy: 0.2550 (255/1000) +[Environmental-Science] accuracy: 0.1930 (193/1000) +[Fashion] accuracy: 0.1350 (135/1000) +[Food-Processing] accuracy: 0.2760 (276/1000) +[Gas-Technology-and-Engineering] accuracy: 0.3160 (316/1000) +[Geomatics] accuracy: 0.3020 (302/1000) +[Health] accuracy: 0.2200 (22/100) +[Industrial-Engineer] accuracy: 0.3280 (328/1000) +[Information-Technology] accuracy: 0.3060 (306/1000) +[Interior-Architecture-and-Design] accuracy: 0.2970 (297/1000) +[Korean-History] accuracy: 0.2200 (22/100) +[Law] accuracy: 0.2470 (247/1000) [truncated context: 3/1000] +[Machine-Design-and-Manufacturing] accuracy: 0.0710 (71/1000) +[Management] accuracy: 0.2030 (203/1000) +[Maritime-Engineering] accuracy: 0.2100 (126/600) +[Marketing] accuracy: 0.2430 (243/1000) +[Materials-Engineering] accuracy: 0.2910 (291/1000) +[Math] accuracy: 0.2733 (82/300) +[Mechanical-Engineering] accuracy: 0.3210 (321/1000) +[Nondestructive-Testing] accuracy: 0.3450 (345/1000) +[Patent] accuracy: 0.2200 (22/100) [truncated context: 29/100] +[Political-Science-and-Sociology] accuracy: 0.2733 (82/300) +[Psychology] accuracy: 0.2370 (237/1000) +[Public-Safety] accuracy: 0.3230 (323/1000) +[Railway-and-Automotive-Engineering] accuracy: 0.2920 (292/1000) +[Real-Estate] accuracy: 0.2200 (44/200) +[Refrigerating-Machinery] accuracy: 0.2540 (254/1000) +[Social-Welfare] accuracy: 0.2730 (273/1000) +[Taxation] accuracy: 0.2900 (58/200) +[Telecommunications-and-Wireless-Technology] accuracy: 0.3410 (341/1000) + +Params: 5,248,815,104 (5.25B) Size: 9.78 GB +STEM: 0.2950 Applied Science: 0.2733 HUMSS: 0.2500 Other: 0.2372 Average: 0.2641 +Summary appended to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/benchmark_results/kmmlu_summary.csv diff --git a/llm-streamline/slurm_output/replace-and-retrain_opt-6.7b-8layer-mlp.out b/llm-streamline/slurm_output/replace-and-retrain_opt-6.7b-8layer-mlp.out new file mode 100644 index 0000000..29823c3 --- /dev/null +++ b/llm-streamline/slurm_output/replace-and-retrain_opt-6.7b-8layer-mlp.out @@ -0,0 +1,329 @@ +modeling_opt_mlp.py imported for opt + mlp replacement layer. +Loading pretrained weights from facebook/opt-6.7b ... +Initialising replace_layer from pretrained layer 2 ... +Trainable params: 134,238,208 / 2,362,355,712 +Downloading SlimPajama-6B ... +Step 500 — eval MSE loss: 31.970427 +[2026-07-17 19:13:27,627] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH + [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.2 + [WARNING]  using untested triton version (2.2.0), only 1.0.0 is known to be compatible +New best (eval MSE 31.970427) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 1000 — eval MSE loss: 12.843388 +New best (eval MSE 12.843388) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 1500 — eval MSE loss: 3.371396 +New best (eval MSE 3.371396) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 2000 — eval MSE loss: 1.045532 +New best (eval MSE 1.045532) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 2500 — eval MSE loss: 0.541206 +New best (eval MSE 0.541206) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 3000 — eval MSE loss: 0.236769 +New best (eval MSE 0.236769) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 3500 — eval MSE loss: 0.115237 +New best (eval MSE 0.115237) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 4000 — eval MSE loss: 0.091583 +New best (eval MSE 0.091583) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 4500 — eval MSE loss: 0.077759 +New best (eval MSE 0.077759) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 5000 — eval MSE loss: 0.069369 +New best (eval MSE 0.069369) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 5500 — eval MSE loss: 0.064657 +New best (eval MSE 0.064657) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 6000 — eval MSE loss: 0.059060 +New best (eval MSE 0.059060) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 6500 — eval MSE loss: 0.053491 +New best (eval MSE 0.053491) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 7000 — eval MSE loss: 0.048818 +New best (eval MSE 0.048818) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 7500 — eval MSE loss: 0.044867 +New best (eval MSE 0.044867) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 8000 — eval MSE loss: 0.041645 +New best (eval MSE 0.041645) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 8500 — eval MSE loss: 0.039275 +New best (eval MSE 0.039275) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 9000 — eval MSE loss: 0.037394 +New best (eval MSE 0.037394) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 9500 — eval MSE loss: 0.035997 +New best (eval MSE 0.035997) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 10000 — eval MSE loss: 0.034626 +New best (eval MSE 0.034626) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 10500 — eval MSE loss: 0.033674 +New best (eval MSE 0.033674) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 11000 — eval MSE loss: 0.032908 +New best (eval MSE 0.032908) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 11500 — eval MSE loss: 0.032144 +New best (eval MSE 0.032144) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 12000 — eval MSE loss: 0.031429 +New best (eval MSE 0.031429) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 12500 — eval MSE loss: 0.030973 +New best (eval MSE 0.030973) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 13000 — eval MSE loss: 0.030389 +New best (eval MSE 0.030389) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 13500 — eval MSE loss: 0.030099 +New best (eval MSE 0.030099) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 14000 — eval MSE loss: 0.029687 +New best (eval MSE 0.029687) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 14500 — eval MSE loss: 0.029354 +New best (eval MSE 0.029354) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 15000 — eval MSE loss: 0.028979 +New best (eval MSE 0.028979) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 15500 — eval MSE loss: 0.028764 +New best (eval MSE 0.028764) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 16000 — eval MSE loss: 0.028455 +New best (eval MSE 0.028455) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 16500 — eval MSE loss: 0.028197 +New best (eval MSE 0.028197) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 17000 — eval MSE loss: 0.027934 +New best (eval MSE 0.027934) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 17500 — eval MSE loss: 0.027792 +New best (eval MSE 0.027792) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 18000 — eval MSE loss: 0.027688 +New best (eval MSE 0.027688) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 18500 — eval MSE loss: 0.027373 +New best (eval MSE 0.027373) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 19000 — eval MSE loss: 0.027244 +New best (eval MSE 0.027244) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 19500 — eval MSE loss: 0.027101 +New best (eval MSE 0.027101) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 20000 — eval MSE loss: 0.026982 +New best (eval MSE 0.026982) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 20500 — eval MSE loss: 0.026735 +New best (eval MSE 0.026735) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 21000 — eval MSE loss: 0.026753 +Step 21500 — eval MSE loss: 0.026637 +New best (eval MSE 0.026637) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 22000 — eval MSE loss: 0.026326 +New best (eval MSE 0.026326) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 22500 — eval MSE loss: 0.026282 +New best (eval MSE 0.026282) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 23000 — eval MSE loss: 0.026197 +New best (eval MSE 0.026197) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 23500 — eval MSE loss: 0.025971 +New best (eval MSE 0.025971) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 24000 — eval MSE loss: 0.025866 +New best (eval MSE 0.025866) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 24500 — eval MSE loss: 0.025771 +New best (eval MSE 0.025771) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 25000 — eval MSE loss: 0.025712 +New best (eval MSE 0.025712) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 25500 — eval MSE loss: 0.025592 +New best (eval MSE 0.025592) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 26000 — eval MSE loss: 0.025480 +New best (eval MSE 0.025480) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 26500 — eval MSE loss: 0.025403 +New best (eval MSE 0.025403) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 27000 — eval MSE loss: 0.025368 +New best (eval MSE 0.025368) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 27500 — eval MSE loss: 0.025356 +New best (eval MSE 0.025356) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 28000 — eval MSE loss: 0.025226 +New best (eval MSE 0.025226) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 28500 — eval MSE loss: 0.025134 +New best (eval MSE 0.025134) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 29000 — eval MSE loss: 0.025211 +Step 29500 — eval MSE loss: 0.024980 +New best (eval MSE 0.024980) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 30000 — eval MSE loss: 0.024952 +New best (eval MSE 0.024952) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 30500 — eval MSE loss: 0.024846 +New best (eval MSE 0.024846) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 31000 — eval MSE loss: 0.024825 +New best (eval MSE 0.024825) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 31500 — eval MSE loss: 0.024724 +New best (eval MSE 0.024724) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 32000 — eval MSE loss: 0.024716 +New best (eval MSE 0.024716) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 32500 — eval MSE loss: 0.024561 +New best (eval MSE 0.024561) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 33000 — eval MSE loss: 0.024492 +New best (eval MSE 0.024492) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 33500 — eval MSE loss: 0.024459 +New best (eval MSE 0.024459) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 34000 — eval MSE loss: 0.024443 +New best (eval MSE 0.024443) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 34500 — eval MSE loss: 0.024348 +New best (eval MSE 0.024348) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 35000 — eval MSE loss: 0.024300 +New best (eval MSE 0.024300) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 35500 — eval MSE loss: 0.024337 +Step 36000 — eval MSE loss: 0.024211 +New best (eval MSE 0.024211) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 36500 — eval MSE loss: 0.024145 +New best (eval MSE 0.024145) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 37000 — eval MSE loss: 0.024152 +Step 37500 — eval MSE loss: 0.024125 +New best (eval MSE 0.024125) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 38000 — eval MSE loss: 0.024053 +New best (eval MSE 0.024053) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 38500 — eval MSE loss: 0.024006 +New best (eval MSE 0.024006) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 39000 — eval MSE loss: 0.024017 +Step 39500 — eval MSE loss: 0.024010 +Step 40000 — eval MSE loss: 0.023930 +New best (eval MSE 0.023930) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 40500 — eval MSE loss: 0.023858 +New best (eval MSE 0.023858) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 41000 — eval MSE loss: 0.023861 +Step 41500 — eval MSE loss: 0.023833 +New best (eval MSE 0.023833) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 42000 — eval MSE loss: 0.023688 +New best (eval MSE 0.023688) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 42500 — eval MSE loss: 0.023687 +New best (eval MSE 0.023687) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 43000 — eval MSE loss: 0.023661 +New best (eval MSE 0.023661) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 43500 — eval MSE loss: 0.023629 +New best (eval MSE 0.023629) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 44000 — eval MSE loss: 0.023604 +New best (eval MSE 0.023604) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 44500 — eval MSE loss: 0.023602 +New best (eval MSE 0.023602) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 45000 — eval MSE loss: 0.023488 +New best (eval MSE 0.023488) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 45500 — eval MSE loss: 0.023553 +Step 46000 — eval MSE loss: 0.023436 +New best (eval MSE 0.023436) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 46500 — eval MSE loss: 0.023415 +New best (eval MSE 0.023415) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 47000 — eval MSE loss: 0.023505 +Step 47500 — eval MSE loss: 0.023379 +New best (eval MSE 0.023379) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 48000 — eval MSE loss: 0.023331 +New best (eval MSE 0.023331) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 48500 — eval MSE loss: 0.023331 +New best (eval MSE 0.023331) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 49000 — eval MSE loss: 0.023261 +New best (eval MSE 0.023261) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 49500 — eval MSE loss: 0.023395 +Step 50000 — eval MSE loss: 0.023274 +Step 50500 — eval MSE loss: 0.023220 +New best (eval MSE 0.023220) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 51000 — eval MSE loss: 0.023210 +New best (eval MSE 0.023210) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 51500 — eval MSE loss: 0.023283 +Step 52000 — eval MSE loss: 0.023198 +New best (eval MSE 0.023198) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 52500 — eval MSE loss: 0.023123 +New best (eval MSE 0.023123) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 53000 — eval MSE loss: 0.023141 +Step 53500 — eval MSE loss: 0.023050 +New best (eval MSE 0.023050) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 54000 — eval MSE loss: 0.023118 +Step 54500 — eval MSE loss: 0.023063 +Step 55000 — eval MSE loss: 0.023032 +New best (eval MSE 0.023032) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 55500 — eval MSE loss: 0.023003 +New best (eval MSE 0.023003) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 56000 — eval MSE loss: 0.022985 +New best (eval MSE 0.022985) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 56500 — eval MSE loss: 0.022929 +New best (eval MSE 0.022929) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 57000 — eval MSE loss: 0.022974 +Step 57500 — eval MSE loss: 0.022987 +Step 58000 — eval MSE loss: 0.022908 +New best (eval MSE 0.022908) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 58500 — eval MSE loss: 0.022975 +Step 59000 — eval MSE loss: 0.022811 +New best (eval MSE 0.022811) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 59500 — eval MSE loss: 0.022842 +Step 60000 — eval MSE loss: 0.022847 +Step 60500 — eval MSE loss: 0.022909 +Step 61000 — eval MSE loss: 0.022791 +New best (eval MSE 0.022791) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 61500 — eval MSE loss: 0.022737 +New best (eval MSE 0.022737) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 62000 — eval MSE loss: 0.022799 +Step 62500 — eval MSE loss: 0.022743 +Step 63000 — eval MSE loss: 0.022715 +New best (eval MSE 0.022715) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 63500 — eval MSE loss: 0.022651 +New best (eval MSE 0.022651) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 64000 — eval MSE loss: 0.022749 +Step 64500 — eval MSE loss: 0.022638 +New best (eval MSE 0.022638) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 65000 — eval MSE loss: 0.022616 +New best (eval MSE 0.022616) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 65500 — eval MSE loss: 0.022578 +New best (eval MSE 0.022578) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 66000 — eval MSE loss: 0.022545 +New best (eval MSE 0.022545) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 66500 — eval MSE loss: 0.022546 +Step 67000 — eval MSE loss: 0.022532 +New best (eval MSE 0.022532) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 67500 — eval MSE loss: 0.022596 +Step 68000 — eval MSE loss: 0.022509 +New best (eval MSE 0.022509) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 68500 — eval MSE loss: 0.022548 +Step 69000 — eval MSE loss: 0.022465 +New best (eval MSE 0.022465) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 69500 — eval MSE loss: 0.022482 +Step 70000 — eval MSE loss: 0.022463 +New best (eval MSE 0.022463) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 70500 — eval MSE loss: 0.022576 +Step 71000 — eval MSE loss: 0.022427 +New best (eval MSE 0.022427) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 71500 — eval MSE loss: 0.022382 +New best (eval MSE 0.022382) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 72000 — eval MSE loss: 0.022368 +New best (eval MSE 0.022368) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 72500 — eval MSE loss: 0.022353 +New best (eval MSE 0.022353) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 73000 — eval MSE loss: 0.022428 +Step 73500 — eval MSE loss: 0.022401 +Step 74000 — eval MSE loss: 0.022316 +New best (eval MSE 0.022316) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 74500 — eval MSE loss: 0.022473 +Step 75000 — eval MSE loss: 0.022308 +New best (eval MSE 0.022308) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 75500 — eval MSE loss: 0.022305 +New best (eval MSE 0.022305) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 76000 — eval MSE loss: 0.022375 +Step 76500 — eval MSE loss: 0.022435 +Step 77000 — eval MSE loss: 0.022366 +Step 77500 — eval MSE loss: 0.022233 +New best (eval MSE 0.022233) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 78000 — eval MSE loss: 0.022302 +Step 78500 — eval MSE loss: 0.022228 +New best (eval MSE 0.022228) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 79000 — eval MSE loss: 0.022180 +New best (eval MSE 0.022180) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 79500 — eval MSE loss: 0.022266 +Step 80000 — eval MSE loss: 0.022324 +Step 80500 — eval MSE loss: 0.022227 +Step 81000 — eval MSE loss: 0.022261 +Step 81500 — eval MSE loss: 0.022125 +New best (eval MSE 0.022125) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 82000 — eval MSE loss: 0.022187 +Step 82500 — eval MSE loss: 0.022200 +Step 83000 — eval MSE loss: 0.022123 +New best (eval MSE 0.022123) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 83500 — eval MSE loss: 0.022092 +New best (eval MSE 0.022092) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 84000 — eval MSE loss: 0.022123 +Step 84500 — eval MSE loss: 0.022065 +New best (eval MSE 0.022065) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 85000 — eval MSE loss: 0.022139 +Step 85500 — eval MSE loss: 0.022083 +Step 86000 — eval MSE loss: 0.022064 +New best (eval MSE 0.022064) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 86500 — eval MSE loss: 0.022055 +New best (eval MSE 0.022055) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 87000 — eval MSE loss: 0.022029 +New best (eval MSE 0.022029) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 87500 — eval MSE loss: 0.022073 +Step 88000 — eval MSE loss: 0.022016 +New best (eval MSE 0.022016) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 88500 — eval MSE loss: 0.022010 +New best (eval MSE 0.022010) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 89000 — eval MSE loss: 0.022121 +Step 89500 — eval MSE loss: 0.021963 +New best (eval MSE 0.021963) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp +Step 90000 — eval MSE loss: 0.021969 +Step 90500 — eval MSE loss: 0.022023 +Step 91000 — eval MSE loss: 0.021992 +Step 91500 — eval MSE loss: 0.022052 +Step 92000 — eval MSE loss: 0.022063 +No new best eval loss for 5 consecutive evals — stopping early at step 92000. diff --git a/llm-streamline/slurm_output/replace-and-retrain_opt-6.7b-8layer-none.out b/llm-streamline/slurm_output/replace-and-retrain_opt-6.7b-8layer-none.out new file mode 100644 index 0000000..196085b --- /dev/null +++ b/llm-streamline/slurm_output/replace-and-retrain_opt-6.7b-8layer-none.out @@ -0,0 +1,10 @@ +modeling_opt_none.py imported for opt + none replacement layer. +Loading pretrained weights from facebook/opt-6.7b ... +[2026-07-14 23:28:19,675] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH + [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.2 + [WARNING]  using untested triton version (2.2.0), only 1.0.0 is known to be compatible +Pruned model saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_none (layers 2–9 removed) diff --git a/llm-streamline/slurm_output/replace-and-retrain_opt-6.7b-8layer-tf.out b/llm-streamline/slurm_output/replace-and-retrain_opt-6.7b-8layer-tf.out new file mode 100644 index 0000000..c920fa3 --- /dev/null +++ b/llm-streamline/slurm_output/replace-and-retrain_opt-6.7b-8layer-tf.out @@ -0,0 +1,154 @@ +modeling_opt_tf.py imported for opt + tf replacement layer. +Loading pretrained weights from facebook/opt-6.7b ... +Initialising replace_layer from pretrained layer 2 ... +Trainable params: 201,379,840 / 2,429,497,344 +Downloading SlimPajama-6B ... +Step 500 — eval MSE loss: 0.059836 +[2026-07-17 19:38:12,501] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect) + [WARNING]  async_io requires the dev libaio .so object and headers but these were not found. + [WARNING]  async_io: please install the libaio-devel package with yum + [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found. + [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH + [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.2 + [WARNING]  using untested triton version (2.2.0), only 1.0.0 is known to be compatible +New best (eval MSE 0.059836) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 1000 — eval MSE loss: 0.055788 +New best (eval MSE 0.055788) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 1500 — eval MSE loss: 0.052881 +New best (eval MSE 0.052881) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 2000 — eval MSE loss: 0.049815 +New best (eval MSE 0.049815) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 2500 — eval MSE loss: 0.047491 +New best (eval MSE 0.047491) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 3000 — eval MSE loss: 0.045000 +New best (eval MSE 0.045000) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 3500 — eval MSE loss: 0.042638 +New best (eval MSE 0.042638) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 4000 — eval MSE loss: 0.041050 +New best (eval MSE 0.041050) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 4500 — eval MSE loss: 0.039271 +New best (eval MSE 0.039271) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 5000 — eval MSE loss: 0.037365 +New best (eval MSE 0.037365) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 5500 — eval MSE loss: 0.036405 +New best (eval MSE 0.036405) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 6000 — eval MSE loss: 0.035525 +New best (eval MSE 0.035525) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 6500 — eval MSE loss: 0.033941 +New best (eval MSE 0.033941) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 7000 — eval MSE loss: 0.033056 +New best (eval MSE 0.033056) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 7500 — eval MSE loss: 0.033340 +Step 8000 — eval MSE loss: 0.031683 +New best (eval MSE 0.031683) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 8500 — eval MSE loss: 0.031682 +New best (eval MSE 0.031682) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 9000 — eval MSE loss: 0.030211 +New best (eval MSE 0.030211) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 9500 — eval MSE loss: 0.029737 +New best (eval MSE 0.029737) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 10000 — eval MSE loss: 0.029402 +New best (eval MSE 0.029402) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 10500 — eval MSE loss: 0.028755 +New best (eval MSE 0.028755) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 11000 — eval MSE loss: 0.028555 +New best (eval MSE 0.028555) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 11500 — eval MSE loss: 0.028266 +New best (eval MSE 0.028266) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 12000 — eval MSE loss: 0.027625 +New best (eval MSE 0.027625) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 12500 — eval MSE loss: 0.027145 +New best (eval MSE 0.027145) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 13000 — eval MSE loss: 0.026950 +New best (eval MSE 0.026950) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 13500 — eval MSE loss: 0.026647 +New best (eval MSE 0.026647) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 14000 — eval MSE loss: 0.026631 +New best (eval MSE 0.026631) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 14500 — eval MSE loss: 0.026371 +New best (eval MSE 0.026371) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 15000 — eval MSE loss: 0.026283 +New best (eval MSE 0.026283) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 15500 — eval MSE loss: 0.025538 +New best (eval MSE 0.025538) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 16000 — eval MSE loss: 0.025680 +Step 16500 — eval MSE loss: 0.025553 +Step 17000 — eval MSE loss: 0.025258 +New best (eval MSE 0.025258) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 17500 — eval MSE loss: 0.025117 +New best (eval MSE 0.025117) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 18000 — eval MSE loss: 0.024766 +New best (eval MSE 0.024766) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 18500 — eval MSE loss: 0.024601 +New best (eval MSE 0.024601) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 19000 — eval MSE loss: 0.024586 +New best (eval MSE 0.024586) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 19500 — eval MSE loss: 0.024297 +New best (eval MSE 0.024297) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 20000 — eval MSE loss: 0.024391 +Step 20500 — eval MSE loss: 0.024004 +New best (eval MSE 0.024004) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 21000 — eval MSE loss: 0.023734 +New best (eval MSE 0.023734) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 21500 — eval MSE loss: 0.023898 +Step 22000 — eval MSE loss: 0.023944 +Step 22500 — eval MSE loss: 0.023631 +New best (eval MSE 0.023631) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 23000 — eval MSE loss: 0.023261 +New best (eval MSE 0.023261) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 23500 — eval MSE loss: 0.023317 +Step 24000 — eval MSE loss: 0.023263 +Step 24500 — eval MSE loss: 0.023147 +New best (eval MSE 0.023147) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 25000 — eval MSE loss: 0.022902 +New best (eval MSE 0.022902) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 25500 — eval MSE loss: 0.023164 +Step 26000 — eval MSE loss: 0.022974 +Step 26500 — eval MSE loss: 0.022783 +New best (eval MSE 0.022783) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 27000 — eval MSE loss: 0.022437 +New best (eval MSE 0.022437) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 27500 — eval MSE loss: 0.022533 +Step 28000 — eval MSE loss: 0.022645 +Step 28500 — eval MSE loss: 0.022451 +Step 29000 — eval MSE loss: 0.022362 +New best (eval MSE 0.022362) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 29500 — eval MSE loss: 0.022698 +Step 30000 — eval MSE loss: 0.021984 +New best (eval MSE 0.021984) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 30500 — eval MSE loss: 0.022164 +Step 31000 — eval MSE loss: 0.022143 +Step 31500 — eval MSE loss: 0.021894 +New best (eval MSE 0.021894) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 32000 — eval MSE loss: 0.022185 +Step 32500 — eval MSE loss: 0.021876 +New best (eval MSE 0.021876) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 33000 — eval MSE loss: 0.022154 +Step 33500 — eval MSE loss: 0.022033 +Step 34000 — eval MSE loss: 0.021762 +New best (eval MSE 0.021762) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 34500 — eval MSE loss: 0.021507 +New best (eval MSE 0.021507) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 35000 — eval MSE loss: 0.021994 +Step 35500 — eval MSE loss: 0.021603 +Step 36000 — eval MSE loss: 0.021545 +Step 36500 — eval MSE loss: 0.021371 +New best (eval MSE 0.021371) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 37000 — eval MSE loss: 0.021257 +New best (eval MSE 0.021257) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 37500 — eval MSE loss: 0.021308 +Step 38000 — eval MSE loss: 0.021432 +Step 38500 — eval MSE loss: 0.021251 +New best (eval MSE 0.021251) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 39000 — eval MSE loss: 0.021273 +Step 39500 — eval MSE loss: 0.021021 +New best (eval MSE 0.021021) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 40000 — eval MSE loss: 0.021090 +Step 40500 — eval MSE loss: 0.020831 +New best (eval MSE 0.020831) saved to /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf +Step 41000 — eval MSE loss: 0.021018 +Step 41500 — eval MSE loss: 0.021124 +Step 42000 — eval MSE loss: 0.021147 +Step 42500 — eval MSE loss: 0.020989 +Step 43000 — eval MSE loss: 0.020947 +No new best eval loss for 5 consecutive evals — stopping early at step 43000. diff --git a/llm-streamline/slurm_output/search-pruning-layer_Llama-2-7b-hf.out b/llm-streamline/slurm_output/search-pruning-layer_Llama-2-7b-hf.out new file mode 100644 index 0000000..a426a30 --- /dev/null +++ b/llm-streamline/slurm_output/search-pruning-layer_Llama-2-7b-hf.out @@ -0,0 +1,30 @@ +Calculating cosine similarity... +The cosine similarity between hidden_states 0 and hidden_states 8 is 0.5823 +The cosine similarity between hidden_states 1 and hidden_states 9 is 0.6588 +The cosine similarity between hidden_states 2 and hidden_states 10 is 0.7011 +The cosine similarity between hidden_states 3 and hidden_states 11 is 0.7049 +The cosine similarity between hidden_states 4 and hidden_states 12 is 0.7229 +The cosine similarity between hidden_states 5 and hidden_states 13 is 0.7397 +The cosine similarity between hidden_states 6 and hidden_states 14 is 0.7506 +The cosine similarity between hidden_states 7 and hidden_states 15 is 0.7615 +The cosine similarity between hidden_states 8 and hidden_states 16 is 0.7707 +The cosine similarity between hidden_states 9 and hidden_states 17 is 0.7653 +The cosine similarity between hidden_states 10 and hidden_states 18 is 0.7630 +The cosine similarity between hidden_states 11 and hidden_states 19 is 0.7607 +The cosine similarity between hidden_states 12 and hidden_states 20 is 0.7633 +The cosine similarity between hidden_states 13 and hidden_states 21 is 0.7615 +The cosine similarity between hidden_states 14 and hidden_states 22 is 0.7694 +The cosine similarity between hidden_states 15 and hidden_states 23 is 0.7775 +The cosine similarity between hidden_states 16 and hidden_states 24 is 0.7908 +The cosine similarity between hidden_states 17 and hidden_states 25 is 0.8123 +The cosine similarity between hidden_states 18 and hidden_states 26 is 0.8256 +The cosine similarity between hidden_states 19 and hidden_states 27 is 0.8344 +The cosine similarity between hidden_states 20 and hidden_states 28 is 0.8440 +The cosine similarity between hidden_states 21 and hidden_states 29 is 0.8524 +The cosine similarity between hidden_states 22 and hidden_states 30 is 0.8511 +The cosine similarity between hidden_states 23 and hidden_states 31 is 0.8382 +The cosine similarity between hidden_states 24 and hidden_states 32 is 0.6924 +The highest cosine similarity comes from hidden_states 21 and hidden_states 29, with a value of 0.8524 + + +pruning_layer: [21, 28] diff --git a/llm-streamline/slurm_output/search-pruning-layer_Llama-3.1-8B.out b/llm-streamline/slurm_output/search-pruning-layer_Llama-3.1-8B.out new file mode 100644 index 0000000..2886280 --- /dev/null +++ b/llm-streamline/slurm_output/search-pruning-layer_Llama-3.1-8B.out @@ -0,0 +1,30 @@ +Calculating cosine similarity... +The cosine similarity between hidden_states 0 and hidden_states 8 is 0.5541 +The cosine similarity between hidden_states 1 and hidden_states 9 is 0.6372 +The cosine similarity between hidden_states 2 and hidden_states 10 is 0.6560 +The cosine similarity between hidden_states 3 and hidden_states 11 is 0.6789 +The cosine similarity between hidden_states 4 and hidden_states 12 is 0.6887 +The cosine similarity between hidden_states 5 and hidden_states 13 is 0.7062 +The cosine similarity between hidden_states 6 and hidden_states 14 is 0.7204 +The cosine similarity between hidden_states 7 and hidden_states 15 is 0.7339 +The cosine similarity between hidden_states 8 and hidden_states 16 is 0.7395 +The cosine similarity between hidden_states 9 and hidden_states 17 is 0.7389 +The cosine similarity between hidden_states 10 and hidden_states 18 is 0.7375 +The cosine similarity between hidden_states 11 and hidden_states 19 is 0.7301 +The cosine similarity between hidden_states 12 and hidden_states 20 is 0.7248 +The cosine similarity between hidden_states 13 and hidden_states 21 is 0.7307 +The cosine similarity between hidden_states 14 and hidden_states 22 is 0.7426 +The cosine similarity between hidden_states 15 and hidden_states 23 is 0.7514 +The cosine similarity between hidden_states 16 and hidden_states 24 is 0.7678 +The cosine similarity between hidden_states 17 and hidden_states 25 is 0.7862 +The cosine similarity between hidden_states 18 and hidden_states 26 is 0.8082 +The cosine similarity between hidden_states 19 and hidden_states 27 is 0.8172 +The cosine similarity between hidden_states 20 and hidden_states 28 is 0.8261 +The cosine similarity between hidden_states 21 and hidden_states 29 is 0.8217 +The cosine similarity between hidden_states 22 and hidden_states 30 is 0.8275 +The cosine similarity between hidden_states 23 and hidden_states 31 is 0.7997 +The cosine similarity between hidden_states 24 and hidden_states 32 is 0.6477 +The highest cosine similarity comes from hidden_states 22 and hidden_states 30, with a value of 0.8275 + + +pruning_layer: [22, 29] diff --git a/llm-streamline/slurm_output/search-pruning-layer_opt-6.7b.out b/llm-streamline/slurm_output/search-pruning-layer_opt-6.7b.out new file mode 100644 index 0000000..0f97dc6 --- /dev/null +++ b/llm-streamline/slurm_output/search-pruning-layer_opt-6.7b.out @@ -0,0 +1,30 @@ +Calculating cosine similarity... +The cosine similarity between hidden_states 0 and hidden_states 8 is 0.5186 +The cosine similarity between hidden_states 1 and hidden_states 9 is 0.9670 +The cosine similarity between hidden_states 2 and hidden_states 10 is 0.9804 +The cosine similarity between hidden_states 3 and hidden_states 11 is 0.9778 +The cosine similarity between hidden_states 4 and hidden_states 12 is 0.9740 +The cosine similarity between hidden_states 5 and hidden_states 13 is 0.9693 +The cosine similarity between hidden_states 6 and hidden_states 14 is 0.9637 +The cosine similarity between hidden_states 7 and hidden_states 15 is 0.9572 +The cosine similarity between hidden_states 8 and hidden_states 16 is 0.9492 +The cosine similarity between hidden_states 9 and hidden_states 17 is 0.9388 +The cosine similarity between hidden_states 10 and hidden_states 18 is 0.9272 +The cosine similarity between hidden_states 11 and hidden_states 19 is 0.9158 +The cosine similarity between hidden_states 12 and hidden_states 20 is 0.9019 +The cosine similarity between hidden_states 13 and hidden_states 21 is 0.8858 +The cosine similarity between hidden_states 14 and hidden_states 22 is 0.8695 +The cosine similarity between hidden_states 15 and hidden_states 23 is 0.8523 +The cosine similarity between hidden_states 16 and hidden_states 24 is 0.8311 +The cosine similarity between hidden_states 17 and hidden_states 25 is 0.8122 +The cosine similarity between hidden_states 18 and hidden_states 26 is 0.8014 +The cosine similarity between hidden_states 19 and hidden_states 27 is 0.7926 +The cosine similarity between hidden_states 20 and hidden_states 28 is 0.7911 +The cosine similarity between hidden_states 21 and hidden_states 29 is 0.7943 +The cosine similarity between hidden_states 22 and hidden_states 30 is 0.7960 +The cosine similarity between hidden_states 23 and hidden_states 31 is 0.7970 +The cosine similarity between hidden_states 24 and hidden_states 32 is 0.8019 +The highest cosine similarity comes from hidden_states 2 and hidden_states 10, with a value of 0.9804 + + +pruning_layer: [2, 9] diff --git a/llm-streamline/slurm_script_benchmark.sh b/llm-streamline/slurm_script_benchmark.sh new file mode 100644 index 0000000..93eea14 --- /dev/null +++ b/llm-streamline/slurm_script_benchmark.sh @@ -0,0 +1,131 @@ +#!/usr/bin/env bash + +# ======== Slurm setting +#SBATCH -J benchmark-kmmlu +#SBATCH --output=benchmark-kmmlu_%j.out +#SBATCH --error=benchmark-kmmlu_%j.err +#SBATCH --time=4-00:00:00 +#SBATCH -D /dss/dsshome1/07/di54rur/kim_he/LLM-VLM-in-Jetson/llm-streamline + +# ======== General resourece setting +#SBATCH --mem=512gb +#SBATCH --clusters=hpda2 +#SBATCH --partition=hpda2_compute_gpu +#SBATCH --cpus-per-task=4 + +# ======== GPU resource setting +#SBATCH --gres=gpu:1 + +# load required modules +module load slurm_setup +eval "$(micromamba shell hook --shell bash)" +micromamba activate llm-streamline +export PYTHONNOUSERSITE=1 +export HYDRA_FULL_ERROR=1 +export HF_HOME=/dss/dsstbyfs02/scratch/07/di54rur/.huggingface + + +cd /dss/dsshome1/07/di54rur/kim_he/LLM-VLM-in-Jetson/llm-streamline + +# Each row of this table is one benchmark.py run; the per-subject CSV lets a +# requeued job resume mid-model instead of re-scoring finished subjects, and +# every run appends one row to /kmmlu_summary.csv. +# checkpoint dir can be a local pruned checkpoint OR a plain HF Hub id (e.g. +# facebook/opt-6.7b for the unpruned baseline) — from_pretrained downloads and +# caches Hub ids under $HF_HOME automatically, same as the other scripts do. + + # checkpoint dir | label | replace_type +# declare -a RUNS=( +# "facebook/opt-6.7b| opt_6.7b| baseline" +# "/dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_none| opt_prune_from2to9_none| none" +# "/dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp| opt_prune_from2to9_mlp| mlp" +# "/dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf| opt_prune_from2to9_tf| tf" +# ) + +# for run in "${RUNS[@]}"; do +# IFS="|" read -r ckpt_dir label replace_type <<< "$run" +# echo "=== Benchmarking ${label} (${replace_type}) : ${ckpt_dir} ===" +# python benchmark.py \ +# --num_fewshot 1 \ +# --model_name "${ckpt_dir}" \ +# --model_label "${label}" \ +# --replace_type "${replace_type}" \ +# --output_dir /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/benchmark_results +# done + + +# declare -a RUNS=( +# "meta-llama/Llama-2-7b-hf| llama_2_7b| baseline" +# "XiaodongChen/Llama-2-4.7B| llama_2_prune_(none)_tf| tf" +# ) + +# for run in "${RUNS[@]}"; do +# IFS="|" read -r ckpt_dir label replace_type <<< "$run" +# echo "=== Benchmarking ${label} (${replace_type}) : ${ckpt_dir} ===" +# python benchmark.py \ +# --num_fewshot 5 \ +# --model_name "${ckpt_dir}" \ +# --model_label "${label}" \ +# --replace_type "${replace_type}" \ +# --output_dir /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/benchmark_results +# done + +# declare -a RUNS=( +# "meta-llama/Llama-3.1-8B| llama_3_8b| baseline" +# "XiaodongChen/Llama-3.1-5.4B| llama_3_prune_(none)_tf| tf" +# ) + +# for run in "${RUNS[@]}"; do +# IFS="|" read -r ckpt_dir label replace_type <<< "$run" +# echo "=== Benchmarking ${label} (${replace_type}) : ${ckpt_dir} ===" +# python benchmark.py \ +# --num_fewshot 5 \ +# --model_name "${ckpt_dir}" \ +# --model_label "${label}" \ +# --replace_type "${replace_type}" \ +# --output_dir /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/benchmark_results +# done + + + + +# ------------------------------------------ +# VRAM Checking with a few examples +# ------------------------------------------ +declare -a RUNS=( + "facebook/opt-6.7b| opt_6.7b| baseline" + "/dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_none| opt_prune_from2to9_none| none" + "/dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_mlp| opt_prune_from2to9_mlp| mlp" + "/dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned/opt_prune_from2to9_tf| opt_prune_from2to9_tf| tf" +) + +for run in "${RUNS[@]}"; do + IFS="|" read -r ckpt_dir label replace_type <<< "$run" + echo "=== Benchmarking ${label} (${replace_type}) : ${ckpt_dir} ===" + python benchmark.py \ + --num_fewshot 1 \ + --max_examples 5 \ + --model_name "${ckpt_dir}" \ + --model_label "${label}_vramcheck" \ + --replace_type "${replace_type}" \ + --output_dir /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/vram_check_results +done + +declare -a RUNS=( + "meta-llama/Llama-2-7b-hf| llama_2_7b| baseline" + "XiaodongChen/Llama-2-4.7B| llama_2_prune_(none)_tf| tf" + "meta-llama/Llama-3.1-8B| llama_3_8b| baseline" + "XiaodongChen/Llama-3.1-5.4B| llama_3_prune_(none)_tf| tf" +) + +for run in "${RUNS[@]}"; do + IFS="|" read -r ckpt_dir label replace_type <<< "$run" + echo "=== Benchmarking ${label} (${replace_type}) : ${ckpt_dir} ===" + python benchmark.py \ + --num_fewshot 5 \ + --max_examples 5 \ + --model_name "${ckpt_dir}" \ + --model_label "${label}_vramcheck" \ + --replace_type "${replace_type}" \ + --output_dir /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/vram_check_results +done \ No newline at end of file diff --git a/llm-streamline/slurm_script_replace_and_retrain.sh b/llm-streamline/slurm_script_replace_and_retrain.sh index e04625e..914ee9a 100644 --- a/llm-streamline/slurm_script_replace_and_retrain.sh +++ b/llm-streamline/slurm_script_replace_and_retrain.sh @@ -4,7 +4,7 @@ #SBATCH -J replace-and-retrain #SBATCH --output=replace-and-retrain_%j.out #SBATCH --error=replace-and-retrain_%j.err -#SBATCH --time=2-00:00:00 +#SBATCH --time=4-00:00:00 #SBATCH -D /dss/dsshome1/07/di54rur/kim_he/LLM-VLM-in-Jetson/llm-streamline # ======== General resourece setting @@ -24,22 +24,42 @@ export PYTHONNOUSERSITE=1 export MASTER_PORT=$((12000 + SLURM_JOB_ID % 20000)) export HYDRA_FULL_ERROR=1 export HF_HOME=/dss/dsstbyfs02/scratch/07/di54rur/.huggingface - +export HF_HUB_ETAG_TIMEOUT=60 cd /dss/dsshome1/07/di54rur/kim_he/LLM-VLM-in-Jetson/llm-streamline # See args.py: ModelArguments + TrainingArguments. # --pruning_start_layer/--pruning_end_layer come from search_pruning_layer.py's # printed "pruning_layer: [start, end]" output — update these before submitting. -python replace_and_retrain.py \ - --model_name facebook/opt-6.7b \ - --model opt \ - --replace none \ - --pruning_start_layer 2 \ - --pruning_end_layer 9 \ - --output_dir /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned +# model_name | model | pruning_start_layer | pruning_end_layer +# facebook/opt-6.7b | opt | 2 | 9 +# meta-llama/Llama-3.1-8B | llama | 22 | 29 +# meta-llama/Llama-2-7b-hf | llama | 21 | 28 + +# python replace_and_retrain.py \ +# --model_name facebook/opt-6.7b \ +# --model opt \ +# --replace tf \ +# --pruning_start_layer 2 \ +# --pruning_end_layer 9 \ +# --output_dir /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/opt-6b-pruned \ +# --patience 5 \ + +python replace_and_retrain.py \ + --model_name meta-llama/Llama-3.1-8B \ + --model llama \ + --replace mlp \ + --pruning_start_layer 22 \ + --pruning_end_layer 29 \ + --output_dir /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/llama-3.1-8B-pruned \ + --patience 5 \ -# 12574070 -> mlp -# 12574039 -> tf -# 12574071 -> none \ No newline at end of file +# python replace_and_retrain.py \ +# --model_name meta-llama/Llama-2-7b-hf \ +# --model llama \ +# --replace tf \ +# --pruning_start_layer 21 \ +# --pruning_end_layer 28 \ +# --output_dir /dss/dsstbyfs02/scratch/07/di54rur/pseudolab/llama-2-7b-hf-pruned \ +# --patience 5 \ \ No newline at end of file diff --git a/llm-streamline/slurm_script_search_pruning_layer.sh b/llm-streamline/slurm_script_search_pruning_layer.sh index d58a05e..de50b04 100644 --- a/llm-streamline/slurm_script_search_pruning_layer.sh +++ b/llm-streamline/slurm_script_search_pruning_layer.sh @@ -5,7 +5,7 @@ #SBATCH --output=search-pruning-layer_%j.out #SBATCH --error=search-pruning-layer_%j.err #SBATCH --time=12:00:00 -#SBATCH -D /dss/dsshome1/07/di54rur/kim_he/LLM-VLM-in-Jetson +#SBATCH -D /dss/dsshome1/07/di54rur/kim_he/LLM-VLM-in-Jetson/llm-streamline # ======== General resourece setting #SBATCH --mem=512gb @@ -24,12 +24,13 @@ export PYTHONNOUSERSITE=1 export MASTER_PORT=$((12000 + SLURM_JOB_ID % 20000)) export HYDRA_FULL_ERROR=1 export HF_HOME=/dss/dsstbyfs02/scratch/07/di54rur/.huggingface +export HF_HUB_ETAG_TIMEOUT=60 -cd /dss/dsshome1/07/di54rur/kim_he/LLM-VLM-in-Jetson +cd /dss/dsshome1/07/di54rur/kim_he/LLM-VLM-in-Jetson/llm-streamline # See args.py: ModelArguments + SearchArguments (no --output_dir here — this # script only searches for the pruning range, it doesn't save a model). python search_pruning_layer.py \ - --model_name facebook/opt-6.7b \ + --model_name meta-llama/Llama-3.1-8B \ --layer_intervals 8 \ No newline at end of file