| Model Evaluation Harness | LAMBADA | lambada_openai_mt | StoryCloze16 | Hellaswag | WinoGrande | ARC_challenge | ARC_easy | HeadQA_en | OpenbookQA | SciQ | ReCoRD | COPA | XStoryCloze | XNLI | truthfulqa | race | pubmedqa | piqa | mc_taco | logiqa | xwinograd | xcopa | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Model | acc | de/ppl | German(de/acc) | es/ppl | Spanish(es/acc) | fr/ppl | French(fr/acc) | it/ppl | it/acc | acc | acc | acc_norm | acc | acc | acc_norm | acc | acc_norm | acc | acc_norm | acc | acc_norm | acc | acc_norm | f1 | em | acc | Arabic(ar/acc) | English(en/acc) | Spanish(es/acc) | Basque(eu/acc) | Hindi(hi/acc) | Indonesian(id/acc) | Burmese(my/acc) | Russian(ru/acc) | Swahili(sw/acc) | te/ac | Chinese(zh/acc) | Arabic(ar/acc) | Bulgarian(bg/acc) | German(de/acc) | Greek(el/acc) | English(en/acc) | Spanish(es/acc) | French(fr/acc) | Hindi(hi/acc) | Russian(ru/acc) | Swahili(sw/acc) | Thai(th/acc) | Turkish(tr/acc) | Urdu(ur/acc) | Vietnamese(vi/acc) | Chinese(zh/acc) | mc1 | mc2 | acc | acc | acc | acc_norm | em | f1 | acc | acc_norm | English(en/acc) | French(fr/acc) | jp/acc | pt/acc | Russian(ru/acc) | Chinese(zh/acc) | et/acc | ht/acc | Indonesian(id/acc) | it/acc | qu/acc | Swahili(sw/acc) | ta/acc | Thai(th/acc) | Turkish(tr/acc) | Vietnamese(vi/acc) | Chinese(zh/acc) | ||
- | bloom-1b7 | 12.5841 | 46.2% | 1406.40 | 15.7% | 93.78 | 31.0% | 56.98 | 37.6% | 691.11 | 22.7% | 65.8% | 37.5% | 48.0% | 56.9% | 23.6% | 26.9% | 56.3% | 48.1% | 27.6% | 32.6% | 21.4% | 30.0% | 85.1% | 77.0% | 78.4% | 77.6% | 70.0% | 54.9% | 64.5% | 60.8% | 54.9% | 56.9% | 59.8% | 47.3% | 50.4% | 52.3% | - | 58.3% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 24.5% | 41.3% | 33.3% | 56.1% | 68.7% | 69.8% | 12.4% | 47.3% | 21.7% | 28.1% | 74.5% | 68.7% | 54.2% | 63.9% | 53.0% | 69.3% | 47.4% | 50.4% | 63.2% | 52.6% | 50.8% | 51.8% | 56.4% | 53.2% | 53.0% | 65.8% | 61.4% |
- | bloomz-1b7 | 11.7537 | 48.2% | 2487.84 | 14.7% | 74.65 | 34.9% | 56.49 | 38.4% | 1014.28 | 20.7% | 68.2% | 38.5% | 49.9% | 57.5% | 31.3% | 33.9% | 64.1% | 62.2% | 27.4% | 31.2% | 27.6% | 34.0% | 96.2% | 96.3% | 89.6% | 89.1% | 69.0% | 54.3% | 66.5% | 57.5% | 49.4% | 54.2% | 53.3% | 46.3% | 48.3% | 50.0% | - | 58.2% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 21.4% | 40.1% | 39.7% | 62.0% | 70.3% | 70.7% | 22.5% | 34.8% | 21.0% | 28.0% | 76.7% | 61.5% | 55.2% | 58.2% | 51.1% | 67.5% | 49.4% | 52.8% | 59.8% | 52.0% | 50.8% | 51.4% | 55.2% | 53.6% | 51.8% | 60.8% | 57.8% |
1.1 | TinyLlama-1.1B-Chat-v0.3 | 10.0697 | 51.0% | 424.54 | 22.6% | 443.39 | 23.0% | 206.48 | 30.4% | 460.25 | 24.8% | 71.3% | 43.9% | 56.8% | 55.8% | 27.6% | 30.0% | 56.8% | 53.2% | 27.9% | 31.6% | 24.4% | 34.2% | 85.7% | 78.7% | 78.7% | 77.8% | 75.0% | 47.9% | 70.3% | 56.1% | 49.7% | 50.9% | 49.8% | 48.2% | 54.7% | 50.2% | - | 54.3% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 22.5% | 36.8% | 34.1% | 57.8% | 70.6% | 69.9% | 12.1% | 49.7% | 22.1% | 27.3% | 74.0% | 51.8% | 51.9% | 55.9% | 57.5% | 59.1% | 48.2% | 53.4% | 53.4% | 55.2% | 51.6% | 50.6% | 52.4% | 53.4% | 51.4% | 51.6% | 54.6% |
1.1 | TinyLlama-1.1B-intermediate-step-480k-1T | 9.9989 | 53.0% | 257.69 | 22.3% | 239.24 | 25.1% | 146.56 | 30.2% | 271.78 | 24.7% | 68.8% | 40.4% | 52.5% | 56.0% | 23.7% | 27.8% | 58.0% | 52.4% | 26.4% | 30.9% | 20.4% | 33.4% | 86.0% | 78.2% | 81.7% | 80.9% | 71.0% | 48.0% | 67.6% | 53.8% | 49.8% | 50.6% | 48.8% | 48.0% | 51.6% | 48.9% | - | 52.5% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 23.3% | 39.4% | 33.1% | 59.8% | 70.6% | 69.9% | 12.1% | 49.8% | 20.1% | 26.7% | 76.6% | 63.9% | 53.9% | 60.1% | 58.1% | 63.9% | 47.2% | 51.8% | 52.4% | 55.8% | 51.4% | 53.4% | 54.8% | 54.2% | 52.0% | 50.6% | 55.4% |
- | bloomz-3b | 9.5723 | 50.1% | 1615.57 | 17.2% | 72.64 | 36.0% | 70.43 | 37.0% | 495.05 | 25.7% | 70.0% | 42.6% | 57.2% | 60.5% | 36.8% | 38.4% | 68.6% | 66.7% | 29.4% | 33.4% | 30.2% | 38.8% | 96.7% | 96.6% | 92.9% | 92.5% | 79.0% | 53.3% | 68.6% | 57.4% | 49.1% | 53.3% | 52.1% | 47.3% | 49.1% | 50.6% | - | 55.7% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 23.4% | 40.4% | 46.0% | 66.3% | 73.3% | 74.7% | 28.6% | 48.8% | 26.0% | 28.9% | 80.6% | 55.4% | 55.5% | 62.0% | 51.4% | 69.0% | 48.4% | 50.4% | 59.8% | 54.2% | 51.2% | 50.0% | 56.8% | 55.6% | 52.2% | 57.2% | 55.8% |
- | bloom-3b | 9.0955 | 51.7% | 709.58 | 19.2% | 67.43 | 34.6% | 41.93 | 41.4% | 359.32 | 26.5% | 68.7% | 41.4% | 54.5% | 58.5% | 28.0% | 30.4% | 59.5% | 53.2% | 28.4% | 33.3% | 21.8% | 32.2% | 89.1% | 81.7% | 81.4% | 80.6% | 74.0% | 56.6% | 66.7% | 64.1% | 55.7% | 57.5% | 61.0% | 46.7% | 50.7% | 53.0% | - | 60.9% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 23.1% | 40.6% | 35.2% | 61.8% | 70.6% | 70.7% | 11.9% | 49.6% | 20.7% | 29.2% | 79.2% | 71.1% | 56.4% | 70.3% | 53.6% | 73.4% | 49.2% | 50.2% | 69.2% | 51.6% | 50.6% | 51.4% | 58.0% | 52.8% | 53.4% | 68.8% | 62.0% |
- | phi-1_5 | 8.9196 | 53.4% | 25729.53 | 7.0% | 2899.89 | 14.7% | 2453.60 | 16.4% | 13814.75 | 11.1% | 78.0% | 48.0% | 62.6% | 72.7% | 44.6% | 48.0% | 76.2% | 73.1% | 28.1% | 33.0% | 38.6% | 48.0% | 93.3% | 91.5% | 74.1% | 73.3% | 83.0% | 47.5% | 78.2% | 51.7% | 50.3% | 47.1% | 47.3% | 47.9% | 47.9% | 49.3% | - | 49.8% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 27.1% | 40.9% | 37.9% | 67.3% | 76.4% | 75.6% | 11.7% | 48.9% | 24.0% | 29.6% | 83.3% | 50.6% | 50.2% | 56.3% | 54.6% | 50.6% | 50.8% | 49.6% | 52.2% | 52.4% | 50.6% | 54.0% | 54.8% | 55.0% | 51.8% | 52.0% | 51.2% |
1.4 | gpt-neo-1.3B | 7.4978 | 57.2% | 229.41 | 23.0% | 232.30 | 25.0% | 144.36 | 29.7% | 242.05 | 26.2% | 67.7% | 38.7% | 48.9% | 54.9% | 23.1% | 25.9% | 56.2% | 50.2% | 28.4% | 32.5% | 21.4% | 33.6% | 86.0% | 77.1% | 81.4% | 80.6% | 69.0% | 47.7% | 65.5% | 51.5% | 50.4% | 49.0% | 48.7% | 48.2% | 47.8% | 49.8% | - | 48.5% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 23.1% | 39.6% | 34.1% | 54.4% | 71.1% | 71.1% | 13.1% | 49.3% | 19.7% | 27.3% | 73.6% | 51.8% | 51.4% | 58.6% | 58.4% | 58.9% | 49.0% | 49.8% | 52.4% | 53.4% | 51.0% | 51.2% | 54.4% | 58.2% | 55.0% | 54.2% | 52.0% |
1.3 | falcon-rw-1b | 7.3651 | 55.0% | 957.88 | 13.7% | 933.42 | 16.0% | 416.90 | 20.6% | 996.07 | 15.2% | 71.4% | 46.7% | 61.6% | 61.0% | 28.6% | 32.3% | 63.2% | 57.0% | 28.3% | 33.3% | 24.8% | 35.0% | 89.9% | 84.8% | 85.7% | 84.9% | 82.0% | 47.4% | 69.6% | 51.5% | 49.6% | 47.6% | 49.6% | 46.6% | 47.9% | 49.0% | - | 51.4% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 21.4% | 35.9% | 35.7% | 62.3% | 75.2% | 74.7% | 12.7% | 49.6% | 20.6% | 27.5% | 81.1% | 53.0% | 49.6% | 55.5% | 51.4% | 59.7% | 49.8% | 50.0% | 53.6% | 52.4% | 48.6% | 52.0% | 56.4% | 54.2% | 54.8% | 51.0% | 52.8% |
- | mpt-1b-redpajama-200b | 7.2239 | 58.0% | 207.27 | 23.0% | 202.55 | 26.5% | 118.64 | 31.9% | 181.09 | 28.7% | 69.3% | 41.5% | 53.2% | 57.3% | 26.9% | 29.1% | 60.7% | 55.5% | 27.2% | 32.1% | 23.8% | 33.4% | 85.4% | 78.8% | 84.0% | 83.1% | 75.0% | 47.1% | 66.1% | 53.5% | 49.6% | 49.5% | 49.4% | 47.7% | 49.8% | 50.0% | - | 49.6% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 22.5% | 38.7% | 34.6% | 56.4% | 71.4% | 72.0% | 12.0% | 49.5% | 21.7% | 27.8% | 79.0% | 62.6% | 50.2% | 58.6% | 57.5% | 56.9% | 50.6% | 51.8% | 53.0% | 53.4% | 50.8% | 53.4% | 55.6% | 52.6% | 53.8% | 51.4% | 52.8% |
- | RWKV-4-Pile-1B5-20220903-8040 | 6.9143 | 57.4% | 133.19 | 27.1% | 148.84 | 28.3% | 93.55 | 33.7% | 146.93 | 29.4% | 68.0% | 41.0% | 52.8% | 54.7% | 24.8% | 29.2% | 60.6% | 54.9% | 30.2% | 34.8% | 23.2% | 34.0% | 84.2% | 77.4% | 77.8% | 77.0% | 77.0% | 47.9% | 66.8% | 55.7% | 50.6% | 49.1% | 50.7% | 48.8% | 51.6% | 49.8% | - | 51.2% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 21.4% | 36.1% | 32.0% | 57.9% | 72.1% | 71.9% | 16.5% | 39.8% | 22.0% | 27.3% | 77.2% | 56.6% | 54.6% | 61.6% | 54.9% | 57.7% | 50.2% | 51.2% | 53.8% | 55.6% | 51.2% | 51.8% | 54.6% | 55.0% | 55.0% | 53.4% | 57.0% |
1.4 | pythia-1.4b-v0 | 6.5837 | 60.4% | 178.66 | 24.0% | 179.82 | 26.5% | 111.68 | 31.7% | 183.88 | 28.4% | 67.7% | 39.8% | 50.8% | 57.0% | 25.5% | 28.6% | 57.7% | 52.0% | 29.3% | 33.3% | 20.2% | 31.0% | 85.5% | 77.8% | 82.2% | 81.4% | 73.0% | 47.6% | 65.0% | 53.9% | 50.8% | 49.3% | 49.6% | 48.6% | 49.2% | 48.4% | - | 49.8% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 23.9% | 39.2% | 34.8% | 60.4% | 71.2% | 71.1% | 14.6% | 48.1% | 22.0% | 30.3% | 75.7% | 61.5% | 51.7% | 58.2% | 54.3% | 59.7% | 49.4% | 50.8% | 53.6% | 53.8% | 50.4% | 53.8% | 52.6% | 54.8% | 52.0% | 50.2% | 52.6% |
- | RWKV-5-World-1.5B-v2-OnlyForTest_14%_trained-20231001-ctx4096 | 6.3977 | 60.2% | 86.12 | 32.5% | 79.72 | 34.3% | 45.44 | 41.8% | 67.84 | 38.4% | 69.9% | 38.2% | 48.5% | 57.2% | 27.1% | 29.6% | 60.9% | 56.5% | 29.2% | 32.4% | 21.2% | 33.2% | 88.2% | 82.7% | 80.3% | 79.5% | 75.0% | 51.6% | 68.6% | 60.1% | 51.9% | 52.6% | 58.4% | 49.8% | 58.4% | 51.6% | - | 57.5% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 25.3% | 40.2% | 33.9% | 58.0% | 69.1% | 69.6% | 12.4% | 44.8% | 22.1% | 28.4% | 79.6% | 65.1% | 56.8% | 63.9% | 61.0% | 72.0% | 52.0% | 51.8% | 63.0% | 60.2% | 49.8% | 51.6% | 55.4% | 55.2% | 55.6% | 58.2% | 61.4% |
1.3 | 42dot_LLM-PLM-1.3B | 6.2828 | 61.5% | 206.87 | 24.4% | 167.92 | 29.1% | 99.08 | 35.0% | 149.34 | 31.3% | 69.9% | 42.2% | 54.4% | 58.3% | 28.8% | 31.7% | 62.7% | 56.4% | 29.8% | 34.4% | 22.2% | 34.0% | 88.9% | 82.3% | 84.7% | 83.9% | 73.0% | 47.6% | 67.5% | 54.6% | 50.5% | 50.4% | 48.8% | 46.3% | 52.0% | 48.2% | - | 50.0% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 23.6% | 38.7% | 35.7% | 67.4% | 72.5% | 72.7% | 12.1% | 49.7% | 19.4% | 26.1% | 80.5% | 62.6% | 60.5% | 65.0% | 59.7% | 56.9% | 50.8% | 50.8% | 53.2% | 56.0% | 51.4% | 54.2% | 56.4% | 55.0% | 53.2% | 50.6% | 52.6% |
- | RWKV-5-World-1.5B-v2-OnlyForTest_21%_trained-20231003-ctx4096 | 6.1662 | 61.8% | 75.23 | 34.2% | 77.15 | 34.5% | 43.13 | 42.0% | 65.33 | 38.5% | 71.2% | 39.7% | 50.8% | 58.9% | 27.5% | 30.3% | 60.9% | 56.4% | 28.7% | 33.5% | 22.8% | 33.8% | 89.8% | 86.0% | 80.9% | 80.0% | 76.0% | 51.0% | 70.2% | 61.0% | 52.4% | 52.8% | 59.0% | 49.4% | 59.0% | 51.2% | - | 57.5% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 22.9% | 38.2% | 33.7% | 52.6% | 69.7% | 71.1% | 13.2% | 41.6% | 21.8% | 28.7% | 79.9% | 65.1% | 58.1% | 67.7% | 64.1% | 72.6% | 52.8% | 50.0% | 63.0% | 62.2% | 50.4% | 53.4% | 54.8% | 55.8% | 53.8% | 60.0% | 63.2% |
- | RWKV-4-World-1.5B-v1-fixed-20230612-ctx4096 | 6.1382 | 60.1% | 97.03 | 31.1% | 93.71 | 32.0% | 60.94 | 37.1% | 106.14 | 32.2% | 70.9% | 40.2% | 51.6% | 56.8% | 27.1% | 30.1% | 59.9% | 56.1% | 27.0% | 31.8% | 22.6% | 35.2% | 85.1% | 79.6% | 78.3% | 77.5% | 76.0% | 50.6% | 70.1% | 61.4% | 53.0% | 53.4% | 58.4% | 49.2% | 57.5% | 51.2% | - | 58.9% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 20.2% | 34.7% | 33.2% | 60.4% | 71.4% | 71.1% | 18.5% | 38.7% | 25.2% | 27.8% | 77.0% | 63.9% | 57.1% | 66.2% | 59.1% | 64.7% | 52.8% | 50.2% | 61.2% | 58.4% | 50.6% | 52.6% | 53.6% | 53.4% | 52.0% | 59.0% | 59.8% |
- | RWKV-5-World-1.5B-v2-OnlyForTest_30%_trained-20231005-ctx4096 | 5.9622 | 62.0% | 76.58 | 32.7% | 74.77 | 34.6% | 41.91 | 42.6% | 61.86 | 38.6% | 71.0% | 40.3% | 51.9% | 59.1% | 29.0% | 31.4% | 62.9% | 59.7% | 30.1% | 33.8% | 24.4% | 35.2% | 89.4% | 85.7% | 81.7% | 80.9% | 75.0% | 51.6% | 70.7% | 62.0% | 52.3% | 53.2% | 59.6% | 48.4% | 59.3% | 51.0% | - | 58.0% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 22.5% | 37.8% | 33.4% | 59.0% | 70.4% | 70.3% | 14.1% | 46.3% | 20.4% | 28.6% | 80.2% | 66.3% | 61.5% | 66.2% | 62.9% | 71.2% | 53.6% | 50.8% | 63.8% | 61.8% | 51.2% | 55.0% | 54.6% | 54.6% | 53.4% | 59.6% | 63.8% |
1.3 | 42dot_LLM-SFT-1.3B | 5.8827 | 60.6% | 244.45 | 24.7% | 193.85 | 27.8% | 96.07 | 37.3% | 158.04 | 33.1% | 74.4% | 45.4% | 59.7% | 59.4% | 31.7% | 34.5% | 66.2% | 60.6% | 31.1% | 35.1% | 27.6% | 36.6% | 92.2% | 88.1% | 83.7% | 82.9% | 75.0% | 48.8% | 73.5% | 57.7% | 50.8% | 50.6% | 48.8% | 47.4% | 55.1% | 48.8% | - | 52.6% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 24.0% | 40.2% | 37.8% | 63.9% | 73.6% | 73.9% | 15.5% | 48.3% | 22.3% | 28.1% | 81.2% | 59.0% | 59.2% | 59.3% | 60.6% | 57.5% | 48.6% | 50.4% | 54.2% | 53.6% | 51.6% | 53.0% | 55.2% | 52.6% | 54.8% | 49.6% | 54.8% |
3.4 | long_llama_3b | 5.6298 | 63.0% | 154.88 | 27.1% | 122.79 | 30.7% | 89.95 | 36.0% | 150.98 | 31.6% | 73.8% | 47.8% | 62.6% | 59.4% | 32.2% | 35.1% | 66.6% | 60.9% | 29.8% | 35.5% | 24.6% | 37.0% | 90.7% | 86.0% | 87.1% | 86.4% | 82.0% | 47.6% | 72.2% | 59.0% | 51.3% | 50.0% | 50.4% | 47.0% | 52.3% | 49.7% | - | 48.5% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 24.0% | 37.4% | 37.0% | 70.3% | 73.0% | 73.9% | 11.9% | 49.1% | 20.6% | 26.3% | 81.9% | 62.6% | 53.5% | 57.8% | 61.9% | 61.5% | 48.6% | 50.8% | 52.8% | 56.2% | 50.2% | 52.2% | 56.2% | 53.6% | 53.2% | 51.8% | 53.8% |
2.8 | gpt-neo-2.7B | 5.6257 | 62.2% | 150.85 | 26.9% | 160.46 | 28.6% | 102.32 | 32.6% | 164.89 | 29.0% | 69.5% | 42.7% | 55.8% | 57.7% | 27.5% | 30.2% | 61.1% | 53.9% | 30.7% | 33.9% | 23.2% | 33.2% | 89.3% | 82.8% | 84.5% | 83.8% | 79.0% | 48.4% | 66.9% | 54.5% | 51.8% | 49.6% | 49.8% | 48.4% | 48.0% | 49.4% | - | 49.5% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 23.9% | 39.9% | 35.2% | 56.5% | 72.2% | 72.9% | 12.0% | 49.5% | 20.3% | 28.4% | 76.0% | 61.5% | 52.3% | 63.1% | 56.5% | 55.2% | 51.0% | 51.2% | 53.4% | 51.4% | 50.2% | 54.4% | 56.4% | 57.4% | 54.2% | 52.8% | 54.8% |
- | RWKV-4-Pile-3B-20221110-ctx4096 | 5.2537 | 64.1% | 109.34 | 28.9% | 108.93 | 30.2% | 73.32 | 36.4% | 108.54 | 33.3% | 70.8% | 45.4% | 59.9% | 59.8% | 30.6% | 33.0% | 65.1% | 59.7% | 32.1% | 37.2% | 24.8% | 36.8% | 86.3% | 80.1% | 81.9% | 81.1% | 82.0% | 47.7% | 70.0% | 59.1% | 51.7% | 50.4% | 51.7% | 49.8% | 54.2% | 49.4% | - | 53.8% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 19.8% | 32.2% | 33.7% | 58.7% | 74.2% | 74.6% | 13.4% | 48.8% | 21.5% | 27.5% | 79.8% | 60.2% | 57.1% | 55.9% | 56.2% | 58.9% | 51.0% | 50.2% | 57.4% | 59.0% | 52.0% | 54.8% | 53.2% | 56.0% | 52.8% | 54.4% | 56.8% |
- | RWKV-5-World-3B-v2-OnlyForTest_14%_trained-20231006-ctx4096 | 5.0137 | 64.7% | 60.51 | 37.1% | 60.60 | 37.3% | 33.87 | 45.7% | 48.39 | 42.4% | 72.8% | 42.0% | 55.4% | 60.2% | 29.4% | 33.9% | 65.1% | 60.9% | 30.1% | 35.3% | 25.0% | 36.2% | 89.6% | 84.4% | 83.1% | 82.3% | 79.0% | 53.5% | 72.0% | 63.1% | 51.7% | 54.0% | 60.6% | 48.0% | 61.4% | 50.4% | - | 58.7% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 22.5% | 37.1% | 33.2% | 63.8% | 71.5% | 71.5% | 12.2% | 43.1% | 21.8% | 28.0% | 81.5% | 68.7% | 63.4% | 66.5% | 63.8% | 74.6% | 51.6% | 51.6% | 64.6% | 61.8% | 50.8% | 52.4% | 55.8% | 55.2% | 56.0% | 60.6% | 64.4% |
- | RedPajama-INCITE-Base-3B-v1 | 4.9311 | 65.4% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
2.8 | pythia-2.8b-v0 | 4.9261 | 65.3% | 109.63 | 29.2% | 105.92 | 30.8% | 72.42 | 37.2% | 110.92 | 33.4% | 70.7% | 45.1% | 59.5% | 61.6% | 28.8% | 32.3% | 63.0% | 57.4% | 32.0% | 36.2% | 22.0% | 35.4% | 87.7% | 80.8% | 85.9% | 85.1% | 77.0% | 47.9% | 68.9% | 57.7% | 51.7% | 50.9% | 51.8% | 48.3% | 52.8% | 49.2% | - | 52.7% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 21.2% | 35.4% | 36.4% | 61.6% | 73.8% | 74.2% | 12.2% | 49.7% | 22.6% | 30.7% | 82.1% | 61.5% | 54.3% | 65.4% | 57.8% | 66.3% | 51.0% | 49.2% | 55.4% | 56.4% | 50.0% | 54.4% | 55.0% | 56.6% | 51.6% | 52.8% | 56.4% |
2.7 | btlm-3b-8k-base | 4.7376 | 66.3% | 81.51 | 33.0% | 72.19 | 36.2% | 47.25 | 42.8% | 65.33 | 39.9% | 75.7% | 51.8% | 69.8% | 65.6% | 34.8% | 37.5% | 70.7% | 66.9% | 30.9% | 36.5% | 27.4% | 40.6% | 92.8% | 89.1% | 89.5% | 88.7% | 85.0% | 47.9% | 75.0% | 62.7% | 50.2% | 50.5% | 50.6% | 47.6% | 58.4% | 50.0% | - | 51.7% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 22.6% | 36.2% | 39.4% | 66.3% | 76.0% | 77.3% | 12.8% | 48.9% | 20.7% | 28.0% | 85.0% | 71.1% | 55.9% | 70.7% | 65.4% | 58.3% | 46.6% | 51.6% | 54.8% | 60.6% | 50.2% | 52.4% | 55.6% | 53.8% | 54.8% | 51.8% | 54.4% |
- | RWKV-4-World-3B-v1-20230619-ctx4096 | 4.6784 | 65.9% | 67.32 | 34.1% | 62.62 | 36.0% | 40.83 | 42.3% | 66.44 | 37.9% | 73.9% | 44.4% | 58.7% | 58.3% | 31.7% | 34.3% | 64.6% | 61.4% | 27.9% | 33.0% | 25.4% | 36.6% | 87.6% | 83.2% | 82.4% | 81.6% | 84.0% | 53.5% | 72.7% | 65.6% | 52.0% | 54.8% | 61.8% | 49.9% | 60.6% | 52.8% | - | 60.8% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 21.2% | 35.7% | 33.1% | 64.9% | 73.3% | 73.3% | 12.7% | 40.5% | 23.2% | 27.0% | 81.3% | 61.5% | 60.7% | 63.1% | 58.4% | 67.3% | 55.4% | 51.2% | 66.6% | 63.4% | 49.6% | 54.6% | 56.0% | 57.8% | 55.8% | 60.8% | 63.8% |
3.4 | open_llama_3b_v2 | 4.5609 | 67.4% | 100.61 | 30.8% | 80.57 | 35.6% | 52.07 | 42.2% | 68.95 | 40.5% | 75.6% | 52.2% | 70.0% | 63.0% | 33.6% | 36.4% | 67.8% | 63.3% | 30.1% | 35.0% | 26.2% | 37.8% | 92.5% | 88.0% | 87.9% | 87.1% | 82.0% | 47.6% | 74.0% | 60.6% | 50.7% | 49.2% | 51.6% | 47.3% | 55.5% | 50.3% | - | 56.5% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 21.3% | 34.6% | 37.8% | 66.5% | 77.0% | 77.8% | 13.0% | 50.3% | 21.7% | 28.1% | 83.9% | 57.8% | 55.9% | 65.0% | 62.5% | 66.7% | 48.8% | 50.2% | 55.0% | 57.6% | 49.4% | 51.4% | 56.4% | 55.8% | 55.4% | 52.0% | 59.0% |
- | pythia-6.9b-v0 | 4.2965 | 67.9% | 88.01 | 31.3% | 81.54 | 33.0% | 54.31 | 40.6% | 80.36 | 36.1% | 73.0% | 48.1% | 63.9% | 61.4% | 32.3% | 35.3% | 67.0% | 63.4% | 32.8% | 38.3% | 27.0% | 38.2% | 90.3% | 86.5% | 87.3% | 86.5% | 85.0% | 49.6% | 70.7% | 59.3% | 51.9% | 51.8% | 52.9% | 49.2% | 53.7% | 49.8% | - | 53.9% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 20.1% | 32.9% | 36.1% | 65.3% | 74.5% | 75.8% | 12.8% | 50.2% | 22.0% | 31.2% | 81.7% | 62.6% | 57.7% | 65.8% | 58.7% | 63.3% | 52.8% | 49.8% | 54.2% | 58.4% | 51.4% | 53.4% | 54.2% | 57.6% | 54.0% | 54.6% | 57.8% |
- | Mistral-7B-Instruct-v0.1 | 4.0523 | 69.1% | 89.21 | 33.8% | 74.78 | 35.4% | 43.47 | 43.7% | 67.08 | 40.4% | 78.5% | 56.3% | 74.6% | 69.6% | 50.3% | 52.2% | 80.0% | 76.8% | 37.8% | 42.0% | 32.4% | 42.8% | 95.9% | 94.4% | 89.3% | 88.5% | 89.0% | 48.2% | 78.6% | 66.7% | 49.8% | 50.1% | 53.4% | 47.6% | 63.7% | 48.8% | - | 62.1% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 39.4% | 55.9% | 42.1% | 77.6% | 79.3% | 79.2% | 38.7% | 69.7% | 25.8% | 30.4% | 87.7% | 67.5% | 62.3% | 70.0% | 62.5% | 70.2% | 49.8% | 51.2% | 52.0% | 62.0% | 49.8% | 52.4% | 55.6% | 54.0% | 51.4% | 53.2% | 63.4% |
2.8 | stablelm-base-alpha-3b-v2 | 3.9952 | 70.3% | 79.55 | 33.5% | 77.19 | 35.8% | 46.77 | 44.2% | 68.82 | 39.9% | 75.5% | 51.1% | 68.6% | 62.1% | 32.4% | 35.1% | 67.3% | 63.3% | 30.4% | 35.1% | 26.4% | 38.6% | 92.1% | 86.8% | 88.3% | 87.5% | 80.0% | 49.8% | 72.7% | 61.2% | 51.2% | 51.6% | 55.9% | 50.6% | 56.3% | 50.2% | - | 55.0% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 22.4% | 35.9% | 39.0% | 72.0% | 76.0% | 78.0% | 13.5% | 46.9% | 22.3% | 27.5% | 83.9% | 65.1% | 55.3% | 64.6% | 58.4% | 65.7% | 52.0% | 49.0% | 58.4% | 58.6% | 51.0% | 55.0% | 55.0% | 56.4% | 55.0% | 56.0% | 56.2% |
- | Qwen-7B | 3.9305 | 69.7% | 64.73 | 36.5% | 118.49 | 25.2% | 37.23 | 44.9% | 58.02 | 41.4% | 78.6% | 57.3% | 76.9% | 68.5% | 44.2% | 46.4% | 74.2% | 72.0% | 33.8% | 38.6% | 32.2% | 45.0% | 90.4% | 86.0% | 89.6% | 88.8% | 88.0% | 51.1% | 77.8% | 65.9% | 50.8% | 54.7% | 56.9% | 49.2% | 64.5% | 50.2% | - | 69.6% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 33.1% | 47.9% | 36.0% | 66.8% | 77.4% | 78.2% | 20.3% | 50.5% | 28.6% | 31.9% | 85.7% | 67.5% | 57.5% | 70.3% | 65.4% | 76.4% | 50.2% | 51.8% | 56.8% | 63.0% | 51.0% | 52.8% | 55.8% | 56.2% | 54.8% | 57.8% | 69.6% |
- | RWKV-4-World-7B-v1-20230626-ctx4096 | 3.928 | 70.2% | 49.51 | 37.8% | 43.12 | 39.3% | 27.87 | 47.2% | 43.25 | 41.9% | 75.6% | 49.1% | 65.3% | 62.0% | 33.3% | 36.6% | 68.0% | 65.3% | 30.0% | 35.8% | 29.2% | 39.8% | 91.2% | 86.4% | 84.9% | 84.1% | 86.0% | 56.6% | 76.0% | 68.8% | 55.9% | 57.8% | 65.7% | 52.1% | 64.3% | 54.7% | - | 62.6% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 22.3% | 34.0% | 34.2% | 69.0% | 75.3% | 75.9% | 14.8% | 41.6% | 21.5% | 23.5% | 84.1% | 66.3% | 67.8% | 70.7% | 63.5% | 75.0% | 60.2% | 51.4% | 71.2% | 65.4% | 51.8% | 55.2% | 56.4% | 56.4% | 60.6% | 64.8% | 67.0% |
- | internlm-7b | 3.8277 | 67.5% | 99.85 | 32.1% | 127.78 | 28.6% | 50.76 | 42.5% | 77.26 | 38.5% | 78.5% | 54.6% | 72.5% | 67.8% | 41.0% | 42.6% | 71.4% | 69.9% | 31.1% | 34.8% | 26.6% | 39.8% | 94.1% | 94.4% | 88.3% | 87.5% | 83.0% | 49.2% | 76.8% | 62.3% | 52.0% | 51.2% | 53.5% | 49.0% | 52.6% | 49.0% | - | 65.6% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 26.4% | 39.3% | 39.8% | 69.2% | 73.2% | 73.8% | 11.9% | 39.6% | 24.7% | 28.4% | 83.2% | 63.9% | 59.9% | 68.8% | 59.1% | 76.0% | 49.0% | 50.4% | 54.2% | 56.2% | 49.6% | 52.8% | 53.2% | 56.4% | 52.4% | 54.4% | 69.4% |
2.8 | stablelm-3b-4e1t | 3.8259 | 70.6% | 68.31 | 34.6% | 60.53 | 37.4% | 38.41 | 43.2% | 49.35 | 41.7% | 76.9% | 54.9% | 73.8% | 66.5% | 37.5% | 39.9% | 72.3% | 67.6% | 33.0% | 39.0% | 31.6% | 40.4% | 94.6% | 91.8% | 89.7% | 88.9% | 85.0% | 54.1% | 75.6% | 64.1% | 51.9% | 54.7% | 59.9% | 50.0% | 60.7% | 53.0% | - | 57.6% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 24.1% | 37.2% | 40.3% | 74.3% | 79.6% | 80.0% | 14.8% | 47.5% | 26.3% | 30.6% | 85.4% | 67.5% | 64.1% | 68.1% | 64.1% | 69.6% | 54.4% | 51.0% | 63.0% | 62.6% | 50.8% | 53.2% | 53.6% | 56.0% | 56.2% | 59.2% | 60.4% |
- | open_llama_7b_v2 | 3.8224 | 71.5% | 71.43 | 34.7% | 58.15 | 39.1% | 37.58 | 45.2% | 47.48 | 44.6% | 77.5% | 55.8% | 74.5% | 65.9% | 38.7% | 41.4% | 72.4% | 69.7% | 32.5% | 37.9% | 29.8% | 40.8% | 93.8% | 90.1% | 89.5% | 88.7% | 89.0% | 47.6% | 76.8% | 64.9% | 50.6% | 49.6% | 55.1% | 48.3% | 58.8% | 50.8% | - | 59.4% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 22.6% | 34.6% | 39.7% | 68.2% | 78.7% | 79.9% | 12.2% | 49.7% | 20.4% | 29.6% | 86.2% | 65.1% | 60.3% | 70.7% | 66.3% | 70.6% | 48.6% | 52.8% | 56.6% | 61.6% | 49.8% | 51.0% | 55.8% | 56.2% | 53.8% | 53.0% | 59.8% |
- | Qwen-7B-Chat | 3.7 | 67.9% | 83.02 | 37.3% | 132.22 | 24.7% | 36.86 | 46.8% | 63.24 | 42.2% | 79.8% | 57.5% | 76.2% | 68.0% | 39.3% | 42.8% | 64.4% | 63.1% | 31.6% | 36.1% | 33.4% | 45.0% | 85.4% | 79.4% | 88.1% | 87.3% | 85.0% | 51.4% | 80.2% | 68.4% | 51.2% | 54.4% | 59.4% | 49.4% | 65.1% | 50.7% | - | 71.1% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 35.6% | 51.6% | 36.8% | 60.8% | 76.9% | 78.3% | 33.5% | 64.2% | 28.3% | 32.3% | 86.0% | 65.1% | 54.0% | 68.4% | 68.6% | 77.0% | 49.8% | 54.4% | 57.4% | 64.6% | 50.6% | 53.4% | 56.4% | 56.8% | 54.6% | 58.6% | 71.8% |
- | Baichuan2-7B-Base | 3.6856 | 70.9% | 57.18 | 37.8% | 42.33 | 46.0% | 30.84 | 47.6% | 42.00 | 44.9% | 77.4% | 53.7% | 72.3% | 68.6% | 42.1% | 42.9% | 74.8% | 72.6% | 34.4% | 39.1% | 30.8% | 40.0% | 94.4% | 91.3% | 90.2% | 89.4% | 87.0% | 51.2% | 75.8% | 65.1% | 51.6% | 52.0% | 53.9% | 48.8% | 61.8% | 50.7% | - | 65.8% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 22.8% | 37.4% | 37.8% | 70.8% | 76.5% | 77.1% | 12.4% | 47.4% | 22.3% | 28.7% | 85.8% | 68.7% | 65.3% | 73.0% | 69.8% | 79.0% | 47.8% | 52.2% | 57.0% | 64.4% | 49.8% | 49.8% | 57.0% | 57.2% | 52.6% | 54.2% | 67.0% |
- | Llama-2-7b-hf | 3.3952 | 73.9% | 48.24 | 39.5% | 39.36 | 43.6% | 25.61 | 50.7% | 35.34 | 46.0% | 78.3% | 57.1% | 76.0% | 69.1% | 43.4% | 46.3% | 76.3% | 74.6% | 35.3% | 40.3% | 31.4% | 44.2% | 93.7% | 90.8% | 91.2% | 90.4% | 86.0% | 49.7% | 77.0% | 67.4% | 50.4% | 53.7% | 59.3% | 48.0% | 62.9% | 50.5% | - | 59.4% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 25.2% | 39.0% | 39.6% | 72.3% | 78.1% | 79.1% | 12.2% | 49.4% | 25.5% | 30.3% | 87.9% | 66.3% | 70.3% | 72.2% | 68.3% | 69.8% | 48.8% | 50.4% | 62.4% | 65.8% | 51.6% | 52.2% | 53.4% | 56.2% | 54.8% | 62.8% | 65.0% |
- | falcon-7b | 3.37 | 74.5% | 51.99 | 38.3% | 39.56 | 42.7% | 19.10 | 52.2% | 29.30 | 48.8% | 78.8% | 57.7% | 76.4% | 67.3% | 40.2% | 43.6% | 74.7% | 70.9% | 34.2% | 40.0% | 30.8% | 44.2% | 94.0% | 91.5% | 90.0% | 89.2% | 86.0% | 47.9% | 78.0% | 69.4% | 50.1% | 48.6% | 55.5% | 48.3% | 53.7% | 48.4% | - | 63.9% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 22.4% | 34.3% | 37.1% | 65.2% | 79.4% | 80.5% | 12.4% | 49.9% | 21.2% | 26.7% | 88.0% | 71.1% | 60.8% | 73.4% | 55.9% | 78.8% | 49.4% | 50.2% | 54.8% | 73.8% | 50.0% | 51.8% | 56.6% | 54.4% | 55.0% | 50.2% | 70.0% |
- | Mistral-7B-v0.1 | 3.1827 | 75.7% | 43.27 | 40.0% | 36.22 | 42.8% | 22.21 | 52.1% | 30.27 | 49.0% | 80.8% | 61.3% | 81.0% | 74.1% | 50.0% | 53.7% | 81.1% | 79.5% | 41.4% | 46.5% | 32.6% | 44.2% | 95.9% | 93.7% | 92.2% | 91.4% | 91.0% | 53.1% | 78.9% | 68.8% | 51.2% | 55.5% | 59.2% | 48.6% | 66.9% | 51.4% | - | 63.4% | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 28.1% | 42.6% | 40.9% | 76.0% | 80.6% | 82.2% | 23.4% | 52.0% | 23.5% | 29.2% | 88.6% | 74.7% | 72.3% | 76.0% | 68.6% | 77.4% | 47.0% | 51.6% | 58.4% | 65.8% | 48.4% | 51.4% | 54.0% | 56.2% | 56.6% | 59.0% | 65.0% |