Parameter problem

[zjd2024]

After changing the --method and --max_capacity_prompts settings, the generated JSON and scores remain exactly the same.
Is this normal? How should I tune the parameters? 🤦‍♂️
Could you provide a detailed parameter configuration guide?

Below are the three commands I used — all of them produced identical JSON outputs and identical scores.
Looking for some clarification. Thanks!
（1）
bash ./scripts/scripts_longBench/eval.sh
--max_capacity_prompts 2048
--method fullkv
--attn_implementation eager
--source_path ./results/
--model_path /home/jdzhou/exper/uncomp/models/tinyllama
--eval_batch_size 1
--name tinyllama_fullkv
--gpu_id 0
--fp16 1
--seed 43
--logger_pattern info
--port 2236
（2）
bash ./scripts/scripts_longBench/eval.sh
--max_capacity_prompts 384
--method uncomp
--attn_implementation eager
--source_path ./results/
--model_path /home/jdzhou/exper/uncomp/models/tinyllama
--eval_batch_size 1
--name tinyllama_uncomp
--gpu_id 0
--fp16 1
--seed 43
--logger_pattern info
--port 2237
（3）
bash ./scripts/scripts_longBench/eval.sh
--max_capacity_prompts 384
--method uncomp_stage
--attn_implementation eager
--source_path ./results/
--model_path /home/jdzhou/exper/uncomp/models/tinyllama
--eval_batch_size 1
--name tinyllama_uncomp_stage
--gpu_id 0
--fp16 1
--seed 43
--logger_pattern info
--port 2238

[menik1126]

You can test by follow scripts:

bash ./scripts/scripts_longBench/eval.sh --max_capacity_prompts 384 --attn_implementation eager --source_path ./results/ --model_path meta-llama/Llama-2-7b-hf --eval_batch_size 1 --method uncomp --name ./output --gpu_id _multi_2 --fp16 1 --seed 43 --logger_pattern info --port 1236
bash ./scripts/scripts_longBench/metrics.sh --results_dir ./results/results_long_bench/llama-2-7b-hf_384/ --switch True --new_method uncomp

bash ./scripts/scripts_longBench/eval.sh --max_capacity_prompts 384 --attn_implementation eager --source_path ./results/ --model_path meta-llama/Llama-2-7b-hf --eval_batch_size 1 --method fullkv --name ./output --gpu_id _multi_2 --fp16 1 --seed 43 --logger_pattern info --port 1236
bash ./scripts/scripts_longBench/metrics.sh --results_dir ./results/results_long_bench/llama-2-7b-hf_384/ --switch True --new_method fullkv

bash ./scripts/scripts_longBench/eval.sh --max_capacity_prompts 384 --attn_implementation eager --source_path ./results/ --model_path meta-llama/Llama-2-7b-hf --eval_batch_size 1 --method uncomp_stage --name ./output --gpu_id _multi_2 --fp16 1 --seed 43 --logger_pattern info --port 1236
bash ./scripts/scripts_longBench/metrics.sh --results_dir ./results/results_long_bench/llama-2-7b-hf_384/ --switch True --new_method uncomp

and you will get this output ：