Issue with running Mistral-7B

[NamburiSrinath]

Hi @asillycat

I am getting the following error when running Mistral-7B with AttnGCG. That's the reason why I am unable to replicate the results of Mistral in #4!!

Traceback (most recent call last):
  File "/AutoDrafter/libs/AttnGCG/experiments/launch_scripts/../main.py", line 99, in <module>
    app.run(main)
  File "/usr/local/lib/python3.10/dist-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.10/dist-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "/AutoDrafter/libs/AttnGCG/experiments/launch_scripts/../main.py", line 76, in main
    control, inner_steps = attack.run(
  File "/AutoDrafter/libs/AttnGCG/AttnGCG/base/attack_manager.py", line 1455, in run
    attack.run(
  File "/AutoDrafter/libs/AttnGCG/AttnGCG/base/attack_manager.py", line 979, in run
    control, loss = self.step(
  File "/AutoDrafter/libs/AttnGCG/AttnGCG/attngcg/attngcg_attack.py", line 215, in step
    _, _, _, loss_infos = zip(*[worker(self.prompts[k][i],
  File "/AutoDrafter/libs/AttnGCG/AttnGCG/attngcg/attngcg_attack.py", line 215, in <listcomp>
    _, _, _, loss_infos = zip(*[worker(self.prompts[k][i],
  File "/AutoDrafter/libs/AttnGCG/AttnGCG/base/attack_manager.py", line 1565, in __call__
    return ob.logits(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/AutoDrafter/libs/AttnGCG/AttnGCG/base/attack_manager.py", line 568, in logits
    raise NotImplementedError("Should not happen")
NotImplementedError: Should not happen

Looking forward to see if there's an issue

[NamburiSrinath]

Hi @asillycat

The below error trace stack is for running GCG for Mistral-7B

.....
  0% 0/1 [00:00<?, ?it/s]
loss_control=0.0240:   0% 0/1 [00:00<?, ?it/s]
loss_control=0.0240: 100% 1/1 [00:00<00:00,  2.15it/s]
loss_control=0.0240: 100% 1/1 [00:00<00:00,  2.15it/s]
Traceback (most recent call last):
  File "/AttnGCG/experiments/launch_scripts/../main.py", line 99, in <module>
    app.run(main)
  File "/usr/local/lib/python3.10/dist-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.10/dist-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "/AttnGCG/experiments/launch_scripts/../main.py", line 76, in main
    control, inner_steps = attack.run(
  File "/AttnGCG/AttnGCG/base/attack_manager.py", line 1455, in run
    attack.run(
  File "/AttnGCG/AttnGCG/base/attack_manager.py", line 979, in run
    control, loss = self.step(
  File "/AttnGCG/AttnGCG/attngcg/attngcg_attack.py", line 201, in step
    control_cands.append(self.get_filtered_cands(j, control_cand, filter_cand=filter_cand, curr_control=curr_control))
  File "/AttnGCG/AttnGCG/base/attack_manager.py", line 907, in get_filtered_cands
    cands = cands + [cands[-1]] * (len(control_cand) - len(cands))
IndexError: list index out of range

[NamburiSrinath]

Scripts used

For AttnGCG

cd experiments/bash_scripts
bash run_direct.sh mistral_7b attngcg 0
cd ../../eval
bash eval.sh mistral_7b attngcg direct

For GCG

cd experiments/bash_scripts
bash run_direct.sh mistral_7b gcg 0
cd ../../eval
bash eval.sh mistral_7b gcg direct

And am using the same config provided for Mistral (https://github.com/UCSC-VLAA/AttnGCG-attack/blob/main/experiments/configs/individual_mistral_7b.py)

Please let me know if you need any more details for debugging the issue