plot_sweep.py is missing ([Errno 2] No such file or directory)

[Victor-QTP]

Hi!
I find your work to be very interesting,
I've followed the instructions and I try to replicate this plot

I ran this command
python plot_sweep.py --csv-path PATH_TO_CSV/celeba_eps50_tf.csv

But it appears that the plot_sweep.py is missing ([Errno 2] No such file or directory)
could you please upload it or show me where to get it?

Thank you!

[Samuel-Maddock]

Hi Victor, thanks for pointing this out! This is a typo and the command should refer to plotting/example_plotter.py.

The associated plot() function assumes you are replicating results on sent140. You can change dataset = "celeba" and xlim=2500 to replicate the plot.

I'll make a PR to fix this but in the meantime you can make the changes suggested above to plot locally.

[Victor-QTP]

Hi Victor, thanks for pointing this out! This is a typo and the command should refer to plotting/example_plotter.py.

The associated plot() function assumes you are replicating results on sent140. You can change dataset = "celeba" and xlim=2500 to replicate the plot.

I'll make a PR to fix this but in the meantime you can make the changes suggested above to plot locally.

Thank you for the reply!
I've tried the provided fix, and got this error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/media/me/me/PPFL/myflsim/plotting/example_plotter.py", line 302, in plot
    sns.lineplot(data=plot_df, x="global_round", y="empirical_global_eps", hue="epsilon", ax=ax, label='_nolegend_', palette=palette)
...
  File "/home/me/miniconda3/envs/canife/lib/python3.9/site-packages/seaborn/_core.py", line 903, in _assign_variables_longform
    raise ValueError(err)
ValueError: Could not interpret value `empirical_global_eps` for parameter `y`

Step to reproduce the error:

1. python launcher.py --dataset celeba --dump-path dump_cpu --data-root 'leaf/data/celeba' --epsilon 50 --canary-setup holdout --canary-test-type train_freeze --canary-loss loss2 --canary-design-pool-size 100 --canary-design-minibatch-size num_users --canary-num-test-batches 100 --canary-insert-offset 40 --users-per-round 100 --local-batch-size 128 --canary-epochs 2000 --fl-epochs 30 --model-arch resnet --canary-norm-matching True --canary-norm-constant 5 --device cpu
2. python plotting/extract_exp.py --path 'dump_cpu/' --csv-name celeba_eps50_tf.csv
3. python -c "from plotting.example_plotter import plot;plot('dump_cpu/celeba_eps50_tf.csv');"

Please find my celeba_eps50_tf, cfg file and recent log in the attachment. I think there might be something wrong...

celeba_eps50_tf.csv
cfg.txt
recent_log.txt

[Samuel-Maddock]

Sorry for the delayed response on this and thanks for the csv/logs - super helpful!

There are two fixes here:

1. Running python plotting/example_plotter.py --csv-path ./local_checkpoints/celeba_eps50_tf.csv will generate the plots fine without an error. The plotting must be run from the command line since it generates an intermediary .csv that calculates the composition of the attack epsilons. You ran the plot() method directly which doesn't generate the intermediary .csv.
2. Your .csv was generated from an experiment that only contains a single empirical epsilon (generated from 1 attack) so the above command generates plots that look empty. You need to let training run to completion to get a full plot (or at least for more than 1 attack to occur).

Let me know if you run into any other issues.

[Victor-QTP]

Hi Samuel,
Many thanks for your reply!
1- I understand, I will try the fix later
2- I see, I tried to reproduce the graph following these steps from README:

1. clone https://github.com/TalwalkarLab/leaf/tree/master/data/celeba and create the necessary jsons for the train-test splits
2. run the command:
   python launcher.py --dataset celeba --dump-path dump_cpu --data-root 'leaf/data/celeba' --epsilon 50 --canary-setup holdout --canary-test-type train_freeze --canary-loss loss2 --canary-design-pool-size 100 --canary-design-minibatch-size num_users --canary-num-test-batches 100 --canary-insert-offset 40 --users-per-round 100 --local-batch-size 128 --canary-epochs 2000 --fl-epochs 30 --model-arch resnet --canary-norm-matching True --canary-norm-constant 5 --device cpu
3. run:
   python plotting/extract_exp.py --path 'dump_cpu/' --csv-name celeba_eps50_tf.csv

I think I may have misunderstood the instructions starting from step 2, so to create the plot above, I think I might have to increase the --fl-epochs to 2500 and run the command multiple times with various --epsilon [10,50], am I understand it correctly?

Could you please show me the command that I should run to replicate the above plot (for CelebA)?