Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"new" data sets are missing in the leader board #98

Closed
gschramm opened this issue Sep 2, 2024 · 5 comments
Closed

"new" data sets are missing in the leader board #98

gschramm opened this issue Sep 2, 2024 · 5 comments

Comments

@gschramm
Copy link

gschramm commented Sep 2, 2024

Seems that the leaderboard only contains the initial 3 data sets.

If possible, they (e.g. the mMR ACR data sets) should be added here:

(SRCDIR / "Siemens_Vision600_thorax", OUTDIR / "Vision600_thorax",

@KrisThielemans
Copy link
Member

As we're getting new data ready, we will likely only automatically run 1 dataset from each scanner, to avoid overloading the STFC computing once people start to submit a lot of jobs (hopefully!).

You can run them locally by adjusting your own petric. Or maybe there's a smarter option...

@gschramm
Copy link
Author

gschramm commented Sep 2, 2024

makes sense. I guess in principle there could be a decision based on the hostname in petric.py which data sets to run.
(if the hostnames on STFC follow a pattern and it is safe to make that pattern public).

otherwise the decision could be made based on a env. variable that users can set to run all data sets locally.

@gschramm gschramm closed this as completed Sep 2, 2024
@KrisThielemans
Copy link
Member

could be doable I guess. @casperdcl, What do you think? 5 min job?

@casperdcl
Copy link
Member

casperdcl commented Sep 2, 2024

you mean truncating the dataset list length in #91 using int(os.getenv("PETRIC_MAX_DATASETS", "3")) or similar?

P.S. my last "5 min job" shaved days off my life

@KrisThielemans
Copy link
Member

let's leave it then. Participants can make this change themselves.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants