Work-in-Progress: Design the API so that approaches can incorporate custom dataset forks and temporal information

[mam10eks]

This is a WIP Pull Request where I aim to discuss how the API can allow to use temporal information without having to hard-code them.

My intention is, that it would be cool if the ir-datasets-longeval extension directly allows to run on "non official" dataset forks that users might modify to support their experiments (e.g., as we did this in the ECIR paper).

There is currently no implementation, I was thinking it would be good to first start to sketch how it would look like from the "developer perspective", i.e., I only modified the README.

With this, approaches submit to LongEval could look like this:

Run something on an official dataset:

./my-submission --dataset longeval-web/2022-06

Run something on a local modification of the dataset:

./my-submission --dataset <PATH-ON-MY-MACHINE>

What do you think?

Best regards,

Maik

[janheinrichmerker]

That is basically this: allenai/ir_datasets#175
So probably it would be nice to add your comments to the main repository's pull request, so that the PR there gains some momentum.

[mam10eks]

Yes, I agree.

At the moment, I think it makes sense to first get it properly right for our specific use case, and later look how to generalize. Because I think the specific implementation should be straight forward, and as soon as we have unit tests, we can later switch out the implementation. But for the moment, I think I would go for the easiest possibility.

[mam10eks]

I had a review, from my perspective, all looks good.

[mam10eks]

From my perspective, this would now be ready to be reviewed and merged.

[jueri]

Looks all good, thank you Maik!