Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement URI scheme for command-line (and SDK) IO #157

Open
nicholasjng opened this issue Nov 11, 2024 · 1 comment
Open

Implement URI scheme for command-line (and SDK) IO #157

nicholasjng opened this issue Nov 11, 2024 · 1 comment
Labels
enhancement New feature or request
Milestone

Comments

@nicholasjng
Copy link
Collaborator

We would like to support benchmark result IO to various places without having to prepare complex logic on the command line directly.

For example, an invocation like nnbench benchmarks.py -o gs://my-bucket/my-result1.json should write the results to json on disk, and then send the file to the given GCS bucket. A tested solution for this is the fsspec project.

The same works for an existing record upload, where nnbench benchmarks.py -o bq://my-proj/my-dataset/my-table could push the results into the given BQ table.

The interesting case is record fetching from databases in evaluation / comparison scenarios. For example, when comparing a local run (say 1.json) to latest results in BQ where a context value foo equals 'bar', it would be nnbench compare 1.json bq://my-proj/my-dataset/my-table, but WHERE context.foo='bar'.

This could be supported with a command-line parameter --query.

Urgent for building a comparison pipeline.

@nicholasjng nicholasjng added the enhancement New feature or request label Nov 11, 2024
@nicholasjng nicholasjng added this to the v0.5.0 milestone Dec 3, 2024
@nicholasjng
Copy link
Collaborator Author

Remote file IO using fsspec has landed in v0.4.0. What's left is to tackle the database streaming case.

Perhaps this is the time (together with the pyproject.toml feature) to implement a URI lookup feature similar to fsspec's - in that spirit, arguments could then be overridden directly in TOML.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant