You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We would like to support benchmark result IO to various places without having to prepare complex logic on the command line directly.
For example, an invocation like nnbench benchmarks.py -o gs://my-bucket/my-result1.json should write the results to json on disk, and then send the file to the given GCS bucket. A tested solution for this is the fsspec project.
The same works for an existing record upload, where nnbench benchmarks.py -o bq://my-proj/my-dataset/my-table could push the results into the given BQ table.
The interesting case is record fetching from databases in evaluation / comparison scenarios. For example, when comparing a local run (say 1.json) to latest results in BQ where a context value foo equals 'bar', it would be nnbench compare 1.json bq://my-proj/my-dataset/my-table, but WHERE context.foo='bar'.
This could be supported with a command-line parameter --query.
Urgent for building a comparison pipeline.
The text was updated successfully, but these errors were encountered:
Remote file IO using fsspec has landed in v0.4.0. What's left is to tackle the database streaming case.
Perhaps this is the time (together with the pyproject.toml feature) to implement a URI lookup feature similar to fsspec's - in that spirit, arguments could then be overridden directly in TOML.
We would like to support benchmark result IO to various places without having to prepare complex logic on the command line directly.
For example, an invocation like
nnbench benchmarks.py -o gs://my-bucket/my-result1.json
should write the results to json on disk, and then send the file to the given GCS bucket. A tested solution for this is the fsspec project.The same works for an existing record upload, where
nnbench benchmarks.py -o bq://my-proj/my-dataset/my-table
could push the results into the given BQ table.The interesting case is record fetching from databases in evaluation / comparison scenarios. For example, when comparing a local run (say 1.json) to latest results in BQ where a context value
foo
equals'bar'
, it would bennbench compare 1.json bq://my-proj/my-dataset/my-table
, butWHERE context.foo='bar'
.This could be supported with a command-line parameter
--query
.Urgent for building a comparison pipeline.
The text was updated successfully, but these errors were encountered: