Skip to content

Raw spectra/echem should not be stored in database #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
ml-evs opened this issue Apr 23, 2025 · 1 comment · May be fixed by #23
Open

Raw spectra/echem should not be stored in database #15

ml-evs opened this issue Apr 23, 2025 · 1 comment · May be fixed by #23

Comments

@ml-evs
Copy link
Member

ml-evs commented Apr 23, 2025 •
edited
Loading

Currently the plotting requires the raw echem/NMR spectra to be stored in the database (since they are saved under self.data of the block). We should do our best to avoid this and instead cache to disk, especially as it will be impossible to store in the database if it goes over 16 MB.
@ml-evs ml-evs mentioned this issue Apr 23, 2025
Open
@ml-evs
Copy link
Member Author

ml-evs commented May 6, 2025

Best case: once the first parsing is done, save a pkl (or ideally something better like hdf5 or parquet) of all the data, to disk, alongside the zip at e.g., /app/files/<file_id>/insituNMR.zip -> /app/files/<file_id>/insituNMR.h5, so then when the user comes back to the block, it checks if the parsed data exists first, then just loads it.

@BenjaminCharmes BenjaminCharmes linked a pull request May 9, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Store HDF5 data in cache datalab-org/datalab-app-plugin-insitu
1 participant
@ml-evs