Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SQL databases for CLEM workflow to Murfey #321

Merged
merged 70 commits into from
Aug 23, 2024
Merged

Add SQL databases for CLEM workflow to Murfey #321

Show file tree
Hide file tree
Changes from 54 commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
c68c307
Added custom NumPy DTypeError class; added function to return most ap…
tieneupin Jul 3, 2024
87f59bc
Merged recent changes from main branch
tieneupin Jul 16, 2024
971ae41
Added more comprehensive type hinting
tieneupin Jul 22, 2024
d23f8f5
Added backwards compatibility of importlib module for Python 3.9
tieneupin Jul 22, 2024
72fd16a
Overhauled functions used for array manipulation during image stack c…
tieneupin Jul 22, 2024
61e4d94
Updated type hinting for Python 3.9 compatibility
tieneupin Jul 23, 2024
c6c8815
Added rudimentary docstrings to help with categorising the different …
tieneupin Jul 23, 2024
414de4f
Added initial framework for databases to be used in CLEM workflow
tieneupin Jul 23, 2024
23e3f52
Fleshed out the database classes for the CLEM workflow
tieneupin Jul 23, 2024
e059af6
Reverted to using List type hint from typing module to troubleshoot S…
tieneupin Jul 23, 2024
9c0f446
Removed Optional typing from List type hint
tieneupin Jul 23, 2024
f872a06
Added more database structures and relationships for CLEM workflow
tieneupin Jul 24, 2024
0f6dbea
Corrected incorrect types for some database fields
tieneupin Jul 24, 2024
f9dba79
Merging recent updates from main branch
tieneupin Jul 24, 2024
89c4788
Merged CLEMSession and Session classes; used only unique IDs for fore…
tieneupin Jul 25, 2024
748a63d
Added custom NumPy DTypeError class; added function to return most ap…
tieneupin Jul 3, 2024
0aa9aee
Overhauled functions used for array manipulation during image stack c…
tieneupin Jul 22, 2024
4355b29
Updated type hinting for Python 3.9 compatibility
tieneupin Jul 23, 2024
9eaad80
Added rudimentary docstrings to help with categorising the different …
tieneupin Jul 23, 2024
ec71fd5
Added initial framework for databases to be used in CLEM workflow
tieneupin Jul 23, 2024
d1ded5d
Fleshed out the database classes for the CLEM workflow
tieneupin Jul 23, 2024
8617c64
Reverted to using List type hint from typing module to troubleshoot S…
tieneupin Jul 23, 2024
247952a
Removed Optional typing from List type hint
tieneupin Jul 23, 2024
08c4f9f
Added more database structures and relationships for CLEM workflow
tieneupin Jul 24, 2024
ebf4582
Corrected incorrect types for some database fields
tieneupin Jul 24, 2024
dfde2da
Merged CLEMSession and Session classes; used only unique IDs for fore…
tieneupin Jul 25, 2024
d376070
Corrected duplicate lines introduced by rebase
tieneupin Aug 8, 2024
81003cd
Merge branch 'murfey-clem-db' of github.com:DiamondLightSource/python…
tieneupin Aug 8, 2024
f10dfa8
Fixed incorrect types in CLEM tables
tieneupin Aug 8, 2024
de22b96
Added more relationships between CLEM table classes to resolve relati…
tieneupin Aug 8, 2024
72eee58
Removed '# WORK IN PROGRESS' comment from CLEM class now that it's go…
tieneupin Aug 12, 2024
f8cc9c0
Updated relationships in CLEM tables to resolve errors revealed durin…
tieneupin Aug 12, 2024
a2cc41f
Added basic API endpoints to test registration of LIF and TIFF files …
tieneupin Aug 12, 2024
1fb6318
Added '# type: ignore' to 'get_murfey_db_session'
tieneupin Aug 13, 2024
abee2b1
Added type hinting to 'murfey_db' variable
tieneupin Aug 13, 2024
1d82b7e
Added 'sqlalchemy[postgresql]' as a server-side dependency
tieneupin Aug 14, 2024
ebb99ec
Updated column properties and database relationships
tieneupin Aug 14, 2024
08d61ee
Added FastAPI endpoints to register LIF, TIFF, and metadata files
tieneupin Aug 14, 2024
7d67c49
Implemented basic file path validation to address CodeQL warnings
tieneupin Aug 14, 2024
ef6c569
Adjusted validation check used
tieneupin Aug 14, 2024
6c97d37
Merged recent changes from main branch
tieneupin Aug 14, 2024
4116dad
Deleted file regenerated during merge
tieneupin Aug 14, 2024
42c706b
Adjust properties and names of some table columns
tieneupin Aug 16, 2024
1f8855a
Added API endpoint to register image stacks; adjusted error handling …
tieneupin Aug 16, 2024
5d9fce9
Standardised order of fields in CLEM databases
tieneupin Aug 16, 2024
4af5368
Created API endpoint to register CLEM image stacks
tieneupin Aug 16, 2024
88ba95c
Changed column 'name' -> 'series_name' in 'CLEMImageSeries' table
tieneupin Aug 16, 2024
6fda8d7
Streamlined database population method and modified validation and sa…
tieneupin Aug 17, 2024
42a1d40
Added more validation
tieneupin Aug 17, 2024
f80a7c5
Updated validation conditions to only allow certain characters
tieneupin Aug 17, 2024
7864765
Added regex validation for strings as well
tieneupin Aug 17, 2024
f8195bd
Adjusted validation logic again
tieneupin Aug 17, 2024
e2f9eb0
Corrected messages raised messages
tieneupin Aug 17, 2024
aaa9121
Tried validating and sanitising using os.path strings instead
tieneupin Aug 17, 2024
4d46291
Commented out function not being tested; used realpath() instead of a…
tieneupin Aug 17, 2024
dbeb60e
Commented out file existence check
tieneupin Aug 17, 2024
9ccab2d
Use new sanitisation and validation function for code
tieneupin Aug 17, 2024
4e78bb9
Updated code comments
tieneupin Aug 17, 2024
8225aee
Fixed broken series-metadata registration
tieneupin Aug 17, 2024
e188dfa
Added docstrings and comments
tieneupin Aug 21, 2024
cc6432f
Updated Field properties of 'parent_lif' column in CLEMImageMetadata
tieneupin Aug 21, 2024
9cf6766
Refactored API endpoint modules to improve modularity
tieneupin Aug 21, 2024
e35f186
Adjusted comments and dependencies
tieneupin Aug 21, 2024
bb8d318
Replaced print messages and raised exceptions with logger instances
tieneupin Aug 21, 2024
f201a90
Updated dependency after API refactor
tieneupin Aug 21, 2024
911306a
Refactored more API endpoint modules2
tieneupin Aug 21, 2024
f4e7792
Handle exception when default rsync base path is too short
tieneupin Aug 21, 2024
2a55f7b
Adjusted patch paths for server and API tests
tieneupin Aug 21, 2024
5468651
Corrected broken import from API refactor
tieneupin Aug 23, 2024
2de40a1
Removed unnecessary list length checks
tieneupin Aug 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ server = [
"pillow",
"prometheus_client",
"python-jose[cryptography]",
"sqlalchemy[postgresql]", # Add as explicit dependency
"sqlmodel",
"stomp-py<=8.1.0", # 8.1.1 (released 2024-04-06) doesn't work with our project
"uvicorn[standard]",
Expand Down
Loading