Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add local disk cache for remote data chunks #99

Merged
merged 52 commits into from
Apr 1, 2025
Merged

Add local disk cache for remote data chunks #99

merged 52 commits into from
Apr 1, 2025

Conversation

sd109
Copy link
Member

@sd109 sd109 commented Feb 3, 2025 •
edited
Loading

TODO:

  • Figure out how to test this (unit tests, integration tests, benchmark tests, all three?)
  • Make cache location (and maybe size?) configurable via Ansible
  • Check cache-related error handling and HTTP error responses

maxstack and others added 6 commits January 20, 2025 08:28
@maxstack
Update Ansible deployment to allow for unprivileged deployment using …
9c95a15 
…Podman

- change Ansible collection from community.docker to containers.podman
- update all Ansible tasks to use podman instead of docker, we install podman-docker for a docker compatible CLI but we need to be podman specific in our Ansible playbook
- move group_vars/reductionist -> group_vars/all so all "reductionist_" prefixed vars can be used across plays, specifically Step and Reductionist
- update documentation
@sd109
WIP: cached crate experiments
c98d16a
@sd109
Add Bytes serde feature
99e61ba
@sd109
Functional disk cache implementation
df2ed74
@sd109
Add prometheus counter for cache misses
0d70386
@sd109
Ensure client ID is included in chunk cache lookup key
f809321
@sd109 sd109 marked this pull request as draft February 3, 2025 18:10
sd109 and others added 4 commits February 7, 2025 18:04
@sd109
Refactor to allow reinstating resource manager
d9dfd2d
@maxstack
Update the podman installation play to enable linger on the unprivile…
528d532 
…ged user account that'll be running the Reductionist, and any other, containers.

We need linger enabled for the podman process to continue running after the user's logged out of the session.
Deployment documentation updated.
@maxstack
Allow Prometheus and HAProxy to be installed as the non-privileged user.
f3b994d 
Reductionist can be used with https, either with the optional Ansible playbook Step deployment or with third party certificates. Documentation updated for installation of 3rd party certificates.
@maxstack
Implement an object chunk cache on disk where we can configure:
8fb4a95 
 * the on disk path to the cache
 * the TTL of cache entries
 * whether a cache hit refreshes the chunk's TTL
sd109
sd109 commented Feb 27, 2025
View reviewed changes
Reductionist and others added 4 commits March 4, 2025 15:10
Refactor the chunk cache so it stores each chunk in a separate file.
2882c47 
Features:
 * configure a maximum size for the cache
 * pruned periodically so expired entries are purged and the cache files deleted
 * pruning removes excess files, that haven't yet expired, to keep within configured maximum cache size
@sd109
Add example of simplified cache implementation
a414f0c
@maxstack
Add SimpleChunkCache pruning, with tests, and integrate with the Redu…
3e6ed6d 
…ctionist by using the ChunkCache object as an interfacing wrapper for ActiveStorageError.

Currently loads and saves state to disk, rather than keeping in memory, which could be a future performance improvement.
@maxstack
Remove unneeded dependencies, added during development of the caching…
37f0c95 
… feature
sd109
sd109 commented Mar 10, 2025
View reviewed changes
maxstack added 13 commits March 11, 2025 10:18
@maxstack
Provide a S3 download handler and a cached S3 download handler with t…
4715e3e 
…he operation handler deciding which to use based on app settings.

Ideally move the decision even further up registering one S3 download handler in the shared state, the actual one used being based on app settings. This would allow a http handler to be registered in the same way. Proves "fun" with the lifetime of objects.
@maxstack
Panic if the cache directory already exists.
5d408c8 
Change tests over to temporary directories so they'll never fail due to an existing directory.
@maxstack
Remove test portion relating to cache wipe on init.
c549b29 
Remove explicit cache wipe on exit, only used for test and not needed for temporary directories.
@maxstack
Channel all cache commits through a mpsc channel to serialise commits…
979d8a3 
… coming from requests in a multi-threaded async API. This protects the cache from concurrent access and the request that downloaded the chunk doesn't have to wait for it to be committed to the cache before it can be returned to the client.
@maxstack
Allow configuration of the pruning interval, in seconds, of cache chu…
04e1f55 
…nks based on ttl.
@maxstack
Make "cargo clippy" happy.
91eb805
@maxstack
Merge remote-tracking branch 'origin/feat/disk-cache' into deployment…
c405c84 
…/wacasoft
@maxstack
Configure chunk cache test deployment.
4cb4ae8
@maxstack
Deploy a local build of the Reductionist.
093e9e4
@maxstack
Update Reductionist tag.
ec5464a
@maxstack
Tweak what I think I think is needed to deploy our local build.
416cce3
@maxstack
Correct the volume setup for the chunk cache.
6fa2203
@maxstack
Tweak Reductionist deployment in group vars.
46a3162
sd109
sd109 commented Mar 17, 2025
View reviewed changes
sd109
sd109 commented Mar 17, 2025
View reviewed changes
maxstack added 10 commits March 17, 2025 18:06
@maxstack
Add "headroom_bytes" to the pruning so we can ensure this many bytes …
988e196 
…will fit into the cache after it is pruned, we'll use this to ensure the chunk we're adding can be accomodated.

Add associated tests.
@maxstack
Tidy up adding code documentation.
a00c942
@maxstack
Change over to tempfile::TempDir for temporary directory creation due…
72fc70a 
… to a security issue with tempdir::TempDir.
@maxstack
Change boolean assert_eq to assert to keep clippy happy.
9e4d08a
@maxstack
Remove dupplicated cache error.
4902ae1
@maxstack
Debug trait no longer needed on ResourceManager.
b20e193
@maxstack
Merge remote-tracking branch 'origin/main' into feat/disk-cache
36a655e
@maxstack
Add chunk cache configuration to group_vars/all.
cd6876b 
Add Reductionist build configuration to group_vars/all to:
 - Specify the location of the repo
 - Disable cloning to the repo location, this is useful if the location already exists with changes that you don't want to lose
@maxstack
Fix for group_vars/all
f1659c8
@maxstack
More sensible default for the cache path that works with our without …
9a7023e 
…the chunk cache enabled.
@sd109 sd109 marked this pull request as ready for review March 21, 2025 16:54
@sd109 sd109 changed the title WIP: Add local disk cache for remote data chunks Add local disk cache for remote data chunks Mar 22, 2025
@sd109 sd109 force-pushed the feat/disk-cache branch from f64ef3d to 063a084 Compare March 24, 2025 23:20
sd109
sd109 commented Mar 25, 2025
View reviewed changes
maxstack
maxstack reviewed Mar 25, 2025
View reviewed changes
@sd109 sd109 force-pushed the feat/disk-cache branch from db2c255 to 195e663 Compare March 27, 2025 19:25
sd109
sd109 commented Apr 1, 2025
View reviewed changes
@sd109 sd109 merged commit 5916eb2 into main Apr 1, 2025
8 checks passed
@sd109 sd109 deleted the feat/disk-cache branch April 1, 2025 10:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maxstack maxstack maxstack left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sd109 @maxstack