Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a cron job to clean the dangling assets and cached assets #1284

Open
severo opened this issue Jun 1, 2023 · 5 comments
Open

Create a cron job to clean the dangling assets and cached assets #1284

severo opened this issue Jun 1, 2023 · 5 comments
Labels
improvement / optimization infra P2 Nice to have

Comments

@severo
Copy link
Collaborator

severo commented Jun 1, 2023

Related to #1122
This was referenced Jun 1, 2023
Closed
Closed
@severo severo mentioned this issue Jun 26, 2023
Closed
@github-actions
Copy link

github-actions bot commented Jul 1, 2023

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

@severo
Copy link
Collaborator Author

severo commented Jul 17, 2023

keep open

@severo severo reopened this Jul 17, 2023
@severo severo added the P2 Nice to have label Jul 26, 2023
@severo severo mentioned this issue Aug 3, 2023
Closed
@AndreaFrancis AndreaFrancis mentioned this issue Sep 19, 2023
Merged
@severo
Copy link
Collaborator Author

severo commented Feb 6, 2024

As we now use signed URLs, only the assets referenced by a cache entry will be accessible. The other ones, if any, will "just" waste space on S3

@severo
Copy link
Collaborator Author

severo commented Feb 6, 2024

Also: the files on S3 should now be deleted when we delete a dataset (see https://github.com/huggingface/datasets-server/blob/c22f563f350a36148a1deaaf12531d4da991a677/libs/libcommon/src/libcommon/orchestrator.py#L678)

@severo
Copy link
Collaborator Author

severo commented Feb 6, 2024

Note: as of today, the production S3 bucket contains 3TB of assets. It would be interesting to know which part is not needed anymore.

Also: /cached-assets directory could have a TTL of 1 day, or less, since the assets are generated on demand.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
improvement / optimization infra P2 Nice to have
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@severo