-
-
Notifications
You must be signed in to change notification settings - Fork 55
Add seed file support to Browsertrix backend #2710
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add seed file support to Browsertrix backend #2710
Conversation
d6cecdb
to
690238b
Compare
9741bac
to
dc9efcf
Compare
Some of the all crawls tests toward the end of the test run |
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
a31fceb
to
69a5c5a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewing functionality of API changes against frontend-upload-seed-url-list
, working well! I didn't review code or other functionality.
Edit: resolved |
5031352
to
ac499f4
Compare
@ikreymer Seed file cleanup is now working, tested locally, passing in CI, and more configurable via helm chart values. Ready for your eyes again. |
b321e35
to
34efaeb
Compare
Also makes delta for how old unused files need to be before getting cleaned up configurable via helm chart so that we can set it to 1 minute for nightly tests.
This is probably a step forward but may still be confusing until we rework collection thumbnails to be stored in the new files db collection as well and rationalize into one set of models.
Co-authored-by: Tessa Walsh <[email protected]>
… either to current origin or to internal origin, if no headers provided - use resolve_relative_access_path() for both file uploads and collection thumbnails
- consolidate UserUploadFile into SeedFile, can split again if needed - add get_absolute_presigned_url() to UserFile for convenience in getting absolute url directly
a0d3679
to
1b6b4e0
Compare
Fixes #2673 Changes in this PR: - Adds a new `file_uploads.py` module and corresponding `/files` API prefix with methods/endpoints for uploading, GETing, and deleting seed files (can be extended to other types of files moving forward) - Seed files are supported via `CrawlConfig.config.seedFileId` on POST and PATCH endpoints. This seedFileId is replaced by a presigned url when passed to the crawler by the operator - Seed files are read when first uploaded to calculate `firstSeed` and `seedCount` and store them in the database, and this is copied into the workflow and crawl documents when they are created. - Logic is added to store `firstSeed` and `seedCount` for other workflows as well, and a migration added to backfill data, to maintain consistency and fix some of the pymongo aggregations that previously assumed all workflows would have at least one `Seed` object in `CrawlConfig.seeds` - Seed file and thumbnail storage stats are added to org stats - Seed file and thumbnail uploads first check that the org's storage quota has not been exceeded and return a 400 if so - A cron background job (run weekly each Sunday at midnight by default, but configurable) is added to look for seed files at least x minutes old (1440 minutes, or 1 day, by default, but configurable) that are not in use in any workflows, and to delete them when they are found. The backend pods will ensure this k8s batch job exists when starting up and create it if it does not already exist. A database entry for each run of the job is created in the operator on job completion so that it'll appear in the `/jobs` API endpoints, but retrying of this type of regularly scheduled background job is not supported as we don't want to accidentally create multiple competing scheduled jobs. - Adds a `min_seed_file_crawler_image` value to the Helm chart that is checked before creating a crawl from a workflow if set. If a workflow cannot be run, return the detail of the exception in `CrawlConfigAddedResponse.errorDetail` so that we can display the reason in the frontend - Add SeedFile model from base UserFile (former ImageFIle), ensure all APIs returning uploaded files return an absolute pre-signed URL (either with external origin or internal service origin) --------- Co-authored-by: Ilya Kreymer <[email protected]>
Resolves #2646 Depends on #2710 ## Changes (Copied from #2689) - Allows users to specify URL list as file. - Allow uploading a text file of URLs - Allow specifying >100 URLs into URL list, where they will turn into an uploaded list automatically. --------- Co-authored-by: sua yoo <[email protected]>
Fixes #2673
Changes in this PR:
file_uploads.py
module and corresponding/files
API prefix with methods/endpoints for uploading, GETing, and deleting seed files (can be extended to other types of files moving forward)CrawlConfig.config.seedFileId
on POST and PATCH endpoints. This seedFileId is replaced by a presigned url when passed to the crawler by the operatorfirstSeed
andseedCount
and store them in the database, and this is copied into the workflow and crawl documents when they are created.firstSeed
andseedCount
for other workflows as well, and a migration added to backfill data, to maintain consistency and fix some of the pymongo aggregations that previously assumed all workflows would have at least oneSeed
object inCrawlConfig.seeds
/jobs
API endpoints, but retrying of this type of regularly scheduled background job is not supported as we don't want to accidentally create multiple competing scheduled jobs.min_seed_file_crawler_image
value to the Helm chart that is checked before creating a crawl from a workflow if set. If a workflow cannot be run, return the detail of the exception inCrawlConfigAddedResponse.errorDetail
so that we can display the reason in the frontend## Todo- Modify test chart crawler release back to latest once crawler with seed file support is properly released