feat: add batch garbage collector#28
Conversation
Add a new batch-gc command that scans for expired batch jobs and removes them from the database. Designed for one-shot execution, suitable for Kubernetes CronJob deployment. Includes unit tests. TODO: Add file garbage collection once the Files API is implemented. Signed-off-by: Antonio Cardace <acardace@redhat.com>
Add Helm chart support for deploying the batch-gc as a Kubernetes CronJob. Configurable schedule, dry-run mode, and database connection. Signed-off-by: Antonio Cardace <acardace@redhat.com>
|
|
||
| gc := collector.NewGarbageCollector(dbClient, *dryRun) | ||
|
|
||
| result, err := gc.Run(ctx) |
There was a problem hiding this comment.
What's the benefit of running this as a cron job, relative to implementing this as a long running background process, and inside it will run the gc process periodically based on a frequency param?
If it's a background process it can also expose metrics on its operation?
| } | ||
|
|
||
| // Check if job has expired | ||
| if statusInfo.ExpiresAt == nil || *statusInfo.ExpiresAt > now { |
There was a problem hiding this comment.
With the new DB interface - get can provide all the expired items, so no need for this check?
i'm assuming we will rebase this after merging the updated db interface.
| } | ||
|
|
||
| result.JobsExpired++ | ||
| logger.V(logging.INFO).Info("Found expired job", "jobID", job.ID, "expiresAt", *statusInfo.ExpiresAt) |
There was a problem hiding this comment.
Maybe info level logging would be on the iteration counters level, and specific items that are operated on would be in a debug logging level?
Another option is to get as an input config param - the level for logging individual item operations?
| logger.V(logging.INFO).Info("Found expired job", "jobID", job.ID, "expiresAt", *statusInfo.ExpiresAt) | ||
|
|
||
| if c.dryRun { | ||
| logger.V(logging.INFO).Info("[DRY RUN] Would delete job", "jobID", job.ID) |
There was a problem hiding this comment.
Instead of multiple logs line for an individual item being operated on, consider a single debug level log line per item being operated on - that log line will summarize what was applied to that item.
2 participants
Summary
batch-gccommand that scans for expired batch jobs and removes them from the database--dry-runflag to preview deletions without executingTODO
ExpiresAtfield)Helm Chart
Adds CronJob template with configurable:
schedule(default: daily at 2am)dryRunmode