Skip to content

Conversation

zhangwl9
Copy link
Contributor

@zhangwl9 zhangwl9 commented Sep 26, 2025

Why are the changes needed?

Close #3781.

Brief change log

To optimize table cleanup operations (Dangling-Delete_file cleanup, data expiration, orphan_file cleanup,snapshotsFile cleanup), the last cleanup time is saved. This ensures that after an AMS service restart, if a cleanup operation was completed just before the restart, the next cleanup operation will only be performed after exceeding the respective execution interval.
The execution flow is as follows:

  • Create the table table_cleanup_process to store the cleanup history of optimized tables.

  • During AMS initialization, batch query the cleanup history of optimized tables.

  • For optimization tables with a previous cleanup time, cache the cleanup history.

  • Determine whether the time interval since the last cleanup exceeds the cleanup operation execution interval.

  • For optimization tables exceeding the execution interval, execute the cleanup operation.

  • Otherwise, schedule for the next round of execution.

  • Since each table's cleanup operation takes varying durations, after a table's cleanup completes, update or insert the cleanup completion time in the table_cleanup_process table.

  • When an optimized table is deleted, its history in the table_cleanup_process must be removed.

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@github-actions github-actions bot added the module:ams-server Ams server module label Sep 26, 2025
@zhangwl9 zhangwl9 force-pushed the AMORO-Save-lastTime-dev branch from bce7e0d to 8162476 Compare September 28, 2025 03:28
@codecov-commenter
Copy link

codecov-commenter commented Sep 28, 2025

Codecov Report

❌ Patch coverage is 63.27434% with 83 lines in your changes missing coverage. Please review.
✅ Project coverage is 29.01%. Comparing base (43c2ee1) to head (8162476).

Files with missing lines Patch % Lines
...amoro/server/scheduler/PeriodicTableScheduler.java 62.60% 38 Missing and 5 partials ⚠️
.../server/table/cleanup/TableCleanupProcessMeta.java 48.83% 22 Missing ⚠️
...rsistence/converter/CleanupOperationConverter.java 30.76% 9 Missing ⚠️
...er/inline/DanglingDeleteFilesCleaningExecutor.java 0.00% 2 Missing ⚠️
.../server/scheduler/inline/DataExpiringExecutor.java 0.00% 2 Missing ⚠️
.../scheduler/inline/OrphanFilesCleaningExecutor.java 0.00% 2 Missing ⚠️
...er/scheduler/inline/SnapshotsExpiringExecutor.java 0.00% 2 Missing ⚠️
...erver/table/cleanup/CleanupProcessPersistence.java 96.87% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3802      +/-   ##
============================================
+ Coverage     21.96%   29.01%   +7.04%     
- Complexity     2431     3860    +1429     
============================================
  Files           441      633     +192     
  Lines         40790    50797   +10007     
  Branches       5756     6531     +775     
============================================
+ Hits           8961    14739    +5778     
- Misses        31078    35010    +3932     
- Partials        751     1048     +297     
Flag Coverage Δ
core 29.01% <63.27%> (?)
trino ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module:ams-server Ams server module
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature]: Save the last completion time of the executor cleanup
2 participants