feat: Replace deprecated qpress with configurable compression options for backups #4340

the-wunmi · 2025-08-10T05:08:06Z

Summary

This PR updates backup compression by replacing the deprecated qpress dependency with configurable compression options, allowing users to choose from multiple compression algorithms for MySQL and MongoDB backups.

Background

Previous MySQL backups enforced qpress compression by default, but qpress has been disabled in newer PMM versions due to deprecation. This change removes the qpress dependency and provides users with flexible compression options.

Changes Made

Compression Algorithm Options

Added BackupCompression enum support: QuickLZ, ZSTD, LZ4, S2, GZIP, Snappy, PGZip, Default and None
Breaking Change: Removed enforced qpress compression from MySQL backups
Users can now explicitly choose their preferred compression algorithm or disable compression entirely

MySQL Backup Updates

Removed mandatory --compress flag from xtrabackup commands
Made qpress dependency conditional (only required when QuickLZ is selected)
Updated MySQLBackupJob and MySQLRestoreJob to accept compression parameters
Improved binary detection to only check for qpress when needed

MongoDB Backup Enhancement

Extended existing MongoDB backup compression support
Added compression parameter passing to pbm commands
Consistent API across both database engines

API and Data Model Updates

Extended backup request/response APIs to include compression settings
Updated scheduled backup configurations to store compression preferences
Modified artifact metadata to track which compression was used
Updated database schemas and models accordingly

Impact

Removes Technical Debt: Eliminates dependency on deprecated qpress
User Control: Provides explicit compression choice instead of forced qpress
Performance Options: Users can select compression algorithms based on their speed/size trade-off preferences
Forward Compatibility: Prepares infrastructure for future compression algorithms

Migration

Existing backups remain compatible, legacy artifacts are updated to use quicklz and s2 for MySQL and MongoDB respectively as currently supported
New backups default to default compression provided by the backup tool (xtrabackup/pbm) unless explicitly specified
Users who need QuickLZ can still select it, but qpress binary must be available
API Docs updated
Link to related pull request: Backup: Replace deprecated qpress with configurable compression options grafana#839

Demo

Before:

pmm-backup-bug.mov

After:

pmm-backup-compression-demo.mov

the-wunmi · 2025-08-10T05:39:29Z

agent/runner/jobs/mongodb_backup_job.go

 	}

+	switch j.compression {
+	case backuppb.BackupCompression_BACKUP_COMPRESSION_DEFAULT:


specifies PBMs configuration options, and by default no specific compression

looks to me like it's defaulting to gzip though.

Its an empty case block, allowing to use PBM's default option to compress with s2, can explicitly specify that if that's preferred.

the-wunmi · 2025-08-10T05:40:47Z

agent/runner/jobs/mysql_backup_job.go

-	if _, err := exec.LookPath(qpressBin); err != nil {
-		return errors.Wrapf(err, "lookpath: %s", qpressBin)
+	if j.compression == backuppb.BackupCompression_BACKUP_COMPRESSION_QUICKLZ {
+		if _, err := exec.LookPath(qpressBin); err != nil {


Ensures we only check that qpress is installed if user wants to explicitly use quicklz, other compression options are installed as part of xtrabackup, so no need to explicitly check for them

the-wunmi · 2025-08-10T05:46:18Z

managed/services/management/backup/backup_service.go

 }

+// ListServiceCompression returns available compression methods for a service.
+func (s *BackupService) ListServiceCompression(ctx context.Context, req *backupv1.ListServiceCompressionRequest) (*backupv1.ListServiceCompressionResponse, error) {


As described, provided an input service, checks for the available compression methods for the related service type

managed/services/management/backup/restore_service.go

idoqo · 2025-08-14T17:01:13Z

agent/runner/jobs/mongodb_backup_job.go

 	}

+	switch j.compression {
+	case backuppb.BackupCompression_BACKUP_COMPRESSION_DEFAULT:


looks to me like it's defaulting to gzip though.

idoqo · 2025-08-14T17:02:47Z

agent/runner/jobs/mongodb_backup_job.go

+	case backuppb.BackupCompression_BACKUP_COMPRESSION_S2:
+		pbmArgs = append(pbmArgs, "--compression=s2")


are we sure pbm supports this compression method?

Yeah, also documented as the default -- https://docs.percona.com/percona-backup-mongodb/reference/backup-options.html#backupcompression

codecov · 2025-08-14T17:05:30Z

Codecov Report

❌ Patch coverage is 36.58537% with 182 lines in your changes missing coverage. Please review.
✅ Project coverage is 44.48%. Comparing base (13c4fd3) to head (534a92c).
⚠️ Report is 3 commits behind head on v3.

Files with missing lines	Patch %	Lines
managed/services/agents/jobs.go	0.00%	49 Missing ⚠️
...naged/services/management/backup/backup_service.go	50.57%	38 Missing and 5 partials ⚠️
agent/runner/jobs/mongodb_backup_job.go	5.26%	18 Missing ⚠️
agent/runner/jobs/mysql_backup_job.go	0.00%	18 Missing ⚠️
agent/runner/jobs/mysql_restore_job.go	0.00%	16 Missing ⚠️
managed/models/scheduled_tasks_helpers.go	0.00%	11 Missing and 1 partial ⚠️
managed/models/artifact_model.go	78.78%	6 Missing and 1 partial ⚠️
...aged/services/management/backup/restore_service.go	0.00%	5 Missing ⚠️
managed/services/scheduler/task.go	28.57%	4 Missing and 1 partial ⚠️
agent/client/client.go	0.00%	4 Missing ⚠️
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##               v3    #4340      +/-   ##
==========================================
- Coverage   44.56%   44.48%   -0.08%     
==========================================
  Files         363      363              
  Lines       45706    45937     +231     
==========================================
+ Hits        20369    20437      +68     
- Misses      23677    23831     +154     
- Partials     1660     1669       +9

Flag	Coverage Δ
admin	`17.33% <ø> (ø)`
agent	`53.03% <1.72%> (-0.28%)`	⬇️
managed	`44.51% <45.41%> (-0.02%)`	⬇️
vmproxy	`74.13% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…cross multiple files

the-wunmi · 2025-08-28T21:10:50Z

@idoqo ICYMI, I've fixed the linter error that caused the workflow trigger to fail, might want to help trigger again

JiriCtvrtka · 2025-09-01T14:00:54Z

@the-wunmi could you resolve conflicts please?

the-wunmi · 2025-09-08T14:54:33Z

@the-wunmi could you resolve conflicts please?

@JiriCtvrtka resolved.

…sts to include compression parameter in BackupTaskParams.

…finitions. Remove references to Qpress in version cache tests.

the-wunmi · 2025-09-15T05:28:55Z

hello @JiriCtvrtka @idoqo conflict on this has been resolved, alongside failing workflows, except for a linkspector check, unrelated to my changes here

CLAassistant · 2025-10-23T08:34:36Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

the-wunmi added 2 commits August 7, 2025 19:49

init.

f60a433

Add ListServiceCompression API and update compression handling

5e2b948

the-wunmi mentioned this pull request Aug 10, 2025

Backup: Replace deprecated qpress with configurable compression options percona/grafana#839

Open

the-wunmi marked this pull request as ready for review August 10, 2025 05:34

the-wunmi requested review from a team and BupycHuk as code owners August 10, 2025 05:34

the-wunmi requested review from JiriCtvrtka and idoqo and removed request for a team August 10, 2025 05:34

the-wunmi commented Aug 10, 2025

View reviewed changes

the-wunmi mentioned this pull request Aug 10, 2025

Backup: Replace deprecated qpress with configurable compression options Percona-Lab/pmm-submodules#4010

Draft

idoqo reviewed Aug 14, 2025

View reviewed changes

fix fmt.

7d79131

the-wunmi requested a review from idoqo August 22, 2025 07:31

Update BackupCompression enum to use UNSPECIFIED instead of INVALID a…

f8a38dd

…cross multiple files

the-wunmi added 3 commits September 1, 2025 15:06

Merge branch 'v3' into backup-compression-3

4f2606f

fix reported errors.

591cbf3

Merge branch 'v3' into backup-compression-3

3503ba4

the-wunmi added 3 commits September 14, 2025 10:24

Merge branch 'v3' into backup-compression-3

f2d7d1a

Fix failing tests.

d4093e6

Update database schema to set default compression value and modify te…

43f5800

…sts to include compression parameter in BackupTaskParams.

the-wunmi force-pushed the backup-compression-3 branch from 7fdd25e to 43f5800 Compare September 14, 2025 11:05

the-wunmi added 3 commits September 14, 2025 15:14

Fix failing tests.

51b5ad3

Deprecate Qpress in GetVersionsRequest and update related protobuf de…

d490673

…finitions. Remove references to Qpress in version cache tests.

fix ci check issue.

50afba6

Merge branch 'v3' into backup-compression-3

534a92c

		case backuppb.BackupCompression_BACKUP_COMPRESSION_S2:
		pbmArgs = append(pbmArgs, "--compression=s2")

feat: Replace deprecated qpress with configurable compression options for backups #4340

Are you sure you want to change the base?

feat: Replace deprecated qpress with configurable compression options for backups #4340

Uh oh!

Conversation

the-wunmi commented Aug 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Background

Changes Made

Compression Algorithm Options

MySQL Backup Updates

MongoDB Backup Enhancement

API and Data Model Updates

Impact

Migration

Demo

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

the-wunmi commented Aug 28, 2025

Uh oh!

JiriCtvrtka commented Sep 1, 2025

Uh oh!

the-wunmi commented Sep 8, 2025

Uh oh!

the-wunmi commented Sep 15, 2025

Uh oh!

CLAassistant commented Oct 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

the-wunmi commented Aug 10, 2025 •

edited

Loading

codecov bot commented Aug 14, 2025 •

edited

Loading