Skip to content

Complete some of function Implementation in BackupWorker V3#13245

Open
akankshamahajan15 wants to merge 7 commits into
apple:mainfrom
akankshamahajan15:bk_complete_worker
Open

Complete some of function Implementation in BackupWorker V3#13245
akankshamahajan15 wants to merge 7 commits into
apple:mainfrom
akankshamahajan15:bk_complete_worker

Conversation

@akankshamahajan15
Copy link
Copy Markdown
Contributor

@akankshamahajan15 akankshamahajan15 commented May 19, 2026

This PR includes following changes:

  1. Initialize LogSystem before PartitionMap is pulled. Pulled the logSystem update logic into a new monitorLogSystemFromDbInfo actor, started via addActor before the partition-map await.

  2. Drop _updateStartedWorkers and the BackupConfig keys it managed
    startedBackupWorkers and allWorkerStarted existed for Backup V2 readiness gating. Range-partitioned already gates progress on tagVersions.size() == totalTags, so neither key is needed.
    Worker no longer reads/writes either; client skips the watch for RANGE_PARTITIONED_LOG.
    PARTITIONED_LOG and DEFAULT paths unchanged.

  3. Fix clang-tidy warnings.

  4. Clean up stopped. Removed the dead per-backup stopped field. But kept worker-level stopped flag + stop() method so uploadData exits cleanly on worker_removed.

  5. Simplify and merge the ready functions The pre-existing pattern used static actor helpers behind public wrapper methods, a Flow convention from before C++20 coroutines. Collapsed both _waitReady/waitReady and _waitAllInfoReady/waitAllInfoReady into single member coroutines. Renamed for clarity.

  6. Remove redundant getMore() and nextMessage() in pullPartitionMapFromTLog

  7. Defer pop() for old epochs and on shutdown

  8. Fix iterator double-advance in onBackupChanges

  9. Remove unused variables from BackupWorker.cpp

Testing

Simulation 100k completed:

20260523-013830-ak_bv3-e12f6ac1120c209d compressed=True data_size=37185082 duration=2642536 ended=100000 fail=1 fail_fast=10 max_runs=100000 pass=99999 priority=100 remaining=0 runtime=0:38:25 sanity=False started=100000 stopped=20260523-021655 submitted=20260523-013830 timeout=5400 username=ak_bv3
Unrelated failure

@akankshamahajan15 akankshamahajan15 added the Backup_v3 Range Partitioned Backup label May 19, 2026
@akankshamahajan15 akankshamahajan15 changed the title Complete Some of functions Implementation in BackupWorker V3 Complete some of function Implementation in BackupWorker V3 May 19, 2026
@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: b613b7d
  • Duration 0:25:17
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: b613b7d
  • Duration 0:45:29
  • Result: ❌ FAILED
  • Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: b613b7d
  • Duration 0:45:27
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: b613b7d
  • Duration 0:48:10
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: b613b7d
  • Duration 1:03:05
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: b613b7d
  • Duration 1:06:49
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos on macOS Ventura 13.x

  • Commit ID: b613b7d
  • Duration 1:18:34
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: e23bf5a
  • Duration 0:29:56
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: e23bf5a
  • Duration 0:44:35
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: e23bf5a
  • Duration 1:08:01
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos on macOS Ventura 13.x

  • Commit ID: e23bf5a
  • Duration 1:08:09
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: e23bf5a
  • Duration 1:19:22
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: e23bf5a
  • Duration 3:01:04
  • Result: ❌ FAILED
  • Error: Build has timed out.
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: e23bf5a
  • Duration 3:02:23
  • Result: ❌ FAILED
  • Error: Build has timed out.
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: 2df0887
  • Duration 0:22:04
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: 2df0887
  • Duration 0:44:48
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: 2df0887
  • Duration 0:55:14
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: 2df0887
  • Duration 0:55:23
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: 2df0887
  • Duration 1:00:38
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: 2df0887
  • Duration 1:25:04
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos on macOS Ventura 13.x

  • Commit ID: 2df0887
  • Duration 2:21:23
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@akankshamahajan15 akankshamahajan15 marked this pull request as ready for review May 23, 2026 01:34
@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: afb46de
  • Duration 0:22:16
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: afb46de
  • Duration 0:33:59
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: afb46de
  • Duration 0:44:05
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: afb46de
  • Duration 0:45:55
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: afb46de
  • Duration 0:52:33
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: afb46de
  • Duration 1:00:15
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos on macOS Ventura 13.x

  • Commit ID: afb46de
  • Duration 1:23:15
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR advances BackupWorker V3 (range-partitioned logs) behavior and robustness by restructuring LogSystem initialization, removing V2 readiness-gating keys, and addressing several correctness/cleanup items in the range-partitioned backup worker and client task flow.

Changes:

  • Start a dedicated actor to keep logSystem/oldestBackupEpoch updated from ServerDBInfo before partition-map processing begins.
  • Remove use of BackupConfig readiness keys (startedBackupWorkers / allWorkerStarted) for RANGE_PARTITIONED_LOG, and skip the corresponding watch in StartFullBackupTaskFunc.
  • Fix/clean up several range-partitioned worker behaviors (stop/shutdown handling, iterator/loop fixes, redundant cursor calls removal, pop deferral).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
fdbserver/core/include/fdbserver/core/BackupPartitionMap.h Modernize type aliases (typedefusing).
fdbserver/backupworker/BackupWorkerRangePartitioned.cpp Core range-partitioned worker correctness/cleanup, LogSystem monitoring actor, shutdown behavior updates.
fdbserver/backupworker/BackupWorker.cpp Minor cleanup (remove unused fields/accessors).
fdbclient/FileBackupAgent.cpp Skip allWorkerStarted watch for RANGE_PARTITIONED_LOG since workers no longer set it.

Comment thread fdbserver/backupworker/BackupWorkerRangePartitioned.cpp
Comment thread fdbserver/core/include/fdbserver/core/BackupPartitionMap.h Outdated
@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos on macOS Ventura 13.x

  • Commit ID: b777449
  • Duration 0:15:30
  • Result: ❌ FAILED
  • Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login ./build_pr_macos.sh. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: b777449
  • Duration 0:21:43
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: b777449
  • Duration 0:46:54
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: b777449
  • Duration 0:51:39
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: b777449
  • Duration 0:56:58
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: b777449
  • Duration 1:01:46
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: b777449
  • Duration 1:06:22
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)


// The task may be restarted. Set the watch if started key has NOT been set.
if (!taskStarted.get().present()) {
if (!isRangePartitioned && !taskStarted.get().present()) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it not better if we make it explicit to MutationLogType::PARTITIONED_LOG

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Backup_v3 Range Partitioned Backup

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants