Skip to content

chore: Improve canister snapshot download/upload #4406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 28 commits into from
Oct 29, 2025
Merged

chore: Improve canister snapshot download/upload #4406

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
89fa5ce
Stream snapshot downloads to disk directly.
vincent-dfinity Oct 13, 2025
af8d9ed
Add progress bar for snapshot downloading.
vincent-dfinity Oct 13, 2025
f9b17cc
Stream snapshot uploading from disk directly.
vincent-dfinity Oct 14, 2025
262b52a
Add progress bar for snapshot uploading.
vincent-dfinity Oct 14, 2025
0f6f44a
Implement new_progress stub in env.
vincent-dfinity Oct 14, 2025
1299854
Change the log level.
vincent-dfinity Oct 15, 2025
c48795c
Support snapshot downloading with resuming.
vincent-dfinity Oct 15, 2025
c484d67
Add snapshot download concurrency.
vincent-dfinity Oct 15, 2025
c2d5464
Support snapshot uploading with resuming.
vincent-dfinity Oct 16, 2025
8caca0d
Add snapshot upload concurrency
vincent-dfinity Oct 16, 2025
7d94221
Revert the testing chunk size.
vincent-dfinity Oct 16, 2025
ddd984f
Update changelog and document.
vincent-dfinity Oct 16, 2025
a083c51
Addressed review comments.
vincent-dfinity Oct 21, 2025
8742534
Added a test for download/upload with latency, disabled it for now.
vincent-dfinity Oct 22, 2025
71f4625
Make the test passed locally with toxiproxy used.
vincent-dfinity Oct 22, 2025
6e987f8
Update CI to install toxiproxy for canister_extra tests.
vincent-dfinity Oct 22, 2025
d654ea0
Added two e2e tests for canister snapshot.
vincent-dfinity Oct 22, 2025
aa36711
Use toxiproxy-cli directly for debugging...
vincent-dfinity Oct 22, 2025
9ac3ccb
Add more debugging info...
vincent-dfinity Oct 22, 2025
0865e85
Make the snapshot tests be able to run in parallel.
vincent-dfinity Oct 23, 2025
2667fd7
Add canister_extra as serial...
vincent-dfinity Oct 23, 2025
35b06c6
Moved toxiproxy installation into provision script.
vincent-dfinity Oct 24, 2025
e5ae4ee
Make the canister snapshot network drop more robust...
vincent-dfinity Oct 24, 2025
a7eb17d
Merge branch 'master' into vincent/SDK-2382
vincent-dfinity Oct 24, 2025
fcbff22
Use limit_data instead.
vincent-dfinity Oct 26, 2025
0df8dac
fix hanging test (and mac `find -printf`)
adamspofford-dfinity Oct 28, 2025
d155edf
remove focus tag
adamspofford-dfinity Oct 28, 2025
dcf1ce3
Add mention of `--resume` to error message
adamspofford-dfinity Oct 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@

# UNRELEASED

### feat: improved the canister snapshot download/upload feature

Improved the canister snapshot download/upload feature by
- adding progress bars to snapshot download/upload
- streaming snapshot download/upload directly to/from disk.
- supporting download/upload with resuming.
- supporting download/upload with concurrency, default to 3 tasks in parallel.

# 0.30.0

### feat: `dfx start --system-canisters` for bootstrapping system canisters
Expand Down
24 changes: 14 additions & 10 deletions docs/cli-reference/dfx-canister.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1028,11 +1028,13 @@ dfx canister snapshot download <canister> <snapshot> --dir <DIR>

You can use the following arguments with the `dfx canister snapshot download` command.

| Argument | Description |
|---------------|----------------------------------------------------------------------------|
| `<canister>` | The canister to download the snapshot from. |
| `<snapshot>` | The ID of the snapshot to download. |
| --dir `<dir>` | The directory to download the snapshot to. It should be created and empty. |
| Argument | Description |
|-------------------------------|--------------------------------------------------------------------------------------------|
| `<canister>` | The canister to download the snapshot from. |
| `<snapshot>` | The ID of the snapshot to download. |
| --dir `<dir>` | The directory to download the snapshot to. It should be created and empty if not resuming. |
| --resume | Whether to resume the download if the previous snapshot download failed. |
| --concurrency `<concurrency>` | The number of concurrent downloads to perform [default: 3]. |

### Examples

Expand All @@ -1056,11 +1058,13 @@ dfx canister snapshot upload <canister> --dir <DIR>

You can use the following arguments with the `dfx canister snapshot upload` command.

| Argument | Description |
|-----------------------|--------------------------------------------------------------------------------|
| `<canister>` | The canister to upload the snapshot to. |
| --dir `<dir>` | The directory to upload the snapshot from. |
| --replace `<replace>` | If a snapshot ID is specified, the snapshot identified by this ID will be deleted and a snapshot with a new ID will be returned. |
| Argument | Description |
|-------------------------------|--------------------------------------------------------------------------------|
| `<canister>` | The canister to upload the snapshot to. |
| --dir `<dir>` | The directory to upload the snapshot from. |
| --replace `<replace>` | If a snapshot ID is specified, the snapshot identified by this ID will be deleted and a snapshot with a new ID will be returned. |
| --resume `<resume>` | The snapshot ID to resume uploading to. |
| --concurrency `<concurrency>` | The number of concurrent uploads to perform [default: 3]. |

### Examples

Expand Down
Loading