feat: blocks backup / restore #3946

n0izn0iz · 2025-03-14T20:41:49Z

Description

Adds a grpc service called Backup in the tendermint2 node that allows to stream blocks efficiently
It has a single method StreamBlocks that take a start and end height. If end height is 0 it will stream to the latest height. It is disabled by default and require enabling it in the config.toml
Adds a contribs binary named tm2backup that pulls blocks from the backup service and store them in compressed 100-blocks files. It takes a start and end height as well as supporting resuming.
The tar format was chosen to bundle blocks since it's widely supported and efficient. The zstandard format was chosen for compression because it's fast, has a good compression ratio and is widely supported.
Adds a restore subcommand to the gnoland binary that allows to replay blocks from a backup. It takes the options from the start subcommand as well as the backup directory and an optional end height.
It will start at the current node height + 1.

The restore command can only restore at backupEndHeight-1 because I did not figure a way to commit block n without block n+1. I'd gladly take ideas on how to do that.

The backup is fast enough for now IMO (< 20min for test5 on my macbook) but can be optimized because it's not parallelized.
The restore bottleneck seems to be the gnovm currently but I would need to profile to be sure.

How to create a backup

Enable the backup service in your node's config.toml
```
[backup]

laddr = "localhost:4242"
```
(Re-)Start your node

Run the tm2backup command

cd contribs/tm2backup
tm2backup -o blocks-backup -remote http://localhost:4242

Example output:

(...)

How to create a node from a backup

Get the genesis file, for example:
```
wget https://example.com/genesis.json
```

Run the restore command

gnoland restore --lazy --backup-dir ../contribs/tm2backup/blocks-backup

Start your node
```
gnoland start
```

TODO

Signed-off-by: Norman <[email protected]>

Gno2D2 · 2025-03-14T20:43:02Z

🛠 PR Checks Summary

All Automated Checks passed. ✅

Manual Checks (for Reviewers):

IGNORE the bot requirements for this PR (force green CI check)

✅ Automated Checks (for Contributors):

🟢 Maintainers must be able to edit this pull request (more info)
🟢 Pending initial approval by a review team member, or review from tech-staff

☑️ Contributor Actions:

Fix any issues flagged by automated checks.
Follow the Contributor Checklist to ensure your PR is ready for review.
- Add new tests, or document why they are unnecessary.
- Provide clear examples/screenshots, if necessary.
- Update documentation, if required.
- Ensure no breaking changes, or include BREAKING CHANGE notes.
- Link related issues/PRs, where applicable.

☑️ Reviewer Actions:

Complete manual checks for the PR, including the guidelines and additional checks if applicable.

📚 Resources:

Debug

Automated Checks

Maintainers must be able to edit this pull request (more info)

If

🟢 Condition met
└── 🟢 And
    ├── 🟢 The base branch matches this pattern: ^master$
    └── 🟢 The pull request was created from a fork (head branch repo: n0izn0iz/gno)

Then

🟢 Requirement satisfied
└── 🟢 Maintainer can modify this pull request

Pending initial approval by a review team member, or review from tech-staff

If

🟢 Condition met
└── 🟢 And
    ├── 🟢 The base branch matches this pattern: ^master$
    └── 🟢 Not (🔴 Pull request author is a member of the team: tech-staff)

Then

🟢 Requirement satisfied
└── 🟢 If
    ├── 🟢 Condition
    │   └── 🟢 Or
    │       ├── 🔴 At least one of these user(s) reviewed the pull request: [jefft0 leohhhn n0izn0iz notJoon omarsy x1unix] (with state "APPROVED")
    │       ├── 🟢 At least 1 user(s) of the team tech-staff reviewed pull request
    │       └── 🔴 This pull request is a draft
    └── 🟢 Then
        └── 🟢 Not (🔴 This label is applied to pull request: review/triage-pending)

Manual Checks

**IGNORE** the bot requirements for this PR (force green CI check)
If
🟢 Condition met
└── 🟢 On every pull request
Can be checked by

Any user with comment edit permission

codecov · 2025-03-14T20:43:56Z

Codecov Report

❌ Patch coverage is 57.90698% with 181 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
tm2/pkg/bft/backup/v1/reader.go	0.00%	76 Missing ⚠️
tm2/pkg/bft/backup/v1/writer.go	55.37%	29 Missing and 25 partials ⚠️
tm2/pkg/bft/backup/v1/util.go	58.97%	9 Missing and 7 partials ⚠️
contribs/tm2backup/main.go	87.14%	6 Missing and 3 partials ⚠️
gnovm/pkg/gnolang/string_methods.go	33.33%	6 Missing and 2 partials ⚠️
tm2/pkg/bft/blockchain/reactor.go	79.31%	4 Missing and 2 partials ⚠️
tm2/pkg/bft/backup/backup_svc.go	89.47%	2 Missing and 2 partials ⚠️
.../backup/backuppb/backuppbconnect/backup.connect.go	85.71%	4 Missing ⚠️
tm2/pkg/bft/node/node.go	69.23%	3 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

Signed-off-by: Norman <[email protected]>

jefft0 · 2025-07-24T08:42:49Z

Hi @n0izn0iz . The CI checks in master are fixed now. Do you want to merge master and run the tests again?

n0izn0iz · 2025-07-24T09:18:31Z

it was up to date and the failing check is a flaky one I think. the gnokms merge from yesterday introduced conflicts that I will fix asap

Signed-off-by: Norman <[email protected]>

ajnavarro · 2025-12-02T11:11:05Z

contribs/tm2backup/testfile

ajnavarro · 2025-12-02T11:15:38Z

contribs/tm2backup/main.go

+			// XXX: we unmarshal the block which is remarshalled in the backup writer just after.
+			// This choice was made to have a more stable interface than bytes.
+			// If this proves to be a bottleneck, we should revist.


This is part of the bottleneck for sure, but 100% not the main problem. What if instead of using amino as the endpoint output, we use JSON Lines format? To reduce bandwidth usage, we can compress the HTTP data using gzip, and instead of marshalling-unmarshalling-marshalling-unmarshalling, we can directly save that JSON to the backup file.

ajnavarro · 2025-12-02T11:19:43Z

tm2/pkg/bft/backup/backup_svc.go

+
+// StreamBlocks implements backuppbconnect.BackupServiceHandler.
+func (b *backupServer) StreamBlocks(_ context.Context, req *connect.Request[backuppb.StreamBlocksRequest], stream *connect.ServerStream[backuppb.StreamBlocksResponse]) error {
+	startHeight := req.Msg.StartHeight


First of all, check that Msg and req are not nil

ajnavarro · 2025-12-02T11:28:23Z

tm2/pkg/bft/backup/backup_svc.go

+	for height := startHeight; height <= endHeight; height++ {
+		block := b.store.LoadBlock(height)
+		if block == nil {
+			return fmt.Errorf("block store returned nil block for height %d", height)
+		}
+
+		data, err := amino.Marshal(block)
+		if err != nil {
+			return err
+		}
+
+		if err := stream.Send(&backuppb.StreamBlocksResponse{Data: data}); err != nil {
+			return err
+		}
+	}


Some things to comment on this:

Fetching blocks one by one from the K/V storage is extremely slow. We need to use iterators for that.

We are getting byte arrays from the K/V storage, the LoadBlock method is unmarshalling that to a Block struct, after that we marshal again blocks to a byte array. After that, we unmarshal a StreamBlocksResponse into protobuf... That's a lot of unnecessary processing.

ajnavarro · 2025-12-02T11:39:15Z

tm2/pkg/bft/blockchain/reactor.go

+		if err := state.Validators.VerifyCommit(
+			chainID, firstID, first.Height, second.LastCommit); err != nil {
+			return fmt.Errorf("invalid commit (%d:%X): %w", first.Height, first.Hash(), err)
+		}


Can we add a flag to skip commit verification? If we trust the backup, it can improve import speed.

ajnavarro · 2025-12-02T11:48:11Z

tm2/pkg/bft/blockchain/reactor.go

+			return fmt.Errorf("invalid commit (%d:%X): %w", first.Height, first.Hash(), err)
+		}
+
+		bcR.store.SaveBlock(first, firstParts, second.LastCommit)


We have to use insert Batches here. It will be an order of magnitude faster.

jefft0 · 2025-12-10T15:45:10Z

Reviewed by core dev ajnavarro. Ready for review by other core devs. (Still need to fix CI checks.)

Villaquiranm · 2025-12-10T17:08:58Z

Reviewed by core dev ajnavarro. Ready for review by other core devs. (Still need to fix CI checks.)

I will work on top of this PR instead of @n0izn0iz
new PR is here: #4950

n0izn0iz added 7 commits March 6, 2025 22:37

feat: backup command

984a889

Signed-off-by: Norman <[email protected]>

feat: use zstd instead of xz

f1a52c8

Signed-off-by: Norman <[email protected]>

chore: move to contribs

ddcb274

Signed-off-by: Norman <[email protected]>

chore: remove worklog

600f085

Signed-off-by: Norman <[email protected]>

chore: revert mischange

47b5677

Signed-off-by: Norman <[email protected]>

chore: improve backupsvc files

ad2c146

Signed-off-by: Norman <[email protected]>

chore: refacto backup service setup

e3ffb38

Signed-off-by: Norman <[email protected]>

github-project-automation bot added this to 💪 Bounties & Worx and 🧙‍♂️Gno.land development Mar 14, 2025

github-project-automation bot moved this to Triage in 🧙‍♂️Gno.land development Mar 14, 2025

github-actions bot assigned n0izn0iz Mar 14, 2025

github-actions bot added the 📦 🌐 tendermint v2 Issues or PRs tm2 related label Mar 14, 2025

Gno2D2 requested a review from a team March 14, 2025 20:42

n0izn0iz and others added 9 commits March 14, 2025 21:45

fix: changed paths and go mod tidy

6bc40e9

Signed-off-by: Norman <[email protected]>

Merge branch 'master' into backup-restore

af010d7

chore: fail earlier

43cc310

Signed-off-by: Norman <[email protected]>

chore: don't crash if backup config does not existsé

79605c6

Signed-off-by: Norman <[email protected]>

fix: prevent slowloris

a99603f

Signed-off-by: Norman <[email protected]>

chore: format

aae47a3

Signed-off-by: Norman <[email protected]>

fix: populate backup section default

69a2e64

Signed-off-by: Norman <[email protected]>

chore: format

f102c59

Signed-off-by: Norman <[email protected]>

chore: better log for backup server stop

381bdd8

Signed-off-by: Norman <[email protected]>

zivkovicmilos requested review from sw360cab and zivkovicmilos and removed request for a team March 14, 2025 22:31

n0izn0iz changed the title ~~feat: backup / restore~~ feat: blocks backup / restore Mar 14, 2025

n0izn0iz added 2 commits March 15, 2025 08:18

chore: use constant

82982d9

Signed-off-by: Norman <[email protected]>

feat: restore command

585d637

Signed-off-by: Norman <[email protected]>

n0izn0iz and others added 3 commits July 21, 2025 17:02

Merge branch 'master' into backup-restore

3316110

chore: go mod tidy

0cc0217

Signed-off-by: Norman <[email protected]>

Merge branch 'master' into backup-restore

4cc740d

n0izn0iz and others added 5 commits July 30, 2025 15:00

chore: Merge remote-tracking branch 'origin/master' into backup-restore

f5bf1ed

fix: skip genesis sig verif in restore test

ddbde81

Signed-off-by: Norman <[email protected]>

fix: skip genesis sig verif in CreateNode test

6638b0e

Signed-off-by: Norman <[email protected]>

Merge branch 'master' into backup-restore

1b737a4

Merge branch 'master' into backup-restore

2024085

Kouteki added this to the ⏭️Next after mainnet beta milestone Aug 4, 2025

sw360cab requested a review from ajnavarro September 3, 2025 12:43

n0izn0iz and others added 8 commits October 22, 2025 16:37

chore: Merge remote-tracking branch 'origin/master' into backup-restore

06b906d

chore: allow to pass nil logger in writer

df48ab9

Signed-off-by: Norman <[email protected]>

fix: don't overwrite start height after resume

38e4856

Signed-off-by: Norman <[email protected]>

chore: commit test

dc26e23

Signed-off-by: Norman <[email protected]>

Merge branch 'master' into backup-restore

a800366

fix: bump go version

733a491

Signed-off-by: Norman <[email protected]>

chore: regen

131a256

Signed-off-by: Norman <[email protected]>

chore: regen

39b880f

Signed-off-by: Norman <[email protected]>

Kouteki removed the request for review from a team November 27, 2025 12:36

Merge branch 'master' into backup-restore

cb5e656

ajnavarro requested changes Dec 2, 2025

View reviewed changes

github-project-automation bot moved this to In Progress in 💪 Bounties & Worx Dec 2, 2025

Villaquiranm mentioned this pull request Dec 8, 2025

feat: blocks backup / restore #4950

Open

jefft0 removed the review/triage-pending PRs opened by external contributors that are waiting for the 1st review label Dec 10, 2025

zxxma assigned omarsy and Villaquiranm Dec 12, 2025

feat: blocks backup / restore #3946

Are you sure you want to change the base?

feat: blocks backup / restore #3946

Uh oh!

Conversation

n0izn0iz commented Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How to create a backup

How to create a node from a backup

TODO

Uh oh!

Gno2D2 commented Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛠 PR Checks Summary

Manual Checks (for Reviewers):

✅ Automated Checks (for Contributors):

☑️ Contributor Actions:

☑️ Reviewer Actions:

📚 Resources:

If

Then

If

Then

If

Can be checked by

Uh oh!

codecov bot commented Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jefft0 commented Jul 24, 2025

Uh oh!

n0izn0iz commented Jul 24, 2025

Uh oh!

ajnavarro Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

ajnavarro Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ajnavarro Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

ajnavarro Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

ajnavarro Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

ajnavarro Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

jefft0 commented Dec 10, 2025

Uh oh!

Villaquiranm commented Dec 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

n0izn0iz commented Mar 14, 2025 •

edited

Loading

Gno2D2 commented Mar 14, 2025 •

edited

Loading

codecov bot commented Mar 14, 2025 •

edited

Loading

ajnavarro Dec 2, 2025 •

edited

Loading