Skip to content

feat: new higher-performance rnote file format - #1578

Open
anesthetice wants to merge 29 commits into
flxzt:mainfrom
anesthetice:re-file
Open

feat: new higher-performance rnote file format#1578
anesthetice wants to merge 29 commits into
flxzt:mainfrom
anesthetice:re-file

Conversation

@anesthetice

@anesthetice anesthetice commented Nov 2, 2025

Copy link
Copy Markdown
Contributor

The overall goal of this PR is to improve the handling of .rnote save files.

tl;dr

  • Shorter loading times (~10x improvement*)
  • Shorter saving times (~6x improvement*)
  • Smaller file sizes (~12% reduction)

* This improvement factor only concerns going back and forth between the already loaded bytes of the file and their corresponding EngineSnapshot, on my system of course, your results may vary.

image

Main changes:

  • Parallelized (de-)serialization and (de-)compression.
  • Files saved with a compatible version are directly converted into an EngineSnapshot instead of always passing through a compatibility layer.
  • Swapped the Deflate (gzip) compression algorithm for Zstandard (zstd).

Here's a diagram of the simplified process to get from EngineSnapshot to a .rnote save file and vice-versa.

diagram

Note that this PR is a re-write of #1177, some of the important changes being:

  • (+) parallelism for serialization and compression
  • (+) the introduction of a file version separate from the version of Rnote
  • (-) scrapped save preferences (not worth complexity cost)
  • (-) scrapped alternative serialization methods (for compatibility reasons)

@anesthetice anesthetice changed the title Re: new rnote file format, zstd compression, serialization parallelism™, atomic file saving Re: new rnote file format with improved handling (parallelism and zstd), atomic file saving Nov 16, 2025
@anesthetice
anesthetice marked this pull request as ready for review November 19, 2025 19:21
@anesthetice

Copy link
Copy Markdown
Contributor Author

@flxzt, @Doublonmousse, I'd say this PR is ready for review. Take all the time you need of course, and feel free to ask me for information, changes, etc. 🫡

@anesthetice anesthetice changed the title Re: new rnote file format with improved handling (parallelism and zstd), atomic file saving re: New higher-performance rnote file format, and atomic file saving Nov 23, 2025
Comment thread crates/rnote-engine/src/fileformats/rnoteformat/prelude.rs Outdated
Comment thread crates/rnote-engine/src/fileformats/rnoteformat/v1/save.rs
Comment thread crates/rnote-engine/src/fileformats/rnoteformat/v1/save.rs
Comment thread crates/rnote-engine/src/fileformats/rnoteformat/mod.rs
Comment thread crates/rnote-engine/src/fileformats/rnoteformat/v1/load.rs Outdated
Comment thread crates/rnote-engine/src/fileformats/rnoteformat/v1/load.rs Outdated
Comment thread crates/rnote-engine/src/fileformats/rnoteformat/v1/load.rs
Comment thread crates/rnote-engine/src/fileformats/rnoteformat/v1/mod.rs
@Doublonmousse

Copy link
Copy Markdown
Collaborator

Another thought on the atomic saving denomination.
I'm not sure we can really call the new version atomic saving per say.

That would be the case for a file format where any change we do inside the file is mapped back to the file through an efficient diff (and not a full write of the new engine state). For sqlite (#1173 (comment)), that would be the case because modifications to the file can be mapped back to the file through a push/commit

@anesthetice

Copy link
Copy Markdown
Contributor Author

Another thought on the atomic saving denomination. I'm not sure we can really call the new version atomic saving per say.

That would be the case for a file format where any change we do inside the file is mapped back to the file through an efficient diff (and not a full write of the new engine state). For sqlite (#1173 (comment)), that would be the case because modifications to the file can be mapped back to the file through a push/commit

I have seen "atomic saving" be used to mean somewhat different things, but for the most part it seems to mean what we are doing here (Geany wiki, 15 year old stackoverflow thread, etc...). A more efficient saving mechanism using diffs might be interesting, but would probably be very complex to implement and then have to deal with forward- and backward-compatibility...

That being said, we should maybe tinker a bit more with the implementation as for instance metadata, permissions, and symlinks are not handled...

@anesthetice

Copy link
Copy Markdown
Contributor Author

@Doublonmousse Apologies for the delay, let me know what you think

Comment thread crates/rnote-engine/src/utils.rs Outdated
@anesthetice anesthetice changed the title re: New higher-performance rnote file format, and atomic file saving re-feat: new higher-performance rnote file format Mar 1, 2026
@anesthetice

Copy link
Copy Markdown
Contributor Author

Atomic FS was split-off into it's own branch (+PR), see #1678

@anesthetice anesthetice changed the title re-feat: new higher-performance rnote file format feat: new higher-performance rnote file format Mar 26, 2026
@NXTler

This comment was marked as off-topic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants