Merge pull request #10438 from ThomasWaldmann/repo-default-compression

repo-create: store default compression and chunker params in the repository, fixes #346

Author: ThomasWaldmann

docs/internals/data-structures.rst

@@ -39,6 +39,9 @@ of their content. The store hash is the unkeyed 256 bit BLAKE3 hash, see
 config/
   config
     the repository config (see :ref:`repo_config`), a text object
+  defaults
+    the repository defaults (see :ref:`repo_defaults`), in the key's store object
+    envelope (see below). Only present if ``borg repo-create`` was given a default.
   space-reserve.N
     purely random binary data to reserve space, e.g. for disk-full emergencies.
     These objects are created and removed by ``borg repo-space``.
@@ -110,8 +113,8 @@ locks/
 
 .. _store_object_envelope:
 
-The index fragments, the lock objects, ``checked-packs`` and the
-``referenced-by-archive.*`` objects are stored in the **store object envelope**: the repository key's ``encrypt()``,
+The index fragments, the lock objects, ``checked-packs``, the
+``referenced-by-archive.*`` objects and ``config/defaults`` are stored in the **store object envelope**: the repository key's ``encrypt()``,
 exactly as for the metadata and data slots of the objects in a pack (see
 :ref:`security_encryption`), with an empty id and an AAD of
 ``b"borg-store-object\0"`` followed by the repository id, the tag ``b"n"`` and the
@@ -129,7 +132,8 @@ packs; any other command that needs the chunks index aborts, except ``borg compa
 and ``borg repo-compress``, which rebuild it from the packs, as they rewrite the whole
 chunks index anyway (under an exclusive lock). A corrupted cache is ignored and
 rebuilt. A lock object that fails the authentication is treated as a foreign exclusive
-lock, see :ref:`storelocking`. The ``chunkindex-invalid`` marker has no content and is
+lock, see :ref:`storelocking`. Commands that use ``config/defaults`` abort if it fails
+the authentication. The ``chunkindex-invalid`` marker has no content and is
 stored as is.
 
 
@@ -308,6 +312,36 @@ a ``keys/`` object, see :ref:`key_files`), and the encryption mode and id hash
 are what ``borg repo-create`` was given (the key type byte of any repository
 object encodes them as well, see ``KeyType`` in ``constants.py``).
 
+.. _repo_defaults:
+
+Repository defaults
+~~~~~~~~~~~~~~~~~~~
+
+The ``config/defaults`` store object holds default values for command options,
+as a msgpacked dict mapping the option name to its value, both as strings::
+
+    {"compression": "zstd,3", "chunker_params": "fastcdc,19,23,21,2"}
+
+*compression* is the default compression spec (see ``borg help compression``),
+set by ``borg repo-create --compression``. The commands with a ``--compression``
+option use it if no compression was given via the command line, the environment or
+the default config file; without it, they use lz4.
+
+*chunker_params* are the default :ref:`chunker-params <chunker-params>`, set by
+``borg repo-create --chunker-params``. ``borg create`` and ``borg import-tar`` use
+them if no chunker params were given (in the same ways as above), ``borg recreate``
+and ``borg transfer`` for ``--chunker-params default``; without them, the built-in
+default chunker params are used.
+
+Each entry is optional, a missing entry means that the repository has no default
+for it.
+
+Unlike the repository config, which borg must read before it knows the key, the
+defaults are stored in the :ref:`store object envelope <store_object_envelope>`,
+so they are authenticated: an attacker with write access to the storage can not
+change them (e.g. remove an ``obfuscate`` compression) without being noticed. A
+repository without the object has no defaults.
+
 .. _archive:
 
 Archives

docs/internals/frontends.rst

@@ -345,7 +345,14 @@ path
     Path to the local repository cache
 
 :ref:`borg_repo-info` additionally emits a *security_dir* key with the path of the local security
-directory of the repository.
+directory of the repository and a *defaults* key with an object containing:
+
+compression
+    The compression spec the commands use if no compression is given: the repository default (see
+    :ref:`borg_repo-create` ``--compression``), else ``lz4``
+chunker_params
+    The chunker params the commands use if no chunker params are given: the repository default (see
+    :ref:`borg_repo-create` ``--chunker-params``), else the built-in default
 
 .. highlight: json
 
@@ -355,6 +362,10 @@ Example ``borg repo-info --json`` output::
         "cache": {
             "path": "/home/user/.cache/borg/65d7898e2142485f44506fb11c0fcd6d7dfd0341716385246068584a62632a94"
         },
+        "defaults": {
+            "chunker_params": "fastcdc,19,23,21,2",
+            "compression": "lz4"
+        },
         "encryption": {
             "encryption": "aes256-ocb",
             "id_hash": "sha256"

docs/quickstart.rst

@@ -352,6 +352,14 @@ specified algorithm::
 You'll need to experiment a bit to find the best compression for your use case.
 Keep an eye on CPU load and throughput.
 
+Instead of giving ``--compression`` to every command, you can set a default compression
+for the repository when creating it::
+
+    $ borg repo-create --encryption aes256-ocb --compression zstd,3
+
+Commands that compress data use this default when you do not give ``--compression``.
+``borg repo-create --chunker-params`` sets the default chunker parameters in the same way.
+
 .. _encrypted_repos:
 
 Repository encryption

src/borg/archiver/_common.py

@@ -10,8 +10,9 @@
 from ..helpers import CommandError, Error
 from ..helpers import SortBySpec, location_validator, Location, relative_time_marker_validator
 from ..helpers import FilesystemPathSpec
+from ..helpers import ChunkerParams, CompressionSpec
 from ..helpers import Highlander, octal_int
-from ..helpers.argparsing import SUPPRESS, PositiveInt
+from ..helpers.argparsing import SUPPRESS, ArgumentTypeError, PositiveInt
 from ..helpers.nanorst import rst_to_terminal
 from ..manifest import Manifest, AI_HUMAN_SORT_KEYS
 from ..patterns import PatternMatcher
@@ -80,6 +81,37 @@ def get_repository(
     return repository
 
 
+def _repository_default(repository, name, parse, builtin):
+    """Return the repository default for option name (see Repository.save_defaults), parsed, else builtin."""
+    value = repository.load_defaults().get(name) if isinstance(repository, Repository) else None
+    if value is None:
+        return builtin
+    try:
+        return parse(value)
+    except (ArgumentTypeError, ValueError) as err:
+        raise Repository.InvalidRepositoryConfig(
+            repository._location.canonical_path(), f"invalid default {name} {value!r}: {err}"
+        ) from None
+
+
+def default_compression(repository):
+    """Return the CompressionSpec a command uses if --compression was not given.
+
+    That is the repository default (set by "borg repo-create --compression"), else lz4.
+    An explicitly configured compression (command line, environment, default.yaml) always wins.
+    """
+    return _repository_default(repository, "compression", CompressionSpec, CompressionSpec(BUILTIN_COMPRESSION))
+
+
+def default_chunker_params(repository):
+    """Return the chunker params for "--chunker-params default" (the default of create and import-tar).
+
+    That is the repository default (set by "borg repo-create --chunker-params"), else CHUNKER_PARAMS.
+    Explicitly configured chunker params (command line, environment, default.yaml) always win.
+    """
+    return _repository_default(repository, "chunker_params", ChunkerParams, CHUNKER_PARAMS)
+
+
 def with_repository(
     create=False,
     lock=True,
@@ -152,7 +184,11 @@ def wrapper(self, args, **kwargs):
                     manifest_ = Manifest.load(repository, other=False, ro_cls=ro_cls)
                     kwargs["manifest"] = manifest_
                     if "compression" in args:
+                        if args.compression is None:  # not given, see default_compression()
+                            args.compression = default_compression(repository)
                         manifest_.repo_objs.compressor = args.compression.compressor
+                    if "chunker_params" in args and args.chunker_params == DEFAULT_CHUNKER_PARAMS:
+                        args.chunker_params = default_chunker_params(repository)
                     if secure:
                         assert_secure(repository, manifest_)
                 if cache:

src/borg/archiver/create_cmd.py

@@ -1333,20 +1333,23 @@ def build_parser_create(self, subparsers, common_parser, mid_common_parser):
             metavar="PARAMS",
             dest="chunker_params",
             type=ChunkerParams,
-            default=CHUNKER_PARAMS,
+            default=DEFAULT_CHUNKER_PARAMS,  # see default_chunker_params()
             action=Highlander,
             help="specify the chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, "
-            "HASH_MASK_BITS, NC_LEVEL). default: %s,%d,%d,%d,%d" % CHUNKER_PARAMS,
+            "HASH_MASK_BITS, NC_LEVEL). default: the repository default (see borg repo-create), "
+            "else %s,%d,%d,%d,%d" % CHUNKER_PARAMS,
         )
         archive_group.add_argument(
             "-C",
             "--compression",
             metavar="COMPRESSION",
             dest="compression",
             type=CompressionSpec,
-            default=CompressionSpec("lz4"),
+            default=None,  # None: not given, see default_compression()
             action=Highlander,
-            help="select compression algorithm, see the output of the " '"borg help compression" command for details.',
+            help="select compression algorithm, see the output of the "
+            '"borg help compression" command for details. '
+            "Default: the repository default (see borg repo-create), else lz4.",
         )
         archive_group.add_argument(
             "--tag",

src/borg/archiver/debug_cmd.py

@@ -462,9 +462,11 @@ def build_parser_debug(self, subparsers, common_parser, mid_common_parser):
             metavar="COMPRESSION",
             dest="compression",
             type=CompressionSpec,
-            default=CompressionSpec("lz4"),
+            default=None,  # None: not given, see default_compression()
             action=Highlander,
-            help="select compression algorithm, see the output of the " '"borg help compression" command for details.',
+            help="select compression algorithm, see the output of the "
+            '"borg help compression" command for details. '
+            "Default: the repository default (see borg repo-create), else lz4.",
         )
         subparser.add_argument(
             "object_path",

src/borg/archiver/help_cmd.py

@@ -487,15 +487,17 @@ class HelpMixIn:
         So if you use different compression specs for the backups, whichever stores a
         chunk first determines its compression. See also ``borg recreate``.
 
-        Compression is lz4 by default. If you want something else, you have to specify what you want.
+        If you do not specify a compression via ``--compression`` (or the environment or the
+        default config file), the repository's default compression is used, which can be set
+        with ``borg repo-create --compression``. Without a repository default, compression is lz4.
 
         Valid compression specifiers are:
 
         none
             Do not compress.
 
         lz4
-            Use lz4 compression. Very high speed, very low compression. (default)
+            Use lz4 compression. Very high speed, very low compression. (built-in default)
 
         zstd[,L]
             Use zstd ("zstandard") compression, a modern wide-range algorithm.

src/borg/archiver/recreate_cmd.py

@@ -159,13 +159,14 @@ def build_parser_recreate(self, subparsers, common_parser, mid_common_parser):
             metavar="COMPRESSION",
             dest="compression",
             type=CompressionSpec,
-            default=CompressionSpec("lz4"),
+            default=None,  # None: not given, see default_compression()
             action=Highlander,
             help="select compression algorithm, see the output of the "
             '"borg help compression" command for details. '
             "Only applies to newly written data, e.g. when re-chunking with --chunker-params "
             "(and to the new archive metadata); data chunks reused from the existing archive "
-            "are not recompressed, use borg repo-compress for that.",
+            "are not recompressed, use borg repo-compress for that. "
+            "Default: the repository default (see borg repo-create), else lz4.",
         )
         archive_group.add_argument(
             "--chunker-params",
@@ -178,7 +179,8 @@ def build_parser_recreate(self, subparsers, common_parser, mid_common_parser):
             "buzhash,CHUNK_MIN_EXP,CHUNK_MAX_EXP,HASH_MASK_BITS,WINDOW_SIZE or "
             "buzhash64,CHUNK_MIN_EXP,CHUNK_MAX_EXP,HASH_MASK_BITS,WINDOW_SIZE,NC_LEVEL or "
             "fastcdc,CHUNK_MIN_EXP,CHUNK_MAX_EXP,HASH_MASK_BITS,NC_LEVEL or "
-            "`default` to use the chunker defaults. default: do not rechunk",
+            "`default` to use the repository default (see borg repo-create), else the built-in "
+            "chunker defaults. default: do not rechunk",
         )
 
         subparser.add_argument(

src/borg/archiver/repo_compress_cmd.py

@@ -223,7 +223,9 @@ def build_parser_repo_compress(self, subparsers, common_parser, mid_common_parse
         Repository (re-)compression (and/or re-obfuscation).
 
         Reads all repository objects and recompresses the ones that are not already using
-        the compression type/level and obfuscation level given via ``--compression``.
+        the compression type/level and obfuscation level given via ``--compression``. Without
+        ``--compression``, that is the repository's default compression (see ``borg repo-create``),
+        else lz4.
 
         The repository is processed one pack file at a time: a pack is read as a whole and,
         if it holds objects that need recompression, rewritten as a whole - objects already
@@ -267,9 +269,10 @@ def build_parser_repo_compress(self, subparsers, common_parser, mid_common_parse
             metavar="COMPRESSION",
             dest="compression",
             type=CompressionSpec,
-            default=CompressionSpec("lz4"),
+            default=None,  # None: not given, see default_compression()
             action=Highlander,
-            help='select compression algorithm, see the output of the "borg help compression" command for details.',
+            help='select compression algorithm, see the output of the "borg help compression" command for details. '
+            "Default: the repository default (see borg repo-create), else lz4.",
         )
 
         subparser.add_argument("-s", "--stats", dest="stats", action="store_true", help="print statistics")

src/borg/archiver/repo_create_cmd.py

@@ -3,7 +3,7 @@
 from ..constants import *  # NOQA
 from ..crypto.key import key_creator, encryption_argument_names, id_hash_argument_names
 from ..helpers import CancelledByUser
-from ..helpers import location_validator, Location
+from ..helpers import location_validator, Location, ChunkerParams, CompressionSpec
 from ..hashindex import ChunkIndex
 from ..helpers.argparsing import ArgumentParser
 from ..manifest import Manifest
@@ -41,6 +41,13 @@ def do_repo_create(self, args, repository, *, other_repository=None, other_manif
             repository.acquire_lock()
             # writing the config is what makes the store a repository, see Repository.create().
             repository.save_config(key)
+            defaults = {}
+            if args.compression is not None:
+                defaults["compression"] = str(args.compression)
+            if args.chunker_params not in (None, DEFAULT_CHUNKER_PARAMS):
+                defaults["chunker_params"] = ",".join(str(p) for p in args.chunker_params)
+            if defaults:
+                repository.save_defaults(defaults)
             # we know repo/packs/ still does not have any chunks stored in it, but for some stores, there
             # might be a lot of empty directories and listing them all might be rather slow, so we better
             # store an empty ChunkIndex now, so that the first repo operation does not have to build the
@@ -191,6 +198,29 @@ def build_parser_repo_create(self, subparsers, common_parser, mid_common_parser)
         To normally work with ``authenticated-*`` repositories, you will need the passphrase, but
         there is an emergency workaround; see ``BORG_WORKAROUNDS=authenticated_no_key`` docs.
 
+        Repository defaults
+        +++++++++++++++++++
+
+        ``--compression`` sets the repository's default compression: the commands that compress
+        data (``borg create``, ``borg recreate``, ``borg import-tar``, ``borg transfer``,
+        ``borg repo-compress``) use it if no compression was given via ``--compression``, the
+        environment or the default config file (``default.yaml``). Without a repository default,
+        they use lz4. See ``borg help compression`` for the compression specs.
+
+        ``--chunker-params`` sets the repository's default chunker parameters: ``borg create`` and
+        ``borg import-tar`` use them if no chunker parameters were given (in the same ways as above).
+        ``borg recreate`` and ``borg transfer`` only rechunk if ``--chunker-params`` is given, with
+        ``--chunker-params default``, they rechunk to the repository's default chunker parameters.
+        Without a repository default, the built-in default chunker parameters are used.
+
+        This is useful if several clients back up into the same repository or if some commands are
+        run manually: they all compress and chunk the same way without having to give these options.
+        Using the same chunker parameters is important for deduplication.
+        ``borg repo-info`` shows the defaults.
+
+        The defaults are stored in the repository and protected by the repository key, so nobody without
+        the key can change them (e.g. remove an ``obfuscate`` compression) without being noticed.
+
         Creating a related repository
         +++++++++++++++++++++++++++++
 
@@ -263,6 +293,27 @@ def build_parser_repo_create(self, subparsers, common_parser, mid_common_parser)
             help="where to store the key: 'repokey' (in the repository, default) or 'keyfile' "
             "(in the local keys directory).",
         )
+        subparser.add_argument(
+            "-C",
+            "--compression",
+            metavar="COMPRESSION",
+            dest="compression",
+            type=CompressionSpec,
+            default=None,
+            action=Highlander,
+            help="set the default compression of the repository, see the output of the "
+            '"borg help compression" command for details. Default: no repository default (lz4 is used).',
+        )
+        subparser.add_argument(
+            "--chunker-params",
+            metavar="PARAMS",
+            dest="chunker_params",
+            type=ChunkerParams,
+            default=None,
+            action=Highlander,
+            help="set the default chunker parameters of the repository (same format as for borg create). "
+            "Default: no repository default (%s,%d,%d,%d,%d is used)." % CHUNKER_PARAMS,
+        )
         subparser.add_argument(
             "--copy-crypt-key",
             dest="copy_crypt_key",