Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions .claude/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,13 +131,13 @@ so the repository storage layer never sees plaintext or has any notion of files/
- **`repository.py`** — `Repository`, a key→value object store built on top of **borgstore**
(`borgstore.store.Store`). Keys are 32-byte object IDs; the repo doesn't know what an "archive" or
"file" is. borgstore provides the actual storage backends and transport: local filesystem,
`ssh://` (borgstore ssh backend, optionally reusing `BORGSTORE_RSH`), and `rest://` (borgstore REST
server, served by `borg serve --rest`). This replaces the borg 1.x segment/journal format — read
`ssh://` (borgstore REST backend talking HTTP over stdio over ssh to a remote `borg serve`,
optionally reusing `BORGSTORE_RSH`), `sftp://`, `http(s)://`, `rclone:`, `s3:`/`b2:`. This replaces the borg 1.x segment/journal format — read
`docs/internals/data-structures.rst` before touching repository/object/format code.
- **`archiver/serve_cmd.py`** — `borg serve` provides the server side for remote repositories: the
default mode serves a repository over stdio to a connecting borgstore ssh backend, and
`borg serve --rest` runs the borgstore REST server. (There is no separate `remote.py` / custom RPC
layer anymore — borgstore owns the client/server transport.)
- **`archiver/serve_cmd.py`** — `borg serve` provides the server side for `ssh://` repositories:
`borg serve --rest` runs the borgstore REST server on stdio for a current repository, plain
`borg serve` serves a legacy borg 1.x repository via the legacy RPC protocol (`--from-borg1`,
`legacy/remote.py`).
- **`crypto/key.py`, `crypto/low_level.pyx`** — encryption/authentication (AEAD: AES-256-OCB or
ChaCha20-Poly1305; ids via HMAC-SHA256 or BLAKE3) and key file handling (keyfile vs repokey
storage). The `id_hash` used for deduplication depends on the key/encryption mode.
Expand Down Expand Up @@ -168,7 +168,7 @@ so the repository storage layer never sees plaintext or has any notion of files/
progress reporting, filesystem helpers); check here before writing a new utility, it may exist.

**Client/server split:** most `do_*` archiver commands operate against a local or remote `Repository`
transparently — the choice of local vs. `ssh://`/`rest://` is resolved by `Location` parsing in
transparently — the choice of local vs. `ssh://` etc. is resolved by `Location` parsing in
`helpers/` and borgstore backend selection, and doesn't otherwise change command logic. `borg serve`
on the remote end provides the server side.

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -282,7 +282,7 @@ jobs:
sudo chmod o+x "$HOME"
# Export SFTP test URL for tox via GITHUB_ENV
echo "BORG_TEST_SFTP_REPO=sftp://sftpuser@localhost:22/borg/sftp-repo" >> $GITHUB_ENV
echo "BORG_TEST_REST_REPO=rest://sftpuser@localhost:22/borg/rest-repo" >> $GITHUB_ENV
echo "BORG_TEST_REST_REPO=ssh://sftpuser@localhost:22/borg/rest-repo" >> $GITHUB_ENV

- name: Install and start moto S3 server (test only)
# The S3 tests run against moto in server mode: it comes from PyPI (pinned in
Expand Down
2 changes: 1 addition & 1 deletion README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ Main features
**Off-site backups**
Borg can store data on any remote host accessible via misc. protocols:

* ``rest://`` (REST-http-over-stdio-over-ssh) and ``http(s)://``
* ``ssh://`` (REST-http-over-stdio-over-ssh) and ``http(s)://``
(REST-http-over-tcp). Significant performance gains can be achieved with
these by having a remote agent (borg must be installed on the repo server).
* ``sftp://``
Expand Down
5 changes: 4 additions & 1 deletion docs/changes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,9 @@ Breaking changes (you must create new repos for b25):
the authenticated modes, #9819, #10235.
- drop OBJ_VERSION_NO_HEADER_AAD (pack object format v1) support, #9973
- KeyType: renumber the authenticated key types to 0x50 / 0x60
- ssh:// repository URLs access current repositories via REST over ssh
(served by a remote "borg serve"), #9765. With --from-borg1, they access
legacy borg 1.x repositories via the legacy protocol (as before).

New features:

Expand All @@ -197,7 +200,7 @@ Fixes:

- crypto: the AEAD ciphers (AES-OCB, ChaCha20-Poly1305) feed their input to OpenSSL in
<= 1GB chunks, to overcome the 32bit size limit of the OpenSSL API.
- repository: raise DoesNotExist for a missing rest:// repo, #10365
- repository: raise DoesNotExist for a missing ssh:// repo, #10365
- compact:

- build the chunk index once, not three times
Expand Down
9 changes: 4 additions & 5 deletions docs/deployment/central-backup-server.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,10 @@ clients.

.. note::

This example predates Borg 2 and uses the legacy ``ssh://`` transport (served
by ``borg serve``) and ``borg init``. With Borg 2, the ``ssh://`` transport is
only used for legacy borg 1.x (v1) repositories; for current repositories use a
``rest://`` repository instead (Borg connects via ssh and runs a borgstore REST
server on the remote host), and use ``borg repo-create`` instead of ``borg init``.
This example predates Borg 2 and uses ``borg init`` and ``borg serve``. With
Borg 2, use ``borg repo-create`` instead of ``borg init``, and ``borg serve --rest``
instead of ``borg serve`` in the forced command (for ``ssh://`` repositories,
Borg connects via ssh and runs ``borg serve --rest`` on the remote host).

Machines
--------
Expand Down
16 changes: 8 additions & 8 deletions docs/deployment/hosting-repositories.rst
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,11 @@ files they will be included in this set.

The ``command`` keyword forces execution of the specified command
upon login. This must be ``borg serve --rest``: ``--rest`` serves a current
repository (the server side of a ``rest://`` repository URL), while ``borg serve``
without it serves a legacy borg 1.x repository using the legacy RPC protocol.
The client cannot supply ``--rest`` itself - the mode is deliberately pinned by
the forced command - so a forced command without ``--rest`` locks out all clients
using current (``rest://``) repositories.
repository (the server side of an ``ssh://`` repository URL), while ``borg serve``
without it serves a legacy borg 1.x repository using the legacy RPC protocol
(used by ``--from-borg1``). The client cannot supply ``--rest`` itself - the mode
is deliberately pinned by the forced command - so a forced command without
``--rest`` locks out all clients using current repositories.

The ``--restrict-to-repository`` option permits access to exactly **one**
repository. It can be given multiple times to permit access to more than
Expand All @@ -77,12 +77,12 @@ come from the forced command.
The user therefore selects the repository through the path in the repository
URL::

borg -r rest://<user>@<host>/repository repo-list
borg -r ssh://<user>@<host>/repository repo-list

A path with a single leading slash is relative to the directory ssh logs into
(the user's home directory), so with the forced command shown above,
``rest://<user>@<host>/repository`` and
``rest://<user>@<host>//home/<user>/repository`` address the same repository.
``ssh://<user>@<host>/repository`` and
``ssh://<user>@<host>//home/<user>/repository`` address the same repository.

Refer to the `sshd(8) <https://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man8/sshd.8>`_
man page for more details on SSH options.
Expand Down
18 changes: 9 additions & 9 deletions docs/deployment/pull-backup.rst
Original file line number Diff line number Diff line change
Expand Up @@ -262,7 +262,7 @@ standard input and output available to a unix socket::
Socat will wait until a connection is opened. Then socat will execute the
command given, redirecting Standard Input and Output to the unix socket.
``--rest`` and ``--backend`` are required here: ``--rest`` serves a current
repository (talking HTTP over stdio, the server side of a ``rest://`` repository
repository (talking HTTP over stdio, the server side of an ``ssh://`` repository
URL), and ``--backend`` selects the repository to serve. ``--restrict-to-path``
is not necessary but a sane default.

Expand All @@ -271,7 +271,7 @@ is not necessary but a sane default.
(see ``BORG_RSH`` below) is discarded - including the ``--backend`` option the
client would normally use to tell the server which repository it wants. The
repository therefore is the one given by ``--backend`` in the socat command
above, no matter which path the client uses in its ``rest://`` URL. Use the
above, no matter which path the client uses in its ``ssh://`` URL. Use the
same path in both places to avoid confusion.

.. note::
Expand Down Expand Up @@ -322,7 +322,7 @@ SSH command. Those arguments can not be handled by socat. We wrap the command wi
All Borg commands can now be executed on *borg-client*. For example to create a
backup execute the ``borg create`` command::

borg-client:~$ borg create --repo rest://borg-server//path/to/repo archive /path_to_backup
borg-client:~$ borg create --repo ssh://borg-server//path/to/repo archive /path_to_backup

When automating backup creation, the
interactive ssh session may seem inappropriate. An alternative way of creating
Expand All @@ -333,7 +333,7 @@ a backup may be the following command::
borgc@borg-client \
BORG_RSH="sh -c 'exec socat STDIO UNIX-CONNECT:/run/borg/reponame.sock'" \
borg create \
--repo rest://borg-server//path/to/repo archive /path_to_backup \
--repo ssh://borg-server//path/to/repo archive /path_to_backup \
';' rm /run/borg/reponame.sock

This command also automatically removes the socket file after the ``borg
Expand Down Expand Up @@ -406,7 +406,7 @@ Initiating borg command execution from *borg-server* (e.g. repo-create)::
eval $(ssh-agent) > /dev/null
ssh-add -q ~/.ssh/borg-client_key
echo 'your secure borg key passphrase' | \
ssh -A -o StrictHostKeyChecking=no borgc@borg-client "BORG_PASSPHRASE=\$(cat) BORG_RSH='ssh -o StrictHostKeyChecking=no' borg repo-create --encryption aes256-ocb --key-location repokey -r rest://borgs@borg-server/repo"
ssh -A -o StrictHostKeyChecking=no borgc@borg-client "BORG_PASSPHRASE=\$(cat) BORG_RSH='ssh -o StrictHostKeyChecking=no' borg repo-create --encryption aes256-ocb --key-location repokey -r ssh://borgs@borg-server/repo"
kill "${SSH_AGENT_PID}"
)

Expand All @@ -432,13 +432,13 @@ Parentheses are not needed when using a dedicated bash process.
* The keys meant to be loaded into the agent must be specified explicitly, not from default locations.
* The *borg-client*'s entry in *borgs@borg-server:~/.ssh/authorized_keys* must be as restrictive as possible.

``echo 'your secure borg key passphrase' | ssh -A -o StrictHostKeyChecking=no borgc@borg-client "BORG_PASSPHRASE=\$(cat) BORG_RSH='ssh -o StrictHostKeyChecking=no' borg repo-create --encryption aes256-ocb --key-location repokey -r rest://borgs@borg-server/repo"``
``echo 'your secure borg key passphrase' | ssh -A -o StrictHostKeyChecking=no borgc@borg-client "BORG_PASSPHRASE=\$(cat) BORG_RSH='ssh -o StrictHostKeyChecking=no' borg repo-create --encryption aes256-ocb --key-location repokey -r ssh://borgs@borg-server/repo"``

Run the *borg repo-create* command on *borg-client*.

*rest://borgs@borg-server/repo* refers to the repository *repo* within borgs's home directory on *borg-server*
*ssh://borgs@borg-server/repo* refers to the repository *repo* within borgs's home directory on *borg-server*
(a path given with a single leading slash is relative to the directory ssh logs into; use
*rest://borgs@borg-server//path/to/repo* for an absolute path).
*ssh://borgs@borg-server//path/to/repo* for an absolute path).

Borg on *borg-client* connects via ssh (using *BORG_RSH*) and runs ``borg serve --rest`` on
*borg-server*, which the forced command in *borgs@borg-server:~/.ssh/authorized_keys* pins down.
Expand Down Expand Up @@ -490,7 +490,7 @@ using ``localhost`` instead of ``mybackup``

2. On machine ``myclient``

``borg create -v --progress --stats -r rest://backup@localhost:8022//home/backup/repos/myclient system /``
``borg create -v --progress --stats -r ssh://backup@localhost:8022//home/backup/repos/myclient system /``

Make sure to use port ``8022`` and ``localhost`` for the repository as this instructs borg on ``myclient`` to use the
remote forwarded ssh connection.
Expand Down
8 changes: 4 additions & 4 deletions docs/faq.rst
Original file line number Diff line number Diff line change
Expand Up @@ -860,7 +860,7 @@ Please see the next question.
Why does Borg disconnect or hang when backing up to a remote server?
--------------------------------------------------------------------

Communication with the remote server (e.g. using a ``rest://user@host/path`` repo URL)
Communication with the remote server (e.g. using an ``ssh://user@host/path`` repo URL)
is tunneled through an SSH connection. This can lead to some issues that would not
occur during a local backup:

Expand Down Expand Up @@ -1142,7 +1142,7 @@ borgstore, which Borg uses for repository access, can limit the transfer rate::

Pros:

- works for all backends (``sftp://``, ``rest://``, ``rclone:``, ``s3://``, ...).
- works for all backends (``sftp://``, ``ssh://``, ``rclone:``, ``s3://``, ...).
- limits both directions.
- needs no additional software.

Expand All @@ -1163,7 +1163,7 @@ backend call.
Using pv on the ssh connection
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For ``rest://`` repositories, Borg connects via ssh, so the transfer can be
For ``ssh://`` repositories, Borg connects via ssh, so the transfer can be
limited with pipeviewer_. Put a ``pv`` on each side of the connection using an
ssh ``ProxyCommand`` (this needs ``nc``), e.g. in ``~/.ssh/config``::

Expand All @@ -1189,7 +1189,7 @@ Pros:

Cons:

- only works for ``rest://`` repositories. ``sftp://`` does not run an external
- only works for ``ssh://`` repositories. ``sftp://`` does not run an external
ssh command (borgstore uses paramiko and creates the connection itself) and
a ``ProxyCommand`` in ``~/.ssh/config`` is not used there.
- needs ``pv`` and ``nc``.
Expand Down
19 changes: 9 additions & 10 deletions docs/internals/security.rst
Original file line number Diff line number Diff line change
Expand Up @@ -402,10 +402,10 @@ used:

Authorization and transport security come from the transport, not from borg.

Remote repositories over SSH: ``rest://``
Remote repositories over SSH: ``ssh://``
-----------------------------------------

For a ``rest://`` repository, borg runs ``borg serve --rest --backend FILE:<path>``
For an ``ssh://`` repository, borg runs ``borg serve --rest --backend FILE:<path>``
on the remote machine and speaks HTTP to it over that process' *stdin/stdout* -- not
over a socket. If the URL contains a host, the process is started through the
system's SSH client (honouring ``BORG_RSH`` / ``BORGSTORE_RSH`` for the ssh command
Expand Down Expand Up @@ -452,7 +452,7 @@ its dependencies, so their security properties are those of the respective libra
- ``http(s)://`` talks to a borgstore REST server over plain HTTP, authenticating
with HTTP Basic auth taken from the URL or from ``BORGSTORE_REST_USERNAME`` /
``BORGSTORE_REST_PASSWORD``. Basic auth sends the credentials to the server on
every request, so use ``https`` if you use this at all. ``rest://`` over SSH needs
every request, so use ``https`` if you use this at all. ``ssh://`` needs
no such credentials.

In every case the repository never receives the borg key or the passphrase, so a
Expand All @@ -465,11 +465,10 @@ Legacy borg 1.x RPC protocol

``borg serve`` *without* ``--rest`` still speaks the borg 1.x RPC protocol:
msgpack'd messages exchanged over stdio (``borg.legacy.remote``). It exists only so
that borg 2 can *read* a borg 1.x repository through an ``ssh://`` URL, e.g. for
``borg transfer --from-borg1``; ``ssh://`` is rejected for current repositories, and
this protocol cannot serve one. Its transport is the system's SSH client, so the same
"authorization and transport security are SSH's" reasoning as for ``rest://``
applies.
that borg 2 can *read* a borg 1.x repository through an ``ssh://`` URL with
``--from-borg1``, e.g. for ``borg transfer --from-borg1``; it cannot serve a current
repository. Its transport is the system's SSH client, so the same
"authorization and transport security are SSH's" reasoning as for REST applies.

Within that protocol, critical vulnerabilities such as remote code execution are
inhibited by its design:
Expand Down Expand Up @@ -526,8 +525,8 @@ while libssl implements TLS and related protocols.
The latter historically contained most vulnerabilities, especially critical ones, and Borg's own
extension modules do not use it: they link ``libcrypto`` only. Note that libssl can still be
reached through Python's ``ssl`` module by the transports that do networking inside the borg
process, i.e. ``s3:``/``b2:`` and ``http(s)://`` (see :ref:`remote_access_security`); ``rest://``
and the legacy ``ssh://`` protocol do not need it, as they only talk to a subprocess over pipes.
process, i.e. ``s3:``/``b2:`` and ``http(s)://`` (see :ref:`remote_access_security`); ``ssh://``
(REST and the legacy protocol) does not need it, as it only talks to a subprocess over pipes.
Accordingly, the binaries released by the project do include Python's ``ssl``/``_ssl`` modules
(they are needed by pyfuse3/trio as well).

Expand Down
2 changes: 1 addition & 1 deletion docs/misc/asciinema/demo.tcl
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ ls ~/Documents
# Backups are stored in a "repository". Let's tell borg where ours is,
# so we do not have to repeat it in every command:
export BORG_REPO=/media/backup/borgdemo
# It could also live on another machine, e.g. rest://user@server/backup.
# It could also live on another machine, e.g. ssh://user@server/backup.

# Creating the repository - encrypted and authenticated, with the key stored in the repo itself:
borg repo-create --encryption=aes256-ocb --key-location=repokey
Expand Down
Loading
Loading