Add UNIX domain socket transport

[PhrozenByte]

/kind enhancement

When creating full system backups one must usually run Borg as root to avoid permission issues. However, it might be undesirable to store these backups as root, as people often prefer storing backups in known places (like /var/backups) owned by dedicated users (like the default backup user in Debian-based distributions).

Right now there are only two possible solutions to accomplish this:

• One might chown the repo after borg create et al. exits
• One might abuse Borg's remote repository feature to connect to localhost using SSH as different user (as suggested by @ThomasWaldmann, see references below)

IMO this is a reasonable use case and Borg should support it without taking such rather desperate solutions.

Thus I'd like to suggest adding a socket:// transport (like socket:///run/borg/borg.sock) to Borg. The socket should be created by borg serve by accepting an additional --socket option with a path (e.g. borg serve --socket /run/user/1000/borg/borg.sock, umask 0117). How borg serve is being invoked is up to the user. borg create et al. can now connect to this socket using the socket:// transport (e.g. borg create socket:///run/borg/borg.sock /path/to/backup).

Even though this is not the goal, it also makes https://github.com/borgbackup/borg/blob/master/docs/deployment/pull-backup.rst#socat easier by eliminating socat. It might also enable some more use cases I just can't think of right now.

As I don't know Borg's sources I can't really judge the complexity, but since Borg already supports remote transports with borg serve, this could be as easy as replacing sys.stdin/sys.stdout of borg serve with a UNIX domain socket and to let borg create et al. communicate with this socket instead of the ssh subprocess.

References

For non-ssh repos and running borg as root, but not having the repo owned by root, you can use the ssh://user@localhost/myrepo trick.

- #4082 (comment)

You can work around that problem by using borg with user@localhost:/path/to/repo as repo (in that scenario, borg client can run as root and borg serve would run as user.

- #3587 (comment)

[ThomasWaldmann]

Sounds good, thanks for the detailled suggestion!

[ThomasWaldmann]

https://docs.python.org/3/library/socket.html

The code in borg.remote uses FDs (of stdin, stdout, stderr) and os.select/read/write/set_blocking.

It seems like select() and setblocking() can also be used with sockets, so would it be just send/recv instead of `write/read? How about error / exception handling?

The remote code is rather complex and not very nice to work with, I'ld like to avoid additional complexity or risks there.

[RonnyPfannschmidt]

Sockets can be turned into file objects for feature parity using the makefile api.

https://docs.python.org/3/library/socket.html#socket.socket.makefile

If those readable / writable objects can be used, it may be possible without pain.

The only caveat would be that the socket server would have a one client at a time limit for lock parity.

[ThomasWaldmann]

But socket.makefile returns a python file object, not a OS-level FD (int).

[RonnyPfannschmidt]

in the case of socket fd integers - on unix read/write are equivalent to send/recv without flags

[PhrozenByte]

@ThomasWaldmann You can call fileno() on the file object to get the fd from a socket.makefile() (Popen.stdin and Popen.stdout are file objects, too). As pointed out by @RonnyPfannschmidt, there should be no differences on Unix-like systems. Windows is a whole different story though, but since Windows isn't supported anyway, this should be no major issue. However, I must admit that I personally never used socket.makefile(), but only the high-level socket methods...

Thanks to #6270 giving me a nice starting point I just started looking into Borg's remote code and I agree that the current implementation is indeed not so easy to work with. The code very much confused me, especially whether borg serve relies on stderr. If borg serve relies on stderr, this is going to be a big issue - because with a Unix socket, you just don't have the separation of stdout and stderr, but only a single bidirectional channel. There are solutions, but it isn't as easy as initially thought then...

However, looking at the code, borg serve doesn't seem to use stderr. Even though it defines a stderr_fd, it only uses it for a single error, but isn't set to stderr anyway, but stdout, and won't be sent over the wire anyway. So, unless I'm missing something, borg serve doesn't use stderr.

borg/src/borg/remote.py

Lines 198 to 201 in 1e1c922

	def serve(self):
	stdin_fd = sys.stdin.fileno()
	stdout_fd = sys.stdout.fileno()
	stderr_fd = sys.stdout.fileno()

What confuses me about this is that the "client" part, i.e. the Popen call, is differentiating between stdout and stderr:

borg/src/borg/remote.py

Lines 589 to 592 in 1e1c922

	self.p = Popen(borg_cmd, bufsize=0, stdin=PIPE, stdout=PIPE, stderr=PIPE, env=env, preexec_fn=ignore_sigint)
	self.stdin_fd = self.p.stdin.fileno()
	self.stdout_fd = self.p.stdout.fileno()
	self.stderr_fd = self.p.stderr.fileno()

Popen's stderr is then sent (here) to handle_remote_line(), which is doing quite sophisticated things to send this stuff to logging (and logging only, which is good). So, looking at this (and not understanding much more about Borg's code ❗), I presume that some older versions of borg serve did in fact use stderr, but no longer. If this is true, handle_remote_line() only exists for backwards compatibility reasons and can be removed in Borg 2.0, because Borg 2.0 isn't supporting old clients anyway. All server errors are sent via stdout and the only thing for which stderr is still needed, is to catch error output of ssh, e.g. reporting about failed auth.

If all of this is true, it's basically really just replacing the fds on both ends by the fds of a socket. The fds behave the same. Also see #6270 and my comment there.

The only caveat would be that the socket server would have a one client at a time limit for lock parity.

On a important side note: My intention with this suggestion is not making borg serve a "real" server accepting multiple clients simultaneously. For now it's totally fine to still require one borg serve process (and unix socket) per client, just as before, and therefore only accept a single client at a time. However, just to make this clear, this can be expanded to support multiple clients simultaneously at some later point. It naturally requires threading then, which AFAIK is a bigger topic for Borg, therefore it's intentionally out of the scope of this suggestion. But it's laying some ground work...

However, out of curiosity @RonnyPfannschmidt, why is socket.makefile() limiting it to a single client? As I said earlier, I never used socket.makefile(), but unless I understand the API docs wrong, we can (and actually must) call this per client and therefore get separate fds per client?

[RonnyPfannschmidt]

Makefile isn't, the need for locking limits based on usage patterns

[ThomasWaldmann]

@PhrozenByte btw, windows support is coming, @RayyanAnsari works on it (no remote repos yet though).