0.0.2 release

.gitignore

@@ -7,3 +7,4 @@ pip-wheel-metadata/
 *.ipynb
 notes.md
 *.yaml
+dist/

CHANGES.md

@@ -0,0 +1,13 @@
+# Changes
+
+## 0.0.2 (2020-07-14)
+
+- FEATURE: New, fully object oriented base library
+- FEATURE: Python 3.8 support added
+- FIX: `cleanup` does not delete snapshots on source if they are not present on target.
+- FIX: Wait for ZFS's garbage collection after `cleanup` for getting a meaningful value for freed space.
+- Dropped Python 3.5 support
+
+## 0.0.1 (2019-08-05)
+
+- Initial release.

README.md

@@ -2,48 +2,97 @@
 
 ## SYNOPSIS
 
-Simple ZFS sync tool. Shows local and remote ZFS dataset trees / zpools. Creates meaningful snapshots only if datasets have actually been changed. Compares a local dataset tree to a remote, backup dataset tree. Pushes backups to remote. Cleanes up older snapshot on local system. Runs form the command line and produces nice, user-friendly, readable, colorized output.
+`abgleich` is a simple ZFS sync tool. It displays source and target ZFS zpool, dataset and snapshot trees. It creates meaningful snapshots only if datasets have actually been changed. It compares a source zpool tree to a target, backup zpool tree. It pushes backups from a source to a target. It cleanes up older snapshots on the source side if they are present on the target side. It runs on a command line and produces nice, user-friendly, human-readable, colorized output.
+
+![demo](https://github.com/pleiszenburg/abgleich/blob/master/docs/demo.png?raw=true "demo")
 
 ## INSTALLATION
 
+```bash
+pip install -vU abgleich
+```
+
+or
+
 ```bash
 pip install -vU git+https://github.com/pleiszenburg/abgleich.git@master
 ```
 
-Requires (C)Python 3.5 or later. Tested with ZoL 0.7 and 0.8.
+Requires [CPython](https://en.wikipedia.org/wiki/CPython) 3.6 or later, a [Unix shell](https://en.wikipedia.org/wiki/Unix_shell) and [ssh](https://en.wikipedia.org/wiki/Secure_Shell). Tested with [OpenZFS](https://en.wikipedia.org/wiki/OpenZFS) 0.8.x on Linux.
+
+`abgleich`, CPython and the Unix shell must only be installed on one of the involved systems. Any remote system will be contacted via ssh and provided with direct ZFS commands.
+
+## INITIALIZATION
+
+All actions involving a remote host assume that `ssh` with public key authentication instead of passwords is correctly configured and working.
+
+Let's assume that everything in `source_tank/data` and below should be synced with `target_tank/some_backup/data`. `source_tank` and `target_tank` are zpools. `data` is the "prefix" for the source zpool, `some_backup/data` is the corresponding "prefix" for the target zpool. For `abgleich` to work, `source_tank/data` and `target_tank/some_backup` must exist. `target_tank/some_backup/data` must not exist. The latter will be created by `abgleich`. It is highly recommended to set the mountpoint of `target_tank/some_backup` to `none` before running `abgleich` for the first time.
+
+Rights to run the following commands are required:
+
+| command        | source | target |
+|----------------|:------:|:------:|
+| `zfs list`     |    x   |    x   |
+| `zfs get`      |    x   |    x   |
+| `zfs snapshot` |    x   |        |
+| `zfs send`     |    x   |        |
+| `zfs receive`  |        |    x   |
+| `zfs destroy`  |    x   |        |
+
+### `config.yaml`
+
+Complete example configuration file:
+
+```yaml
+source:
+    zpool: tank_ssd
+    prefix:
+    host: localhost
+    user:
+target:
+    zpool: tank_hdd
+    prefix: BACKUP_SOMEMACHINE
+    host: bigdata
+    user: zfsadmin
+keep_snapshots: 2
+suffix: _backup
+digits: 2
+ignore:
+    - home/user/CACHE
+    - home/user/CCACHE
+ssh:
+    compression: no
+    cipher: [email protected]
+```
+
+The prefix can be empty on either side. If a `host` is set to `localhost`, the `user` field can be left empty. Both source and target can be remote hosts or localhost at the same time. `keep_snapshots` is an integer and must be greater or equal to `1`. It specifies the number of snapshots that are kept per dataset on the source side when a cleanup operation is triggered. `suffix` contains the name suffix for new snapshots. `digits` specifies how many digits are used for a decimal number describing the n-th snapshot per dataset per day as part of the name of new snapshots. `ignore` lists stuff underneath the `prefix` which will be ignored by this tool, i.e. no snapshots, backups or cleanups. `ssh` allows to fine-tune the speed of backups. In fast local networks, it is best to set `compression` to `no` because the compression is usually slowing down the transfer. However, for low-bandwidth transmissions, it makes sense to set it to `yes`. For significantly better speed in fast local networks, make sure that both the source and the target system support a common cipher, which is accelerated by [AES-NI](https://en.wikipedia.org/wiki/AES_instruction_set) on both ends.
 
 ## USAGE
 
-### `abgleich tree [hostname]`
+All potentially changing or destructive actions are listed in detail before the user is asked to confirm them. None of the commands listed below create, change or destroy a zpool, dataset or snapshot on their own without the user's explicit consent.
 
-Show zfs tree with snapshots, disk space and compression ratio. Append `hostname` (optional) for remote tree. `ssh` without password (public key) required.
+### `abgleich tree config.yaml [source|target]`
+
+Show ZFS tree with snapshots, disk space and compression ratio. Append `source` or `target` (optional).
 
 ### `abgleich snap config.yaml`
 
-Determine which datasets have been changed since last snapshot. Generate snapshots where applicable. Superuser privileges required.
+Determine which datasets on the source side have been changed since last snapshot. Generate snapshots on the source side where applicable.
 
 ### `abgleich compare config.yaml`
 
-Compare local machine with remote host. See what is missing where. `ssh` without password (public key) required. Superuser privileges required.
+Compare source ZFS tree with target ZFS tree. See what is missing where.
 
 ### `abgleich backup config.yaml`
 
-Send (new) datasets and snapshots to remote host. `ssh` without password (public key) required. Superuser privileges required.
+Send (new) datasets and new snapshots from source to target.
 
 ### `abgleich cleanup config.yaml`
 
-Cleanup older local snapshots. Keep `keep_snapshots` number of snapshots. Superuser privileges required.
+Cleanup older local snapshots on source side if they are present on both sides. Of those snapshots present on both sides, keep at least `keep_snapshots` number of snapshots on source side.
 
-### `config.yaml`
+## SPEED
 
-Example configuration file:
+`abgleich` uses Python's [type hints](https://docs.python.org/3/library/typing.html) and enforces them with [typeguard](https://github.com/agronholm/typeguard) at runtime. It furthermore makes countless assertions.
 
-```yaml
-prefix_local: tank_ssd
-prefix_remote: tank_hdd/BACKUP_SOMEMACHINE
-host: bigdata
-keep_snapshots: 2
-ignore:
-    - /ernst/CACHE
-    - /ernst/CCACHE
-```
+The enforcement of types and assertions can be controlled through the `PYTHONOPTIMIZE` environment variable. If set to `0` (the implicit default value), all checks are activated. `abgleich` will run slow. For safety, this mode is highly recommended. For significantly higher speed, all type checks and most assertions can be deactivated by setting `PYTHONOPTIMIZE` to `1` or `2`, e.g. `PYTHONOPTIMIZE=1 abgleich tree config.yaml`. This is not recommended. You may want to check if another tool or configuration has altered this environment variable by running `echo $PYTHONOPTIMIZE`.

makefile

@@ -1,10 +1,10 @@
 
 clean:
 	-rm -r build/*
-	find src/ -name '*.pyc' -exec rm -f {} +
-	find src/ -name '*.pyo' -exec rm -f {} +
+	find src/ -name '*.pyc' -exec sudo rm -f {} +
+	find src/ -name '*.pyo' -exec sudo rm -f {} +
 	find src/ -name '*~' -exec rm -f {} +
-	find src/ -name '__pycache__' -exec rm -fr {} +
+	find src/ -name '__pycache__' -exec sudo rm -fr {} +
 	find src/ -name '*.htm' -exec rm -f {} +
 	find src/ -name '*.html' -exec rm -f {} +
 	find src/ -name '*.so' -exec rm -f {} +
@@ -20,4 +20,10 @@ release:
 	gpg --detach-sign -a dist/abgleich*.tar.gz
 
 install:
-	pip install -v -e .
+	pip install -vU pip setuptools
+	pip install -v -e .[dev]
+
+upload:
+	for filename in $$(ls dist/*.tar.gz dist/*.whl) ; do \
+		twine upload $$filename $$filename.asc ; \
+	done

setup.py

@@ -8,7 +8,7 @@
 
 	setup.py: Used for package distribution
 
-	Copyright (C) 2019 Sebastian M. Ernst <[email protected]>
+	Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
 
 <LICENSE_BLOCK>
 The contents of this file are subject to the GNU Lesser General Public License
@@ -30,95 +30,90 @@
 # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
 from setuptools import (
-	find_packages,
-	setup,
-	)
+    find_packages,
+    setup,
+)
 import os
 
 # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 # SETUP
 # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
 # Package version
-__version__ = '0.0.1'
+__version__ = "0.0.2"
 
 # List all versions of Python which are supported
+python_minor_min = 6
+python_minor_max = 8
 confirmed_python_versions = [
-	('Programming Language :: Python :: %s' % x)
-	for x in '3.5 3.6 3.7'.split(' ')
-	]
+    "Programming Language :: Python :: 3.{MINOR:d}".format(MINOR=minor)
+    for minor in range(python_minor_min, python_minor_max + 1)
+]
 
 # Fetch readme file
-with open(os.path.join(os.path.dirname(__file__), 'README.md')) as f:
-	long_description = f.read()
+with open(os.path.join(os.path.dirname(__file__), "README.md")) as f:
+    long_description = f.read()
 
 # Define source directory (path)
-SRC_DIR = 'src'
+SRC_DIR = "src"
 
 # Install package
 setup(
-	name = 'abgleich',
-	packages = find_packages(SRC_DIR),
-	package_dir = {'': SRC_DIR},
-	version = __version__,
-	description = 'zfs sync tool',
-	long_description = long_description,
-	long_description_content_type = 'text/markdown',
-	author = 'Sebastian M. Ernst',
-	author_email = '[email protected]',
-	url = 'https://github.com/pleiszenburg/abgleich',
-	download_url = 'https://github.com/pleiszenburg/abgleich/archive/v%s.tar.gz' % __version__,
-	license = 'LGPLv2',
-	keywords = [
-		'zfs',
-		'ssh',
-		],
-	scripts = [],
-	include_package_data = True,
-	setup_requires = [],
-	install_requires = [
-		'click',
-		'tabulate',
-		'pyyaml',
-		],
-	extras_require = {'dev': [
-		# 'pytest',
-		'python-language-server',
-		'setuptools',
-		# 'Sphinx',
-		# 'sphinx_rtd_theme',
-		'twine',
-		'wheel',
-		]},
-	zip_safe = False,
-	entry_points = {
-		'console_scripts': [
-			'abgleich = abgleich.cli:cli',
-			],
-		},
-	classifiers = [
-		'Development Status :: 5 - Production/Stable',
-		'Environment :: Console',
-		'Intended Audience :: Developers',
-		'Intended Audience :: Education',
-		'Intended Audience :: Information Technology',
-		'Intended Audience :: Science/Research',
-		'Intended Audience :: System Administrators',
-		'License :: OSI Approved :: GNU Lesser General Public License v2 (LGPLv2)',
-		'Operating System :: MacOS',
-		'Operating System :: POSIX :: BSD',
-		'Operating System :: POSIX :: Linux',
-		'Programming Language :: Python :: 3'
-		] + confirmed_python_versions + [
-		'Programming Language :: Python :: 3 :: Only',
-		'Programming Language :: Python :: Implementation :: CPython',
-		'Topic :: Scientific/Engineering',
-		'Topic :: System',
-		'Topic :: System :: Archiving',
-		'Topic :: System :: Archiving :: Backup',
-		'Topic :: System :: Archiving :: Mirroring',
-		'Topic :: System :: Filesystems',
-		'Topic :: System :: Systems Administration',
-		'Topic :: Utilities'
-		]
-	)
+    name="abgleich",
+    packages=find_packages(SRC_DIR),
+    package_dir={"": SRC_DIR},
+    version=__version__,
+    description="zfs sync tool",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    author="Sebastian M. Ernst",
+    author_email="[email protected]",
+    url="https://github.com/pleiszenburg/abgleich",
+    download_url="https://github.com/pleiszenburg/abgleich/archive/v%s.tar.gz"
+    % __version__,
+    license="LGPLv2",
+    keywords=["zfs", "ssh",],
+    scripts=[],
+    include_package_data=True,
+    python_requires=">=3.{MINOR:d}".format(MINOR=python_minor_min),
+    setup_requires=[],
+    install_requires=["click", "tabulate", "pyyaml", "typeguard",],
+    extras_require={
+        "dev": [
+            "black",
+            "python-language-server[all]",
+            "setuptools",
+            "twine",
+            "wheel",
+        ]
+    },
+    zip_safe=False,
+    entry_points={"console_scripts": ["abgleich = abgleich.cli:cli",],},
+    classifiers=[
+        "Development Status :: 5 - Production/Stable",
+        "Environment :: Console",
+        "Intended Audience :: Developers",
+        "Intended Audience :: Education",
+        "Intended Audience :: Information Technology",
+        "Intended Audience :: Science/Research",
+        "Intended Audience :: System Administrators",
+        "License :: OSI Approved :: GNU Lesser General Public License v2 (LGPLv2)",
+        "Operating System :: MacOS",
+        "Operating System :: POSIX :: BSD",
+        "Operating System :: POSIX :: Linux",
+        "Programming Language :: Python :: 3",
+    ]
+    + confirmed_python_versions
+    + [
+        "Programming Language :: Python :: 3 :: Only",
+        "Programming Language :: Python :: Implementation :: CPython",
+        "Topic :: Scientific/Engineering",
+        "Topic :: System",
+        "Topic :: System :: Archiving",
+        "Topic :: System :: Archiving :: Backup",
+        "Topic :: System :: Archiving :: Mirroring",
+        "Topic :: System :: Filesystems",
+        "Topic :: System :: Systems Administration",
+        "Topic :: Utilities",
+    ],
+)

src/abgleich/__init__.py

@@ -8,7 +8,7 @@
 
 	src/abgleich/__init__.py: Package root
 
-	Copyright (C) 2019 Sebastian M. Ernst <[email protected]>
+	Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
 
 <LICENSE_BLOCK>
 The contents of this file are subject to the GNU Lesser General Public License

src/abgleich/cli/__init__.py

@@ -8,7 +8,7 @@
 
 	src/abgleich/cli/__init__.py: CLI package root
 
-	Copyright (C) 2019 Sebastian M. Ernst <[email protected]>
+	Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
 
 <LICENSE_BLOCK>
 The contents of this file are subject to the GNU Lesser General Public License

src/abgleich/cli/_main_.py

@@ -8,7 +8,7 @@
 
 	src/abgleich/cli/_main_.py: CLI auto-detection
 
-	Copyright (C) 2019 Sebastian M. Ernst <[email protected]>
+	Copyright (C) 2019-2020 Sebastian M. Ernst <[email protected]>
 
 <LICENSE_BLOCK>
 The contents of this file are subject to the GNU Lesser General Public License
@@ -38,19 +38,20 @@
 # ROUTINES
 # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
+
 def _add_commands(ctx):
-	"""auto-detects sub-commands"""
-	for cmd in (
-		item[:-3] if item.lower().endswith('.py') else item[:]
-		for item in os.listdir(os.path.dirname(__file__))
-		if not item.startswith('_')
-		):
-		ctx.add_command(getattr(importlib.import_module(
-			'abgleich.cli.%s' % cmd
-			), cmd))
+    """auto-detects sub-commands"""
+    for cmd in (
+        item[:-3] if item.lower().endswith(".py") else item[:]
+        for item in os.listdir(os.path.dirname(__file__))
+        if not item.startswith("_")
+    ):
+        ctx.add_command(getattr(importlib.import_module("abgleich.cli.%s" % cmd), cmd))
+
 
 @click.group()
 def cli():
-	"""abgleich, zfs sync tool"""
+    """abgleich, zfs sync tool"""
+
 
 _add_commands(cli)