Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Backblaze for Backup #1812

Merged
merged 15 commits into from
Nov 26, 2020
+71 −11
Merged

Implement Backblaze for Backup #1812

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
c7ae4cc
Installing b2sdk for b2 support
hija Aug 26, 2020
7b4d621
Added Duplicity PPA so the most recent version is used
hija Aug 26, 2020
9199294
Implemented list_target_files for b2
hija Aug 26, 2020
1081be8
Implemented b2 in frontend
hija Aug 26, 2020
df6fcf3
Added b2sdk to virtualenv so backup.py can use it
hija Aug 26, 2020
8085246
Changed indentation to match the old one
hija Aug 26, 2020
aa5ba99
removed unnecessary, readonly field in b2 config
hija Aug 30, 2020
ace2bbf
Removed placeholder text for b2 bucket
hija Aug 30, 2020
6cf202b
Merge pull request #1 from hija/backblaze-trimmed
hija Aug 30, 2020
b62dcc5
Assigning targetPath to a new var
hija Sep 4, 2020
eecf328
Added link to backblaze
hija Sep 4, 2020
14bbaaf
Fixed indentation
hija Sep 4, 2020
8185b76
removed python2 boto package
hija Sep 4, 2020
35de916
Merge pull request #3 from hija/backblaze-adjustments
hija Sep 4, 2020
eb41ca4
Merge branch 'master' into backblaze
hija Nov 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions management/backup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -456,6 +456,23 @@ def list_target_files(config):
raise ValueError(e.reason)

return [(key.name[len(path):], key.size) for key in bucket.list(prefix=path)]
elif target.scheme == 'b2':
from b2sdk.v1 import InMemoryAccountInfo, B2Api
from b2sdk.v1.exception import NonExistentBucket
info = InMemoryAccountInfo()
b2_api = B2Api(info)

# Extract information from target
b2_application_keyid = target.netloc[:target.netloc.index(':')]
b2_application_key = target.netloc[target.netloc.index(':')+1:target.netloc.index('@')]
b2_bucket = target.netloc[target.netloc.index('@')+1:]

try:
b2_api.authorize_account("production", b2_application_keyid, b2_application_key)
bucket = b2_api.get_bucket_by_name(b2_bucket)
except NonExistentBucket as e:
raise ValueError("B2 Bucket does not exist. Please double check your information!")
return [(key.file_name, key.size) for key, _ in bucket.ls()]

else:
raise ValueError(config["target"])
Expand Down
46 changes: 43 additions & 3 deletions management/templates/system-backup.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ <h3>Configuration</h3>
<option value="local">{{hostname}}</option>
<option value="rsync">rsync</option>
<option value="s3">Amazon S3</option>
<option value="b2">Backblaze B2</option>
</select>
</div>
</div>
Expand Down Expand Up @@ -111,6 +112,31 @@ <h3>Configuration</h3>
<input type="text" class="form-control" rows="1" id="backup-target-pass">
</div>
</div>
<!-- Backblaze -->
<div class="form-group backup-target-b2">
<div class="col-sm-10 col-sm-offset-2">
<p>Backups are stored in a <a href="https://www.backblaze.com/" target="_blank" rel="noreferrer">Backblaze</a> B2 bucket. You must have a Backblaze account already.</p>
<p>You MUST manually copy the encryption password from <tt class="backup-encpassword-file"></tt> to a safe and secure location. You will need this file to decrypt backup files. It is NOT stored in your Backblaze B2 bucket.</p>
</div>
</div>
<div class="form-group backup-target-b2">
<label for="backup-target-b2-user" class="col-sm-2 control-label">B2 Application KeyID</label>
<div class="col-sm-8">
<input type="text" class="form-control" rows="1" id="backup-target-b2-user">
</div>
</div>
<div class="form-group backup-target-b2">
<label for="backup-target-b2-pass" class="col-sm-2 control-label">B2 Application Key</label>
<div class="col-sm-8">
<input type="text" class="form-control" rows="1" id="backup-target-b2-pass">
</div>
</div>
<div class="form-group backup-target-b2">
<label for="backup-target-b2-bucket" class="col-sm-2 control-label">B2 Bucket</label>
<div class="col-sm-8">
<input type="text" class="form-control" rows="1" id="backup-target-b2-bucket">
</div>
</div>
<!-- Common -->
<div class="form-group backup-target-local backup-target-rsync backup-target-s3">
<label for="min-age" class="col-sm-2 control-label">Retention Days:</label>
Expand Down Expand Up @@ -144,7 +170,7 @@ <h3>Available backups</h3>

function toggle_form() {
var target_type = $("#backup-target-type").val();
$(".backup-target-local, .backup-target-rsync, .backup-target-s3").hide();
$(".backup-target-local, .backup-target-rsync, .backup-target-s3, .backup-target-b2").hide();
$(".backup-target-" + target_type).show();

init_inputs(target_type);
Expand Down Expand Up @@ -215,7 +241,7 @@ <h3>Available backups</h3>
}

function show_custom_backup() {
$(".backup-target-local, .backup-target-rsync, .backup-target-s3").hide();
$(".backup-target-local, .backup-target-rsync, .backup-target-s3, .backup-target-b2").hide();
api(
"/system/backup/config",
"GET",
Expand Down Expand Up @@ -245,6 +271,15 @@ <h3>Available backups</h3>
var host = hostpath.shift();
$("#backup-target-s3-host").val(host);
$("#backup-target-s3-path").val(hostpath.join('/'));
} else if (r.target.substring(0, 5) == "b2://") {
$("#backup-target-type").val("b2");
var targetPath = r.target.substring(5);
var b2_application_keyid = targetPath.split(':')[0];
var b2_applicationkey = targetPath.split(':')[1].split('@')[0];
var b2_bucket = targetPath.split('@')[1];
$("#backup-target-b2-user").val(b2_application_keyid);
$("#backup-target-b2-pass").val(b2_applicationkey);
$("#backup-target-b2-bucket").val(b2_bucket);
}
toggle_form()
})
Expand All @@ -264,6 +299,11 @@ <h3>Available backups</h3>
target = "rsync://" + $("#backup-target-rsync-user").val() + "@" + $("#backup-target-rsync-host").val()
+ "/" + $("#backup-target-rsync-path").val();
target_user = '';
} else if (target_type == "b2") {
target = 'b2://' + $('#backup-target-b2-user').val() + ':' + $('#backup-target-b2-pass').val()
+ '@' + $('#backup-target-b2-bucket').val()
target_user = '';
target_pass = '';
}


Expand Down Expand Up @@ -303,4 +343,4 @@ <h3>Available backups</h3>
set_host($('#backup-target-s3-host-select').val());
}
}
</script>
</script>
16 changes: 8 additions & 8 deletions setup/management.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,19 +18,19 @@ while [ -d /usr/local/lib/python3.4/dist-packages/acme ]; do
pip3 uninstall -y acme;
done

# duplicity is used to make backups of user data. It uses boto
# (via Python 2) to do backups to AWS S3. boto from the Ubuntu
# package manager is too out-of-date -- it doesn't support the newer
# S3 api used in some regions, which breaks backups to those regions.
# See #627, #653.
# duplicity is used to make backups of user data.
#
# virtualenv is used to isolate the Python 3 packages we
# install via pip from the system-installed packages.
#
# certbot installs EFF's certbot which we use to
# provision free TLS certificates.
apt_install duplicity python-pip virtualenv certbot
hide_output pip2 install --upgrade boto

# b2sdk is used for backblaze backups.
# boto is used for amazon aws backups.
# Both are installed outside the pipenv, so they can be used by duplicity
hide_output pip3 install --upgrade b2sdk boto

# Create a virtualenv for the installation of Python 3 packages
# used by the management daemon.
Expand All @@ -50,8 +50,8 @@ hide_output $venv/bin/pip install --upgrade pip
hide_output $venv/bin/pip install --upgrade \
rtyaml "email_validator>=1.0.0" "exclusiveprocess" \
flask dnspython python-dateutil \
qrcode[pil] pyotp \
"idna>=2.0.0" "cryptography==2.2.2" boto psutil postfix-mta-sts-resolver
qrcode[pil] pyotp \
"idna>=2.0.0" "cryptography==2.2.2" boto psutil postfix-mta-sts-resolver b2sdk

# CONFIGURATION

Expand Down
3 changes: 3 additions & 0 deletions setup/system.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,9 @@ hide_output add-apt-repository -y universe
# Install the certbot PPA.
hide_output add-apt-repository -y ppa:certbot/certbot

# Install the duplicity PPA.
hide_output add-apt-repository -y ppa:duplicity-team/duplicity-release-git

# ### Update Packages

# Update system packages to make sure we have the latest upstream versions
Expand Down