Implement Backblaze for Backup

[hija]

Hello :)

I started implementing Backblaze as backup option for miab.

There is one major change I would like to point out: I added the duplicity PPA, so the lastest version of duplicity gets installed. I struggled to get b2 working with the version of duplicity shipped in the default repository and it worked without any problems with the duplicity PPA.

How I tested:

• Created new VM and installed MIAB and configured it to use B2 for backup
• Ran sudo ./management/backup.py && sudo ./management/backup.py --verify
• Result: Verify complete: 84 files compared, 0 differences found.
• Looked at Web UI and saw that the backup is stored and the size of backup is determined correctly

Where I need help:

• Verify that it works correctly on other machines as well
• Make sure that the duplicity PPA does not break the already working mechanisms, e.g. Amazon s3. However, I do not have an Amazon S3 account and would appreciate help with this.

I would love to get feedback on this PR.

[hija]

I tested the change on a development server (since four days) and it is running since two days on my production server. 👍

[fspoettel]

Nice work! 🙂 Love seeing B2 support for backups. I ran into some problems with S3 backups after the ppa change (duplicity couldn't find boto) and added some minor suggestions

[hija]

Hey @fspoettel,

I think I integrated your changes 👍 Thanks for your review and your suggestions. I really appreciate them!

[nomandera]

Devils advocate. Would it be worth considering deploying this as two pulls:

1. Update duplicity
2. Add Backblaze custom config options

These would be the smallest possible iterations and has the advantage of allowing quicker feedback if the duplicity upgrade breaks any existing edge case backups.

Very keen to see this deployed as Backblaze is legitimately 3-4 times cheaper than almost everyone else whilst being a open and professional company.

[hija]

Thanks for your idea, @anoma :) I did not think about splitting this up before, but it might be worth it 👍

@JoshData Do you think this pull request is interesting for the project? If so: Should I split it up or are you ok with one "big" pull request?

[fspoettel]

PR relates to #717

[fspoettel]

recent discourse thread where b2 came up again. it seems like this is a feature that some users really want and have gone the length of implementing homebrew solutions.

[JoshData]

Ok. I appreciate the work here. I am merging it. If it breaks anything, you all are on the hook for fixing it quickly. :)

[barrybingo]

So my B2 Application Key had a slash in it and this causes a "substring not found" error.
Quick fix was to generate a new API key.
Here is the error POC given app key = 'firstpartofappkey/secondpartofappkey'

>>> import urllib.parse
>>> target = urllib.parse.urlparse('b2://00KeyID00000:firstpartofappkey/secondpartofappkey@somebucket')
>>> b2_application_keyid = target.netloc[:target.netloc.index(':')]
>>> b2_application_key = target.netloc[target.netloc.index(':')+1:target.netloc.index('@')]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: substring not found
>>> 
>>> 
>>> print(target)
ParseResult(scheme='b2', netloc='00KeyID00000:firstpartofappkey', path='/secondpartofappkey@somebucket', params='', query='', fragment='')

[hija]

Hey @barrybingo,

unfortunately I don't see how to fix this issue on our side since duplicity cannot parse the URL if the API key contains a slash (so even if we would fix the code you pointed out above the backup would still not work):

• https://bugs.launchpad.net/duplicity/+bug/1819390

Maybe we should disallow entering application keys and application ids if they contain a slash and ask to regenerate the API if it contains a slash?

[barrybingo]

Yeah I vote for disallow and give instructions.
The problem at the moment is the error "substring not found" has zero meaning.

[barrybingo]

Also if you're going to do some changes then you could add backup-target-b2 to this line so retention days is available for edit

mailinabox/management/templates/system-backup.html

Line 141 in 8664afa

<div class="form-group backup-target-local backup-target-rsync backup-target-s3">

So far first backup is working fine so thumbs up

[hija]

Also if you're going to do some changes then you could add backup-target-b2 to this line so retention days is available for edit

mailinabox/management/templates/system-backup.html

Line 141 in 8664afa

<div class="form-group backup-target-local backup-target-rsync backup-target-s3">

So far first backup is working fine so thumbs up

Could you clarify what you mean with that?

[barrybingo]

That div is hidden for b2 as the div class does not contain backup-target-b2

[hija]

Ahhh okay. So your point is that we should be able to set a custom retention time because there is no reason to not do so, right?

[barrybingo]

Yup. That setting's value is passed to duplicity so should be configurable.

[hija]

Got ya! Good catch! Do you want to do a PR or should I do one?

[barrybingo]

I don't want to step on your toes and it is Saturday night here 🍻
If you haven't submitted one by tomorrow I'll do it.

But can't you just add the commits onto your branch then update this PR for Josh to review and re-merge?
Keep everything in one place.

[fspoettel]

But can't you just add the commits onto your branch then update this PR for Josh to review and re-merge?
Keep everything in one place.

no, once a PR is merged, it cannot be re-opened. Depending on the repository's merge strategy and activity, it's also not a good idea to re-use stale branches.

[hija]

Deleted the fork anyways (don't remember why exactly). I will see if I can do a new PR today / tonight, otherwise barrybingo can do one :)

[hija]

I don't want to step on your toes and it is Saturday night here 🍻
If you haven't submitted one by tomorrow I'll do it.

But can't you just add the commits onto your branch then update this PR for Josh to review and re-merge?
Keep everything in one place.

I won't make it today. Go ahead if you want to :) @barrybingo

[barrybingo]

First draft
#1882