ETCD-610: automated backups no config

[Elbehery]

This PR add enhancement proposal for Etcd Automated Backups No Config.

Resolves https://issues.redhat.com/browse/ETCD-610

cc @openshift/openshift-team-etcd

[openshift-ci-robot]

@Elbehery: This pull request references ETCD-610 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.17.0" version, but no target version was set.

In response to this:

This PR add enhancement proposal for Etcd Automated Backups No Config.

Resolves https://issues.redhat.com/browse/ETCD-610

cc @openshift/openshift-team-etcd

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[Elbehery]

/assign @hasbro17
/assign @tjungblu
/assign @dusk125
/assign @soltysh

[tjungblu]

how would we restore from a localVolume? Is the data stored somewhere in /var/?

[Elbehery]

so the localVolume is being mounted anywhere as specified in the Pod. In fact, you can mount a whole external drive.

iiuc, hostPath supports only mount points from the node's FS

[tjungblu]

imagine we have to run the restore script, how would we get the local volume mounted into a recovery machine? would we create another static pod that would mount the localVolume?

I imagine the local storage plugin must be running along with the kubelet for that to work? @jsafrane / @gnufied maybe you guys can elaborate a bit how that works, I don't have much time this week to read through the CSI implementation 🐎

[gnufied]

pardon my ignorance for not being familiar with etcd deployment, but even if there is a default SC available, why are we backing up just one replica of etcd? I thought, we would want to backup all replicas - no? Such as - what if in case of network partition or something, the replica we are backing up is behind?

So IMO it sounds whether we are using LocalVolume or something else, we should always be creating backups across master nodes? Is that accurate?

[gnufied]

Also to answer @tjungblu question. Local Volumes are a inbuilt feature of k8s and hence no additional plugin is necessary. Heck, a local PV can be provisioned statically and local-storage-operator is not necessary either, https://docs.openshift.com/container-platform/4.15/storage/persistent_storage/persistent_storage_local/persistent-storage-local.html#local-create-cr-manual_persistent-storage-local

[tjungblu]

... and can be mounted in the new etcd pod and the backup will exist.

check the restore script, you will need to unpack the snapshot before you can run any pod

[Elbehery]

@gnufied in order to make sure the backups are accessible, and that we can round-robin the backups across all master nodes, which are up-to-date with the etcd-cluster leader

Is it possible to

• Create StatefulSet across the master nodes, where the PVC template uses the localVolume to provision PV.
• Since we need to do the backup according to a schedule as per EtcdBackupSpec, the issue to manage the backup pods within the STS using the CronJob
• I am not aware if this is possible, but at least, we can generate Event from the CronJob and the STS could react to these event by taking a backup

[Elbehery]

check the restore script, you will need to unpack the snapshot before you can run any pod

We can use init-container to unpack the backup, and then the etcd-pod can start ?

Otherwise, we can start a fresh etcd-pod and then run etcdctl restore ?

Please correct me if I am wrong

[soltysh]

• Create StatefulSet across the master nodes, where the PVC template uses the localVolume to provision PV.

I believe you mean DaemonSet across the master nodes 😉

[Elbehery]

I actually thought about this as well, but there is no PersistentVolumeClaim template on DaemonSet.

Having a separate PVC & PV for each master node would be ideal for spreading the backups across the master nodes.

That being said, according to the latest update, we are going to use SideCar container && hostPath volume

[tjungblu]

would/could a DaemonSet do that for us?

[Elbehery]

Good point.

• We can use DS and use node affinity to run the DS pods only on master nodes. However, I am not aware of how the storage will be handled in this case.

• Is there is a way to keep the backups in round-robin fashion among all master nodes ?

• Also what about STS ? wdyt ?

[tjungblu]

yep, STS could also work, especially with the storage. Would also be useful to have this compared to the static pod/hostpath approach

[Elbehery]

Alright, I think if we have STS, then the Pods, PVC, and PVs are being managed together

I think we can in this case create backups on RoundRobin manner across the master nodes.

For BM, I think we can still use local volume as storage option and being managed by STS

[gnufied]

How does this round robin backup work?

[gnufied]

I read the original enhancement - https://github.com/openshift/enhancements/blob/master/enhancements/etcd/automated-backups.md and I understand this now better. But still this begs the question - how do you know which snapshot to restore from? What if you take snapshot of a etcd replica which was behind and is just catching up with other master nodes? Is that possible?

[Elbehery]

So we will make sure that the backup is taken from the a member whose log is identical to the leader. This way we make sure that we are not lagging behind.

But, could this approach work ?

[soltysh]

Why do you want to handle the balancing? That's not what you should care for at all.

[Elbehery]

Yes I agree, therefore the SideCar approach is the most appropriate

[tjungblu]

how is the customer able to access a dynamically created PV to get their snapshots for restoration?

[Elbehery]

Gr8 question, so I have defaulted the RetentionPolicy in my sample configs to Retain.

This way the PV contents will never be erased automatically.

Now to answer the restoration

• If the cluster still running, the PV can be attached to any node, as long as it is in the same availability zone.
• If the cluster is completely down, the PV can be accessed by the CU, it will never be delete unless manually.

[tjungblu]

it's not about the retention, it's about the access and mounting possibilities

If the cluster still running, the PV can be attached to any node, as long as it is in the same availability zone.

that's quite a bad constraint for a disaster recovery procedure, isn't it? :)

[Elbehery]

:D :D .. I know, well the best option imho is to push backups to Remote storage option, then even in case of the whole cluster is down, a new installation can be restored using the remote backup.

However, in my discussion yesterday, the remote storage was not an option for BM. Also they recommended to keep it simple not too complicated

If it were to me, I would create a side car which pushes the backups to remote storage, I think this option would work for any OCP variant. The master nodes dont have to be in the cloud, the side car could authenticate and push backups regardless of OCP underlying infrastructure, wdyt

[gnufied]

If dyanmically provisioned volumes are available in the cluster, it is 100% likely that, cluster has Remote Storage and hence backups are available even if cluster is down.

As for availability zones concerns, yeah that is why, backing up into a PV is not enough. We should consider using CSI snapshots of those PVCs, so as snapshot of backups can be available across availability zones and in case file system on PV gets corrupt or something.

[Elbehery]

I actually want to fall back to the localVolume approach, since having two separate approaches is harder to maintain.

I believe the localVolume should be sufficient. However, if we could utilize STS across all the master nodes would be ideal in situation where we lose one or more master nodes.

[gnufied]

If I were to design this, I would definitely use two different backup strategies. It is far more reliable to use dynamically provisioned PVCs for backups. They can be snapshotted and can be accessible if node goes down.

It seems to me that - any hostpath/localvolume based approach is basically inferior. If last backup was taken on leader and then leader node goes down, then so does our backup(other nodes may have slightly behind backups). So hostpath/localvolume requires fundamentally different recovery mechanism.

IMO - if we take other components that use persistent storage in Openshift, if customers don't provide Storage then no persistent configuration is created. Prometheus or image-registry, they all require storage. And they don't do anything automatically.

So, what I am saying is - we should probably limit scope of this KEP only to environments where an StorageClass is available. If no StorageClass is configured then no automatic backups are taken.

We should not take upon ourselves to decide a storage strategy for customer. cc @tjungblu

[gnufied]

So to summarize - I would simply not bother to configure backups via localvolume/hostpath in environments where no storage is available and solve that problem via documentation or ask customer to configure storage. I do not think, we should automatically configuring local-storage for these clusters.

[Elbehery]

I actually agree that using dynamic provisioning and CSI snapshot is vital for the automated backup.

I also agree to limit the automated backup for clusters with dynamic provisioning only.

wdyt @tjungblu @hasbro17 @dusk125 @soltysh

[Elbehery]

Also see this default I used to test the approach

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: etcd-backup-local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: Immediate

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: etcd-backup-pvc
  namespace: openshift-etcd
spec:
  accessModes:
  - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 10Gi 
  storageClassName: etcd-backup-local-storage

---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: etcd-backup-pv-fs
spec:
  capacity:
    storage: 100Gi 
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: etcd-backup-local-storage
  local:
    path: /mnt
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:

---

apiVersion: operator.openshift.io/v1alpha1
kind: EtcdBackup
metadata:
  name: etcd-single-backup
  namespace: openshift-etcd
spec:
  pvcName: etcd-backup-pvc

[gnufied]

You don't have to necessarily install local-storage-operator to use/consume local-storage. For simple one-off use cases like this, it may be possible that etcd-backup operator or something creates static-pv as documented - https://docs.openshift.com/container-platform/4.15/storage/persistent_storage/persistent_storage_local/persistent-storage-local.html#local-create-cr-manual_persistent-storage-local and uses it to perform etcd backups.

[soltysh]

Given that AutomatedEtcdBackup is still a tech preview feature, I'd consider expanding that functionality with the default configuration, rather than adding another one. This will save a lot of problems for some of the fields in the API you're planning to provide default values for.

[Elbehery]

+1, I will fix this 👍🏽

[soltysh]

• Create StatefulSet across the master nodes, where the PVC template uses the localVolume to provision PV.

I believe you mean DaemonSet across the master nodes 😉

[soltysh]

Why do you want to handle the balancing? That's not what you should care for at all.

[soltysh]

Suggested change

	However, I am strongly against using it for the following reasons
	However, the following reasons suggest against that solution:

[Elbehery]

this whole has been removed, radical changes :D :D

[soltysh]

You forgot to add the locality of the backup. Iow. if it happens that backup is kept in the current leader, and that dies, we're loosing that data with it.

[Elbehery]

So the current approach will take backups among all master nodes. Actually we have another issue now, how to distinguish which backup is most up-to-date since we have backups from all master nodes :)

[Elbehery]

/label tide/merge-method-squash

[Elbehery]

@soltysh Would you kindly help me answering these questions ?

[tjungblu]

we will call it name="default" and in the controller we'll skip the reconciliation of that CRD

[Elbehery]

so if the sidecar is enabled we skip the backup using the controller, or we skip the default backups if the controllers is enabled ?

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from hasbro17. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Elbehery]

Overrunning disk space on the control-plane nodes or temporarily I/O saturating them resulting in etcd latency spikes are my primary concerns. Maybe we'd want to stagger the schedule among etcd members. Though I'm not sure if that's something a manually speficied Backup CR allows.

@sjenning I also vote for the DaemonSet approach

you can find here a test of the CronBased approach #1646 (comment) .. IMHO the DS approach is more flexible, and easier to modify and adapt.

For instance, I could just add a container to the Pod template, in order to push the backup to external target (e.g. S3 bucket)

[Elbehery]

I have created this poll for easier voting, please click on the image and then vote accordingly

cc @JoelSpeed
@tjungblu
@vrutkovs
@jmhbnz
@deads2k
@csrwng
@sjenning
@wking
@rhuss
@cuppett

[openshift-ci]

@Elbehery: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[JoelSpeed]

Also posted on the implementation PR, but perhaps we should continue the discussion here

The limitation is that, the pruning runs on only one master node at a time.

I guess I didn't fully understand the flow here, did not expect to see a cronjob to create EtcdBackup which then triggers the actual backups, I thought that would be a controller of itself and that we only had one Job involved here.

Anyway, is there any particular reason why the pruning has to be done prior to the EtcdBackup being created? Why does the pruning not happen as part of the second Job, the one triggered by the EtcdBackup being created? (Do you have well known names for these two jobs?)

To summarize, this approach works but as you can see, the DaemonSet approach is much more flexible

How would you make the argument that it is more flexible? Which parts are more flexible? Is this flexibility required, and, is it worth having to implement and maintain a completely new backup system, vs just re-using/extending the existing system and having 1 path?

spec:
  pvcName: no-config

The intention in the future would be to extend this API and create a specific area to specify what kind of storage is being used, and assume not to just assume based on a magic keyname.

I believe we discussed making this something like

spec:
  backupStorage:
    type: HostPath | PVC
    pvc: # Only valid when type is PVC
      name: ...

[Elbehery]

@JoelSpeed yes, u r right, lets move here

Citing PoC implemented in openshift/cluster-etcd-operator#1381

I guess I didn't fully understand the flow here, did not expect to see a cronjob to create EtcdBackup which then triggers the actual backups, I thought that would be a controller of itself and that we only had one Job involved here.

So this is the current implementation, as I explained in openshift/cluster-etcd-operator#1381 (comment)

Anyway, is there any particular reason why the pruning has to be done prior to the EtcdBackup being created? Why does the pruning not happen as part of the second Job, the one triggered by the EtcdBackup being created? (Do you have well known names for these two jobs?)

I am not the one who can answer this, I believe @tjungblu and @hasbro17 are the best to answer

However, iirc, the prune container had to run as init-container, since it should run in serial order with the cluster-backup container.

I can also assume that since the we have a one-time cluster backup, which is supported by the request-backup container and being handled by backup controller, and a recurring backup being handled by periodicbackupcontroller, maybe this is the reason we have two separate resources.

The templates used for creating the CronJob and the Job resources are below

• CronJob
• Job

How would you make the argument that it is more flexible? Which parts are more flexible? Is this flexibility required, and, is it worth having to implement and maintain a completely new backup system, vs just re-using/extending the existing system and having 1 path?

As the whole code of the etcd-backup-server runs as a single entity (i.e. Pod), it is easier to adapt, and edit.

For instance, if we were now moving the prune container from the CronJob level to the BackupJob level, this could be a big change. On the other hand, doing the same changes on the etcd-backup-server Pod, is bound to the same entity, and would not have a big impact on CU experience with the product.

The intention in the future would be to extend this API and create a specific area to specify what kind of storage is being used, and assume not to just assume based on a magic keyname.

Indeed, if we were to move forward with this approach, then we would make the API changes 👍🏽

[tjungblu]

Anyway, is there any particular reason why the pruning has to be done prior to the EtcdBackup being created? Why does the pruning not happen as part of the second Job, the one triggered by the EtcdBackup being created? (Do you have well known names for these two jobs?)

I am not the one who can answer this, I believe @tjungblu and @hasbro17 are the best to answer

Very simply because an EtcdBackup is a one-off backup. There is nothing to prune in such a scenario, in the recurring one we need to ensure that we do not overflow the disk. That's also the reason why it's done prior to creating the backup, to ensure there is enough space left for another backup.

[Elbehery]

Anyway, is there any particular reason why the pruning has to be done prior to the EtcdBackup being created? Why does the pruning not happen as part of the second Job, the one triggered by the EtcdBackup being created? (Do you have well known names for these two jobs?)

I am not the one who can answer this, I believe @tjungblu and @hasbro17 are the best to answer

Very simply because an EtcdBackup is a one-off backup. There is nothing to prune in such a scenario, in the recurring one we need to ensure that we do not overflow the disk. That's also the reason why it's done prior to creating the backup, to ensure there is enough space left for another backup.

so moving the prune container to the cluster-backup-job is not an option ?

because this is the only way to have the pruning functionality while using the parallel-job approach 🤔

[tjungblu]

what's the point of that? I want to create a one-off-backup because I'm making a risky change, I don't want the job to delete all prior backups.

because this is the only way to have the pruning functionality while using the parallel-job approach 🤔

I thought we have settled on DS and a separate hostPath api change now, why are we debating this again?

[Elbehery]

what's the point of that? I want to create a one-off-backup because I'm making a risky change, I don't want the job to delete all prior backups.

because this is the only way to have the pruning functionality while using the parallel-job approach 🤔

I thought we have settled on DS and a separate hostPath api change now, why are we debating this again?

I am replying to the reviews requests, if we are settled then I am happy 🥳

What API change I need to create ?

[JoelSpeed]

I can also assume that since the we have a one-time cluster backup, which is supported by the request-backup container and being handled by backup controller, and a recurring backup being handled by periodicbackupcontroller, maybe this is the reason we have two separate resources.

Very simply because an EtcdBackup is a one-off backup. There is nothing to prune in such a scenario, in the recurring one we need to ensure that we do not overflow the disk. That's also the reason why it's done prior to creating the backup, to ensure there is enough space left for another backup.

With a one-off backup, you have no pre-flight check to make sure there is enough space? Would it not be helpful, in the case where a user requests a one-off backup to run this check? For example, we are talking about enabling the use of EtcdBackup to trigger hostpath based backups (and IIRC that was also in scope beyond this EP?), why would you not want pruning when an admin triggers a one off hostpath back?

I want to create a one-off-backup because I'm making a risky change

A user would do this prior to the risky change, so pruning before this backup isn't actually harmful as the cluster should still be in a good state

I don't want the job to delete all prior backups.

Pruning deletes all previous backups? I would have assumed it prunes some amount to make sure there is space, but leaving some sort of history?

I thought we have settled on DS and a separate hostPath api change now, why are we debating this again?

IIUC, the last discussion assumed that the idea of just re-using all of the existing mechanisms and setting up a Job with parrallelism didn't work. But Mustafa hadn't actually tested the key part of that prior to ending that experiment. I prompted him about this and he has now completed the experiment, and has shown it works.

The previous discussion assumed only the DS worked. So now the decision should be re-discussed. Do we want a completely new backup system, or do we want to just extend the current one? Do we want 2 different implementations reacting to one CRD? Or just one? Both solutions work, both have pros, both have cons.

[tjungblu]

With a one-off backup, you have no pre-flight check to make sure there is enough space? Would it not be helpful, in the case where a user requests a one-off backup to run this check?

Now you're conflating two different things, pre-flight space checks and retention. What should be retained on a hostPath that you just want to have for a one-time backup?

A user would do this prior to the risky change, so pruning before this backup isn't actually harmful as the cluster should still be in a good state

So I shall randomly delete things under a hostPath that we don't know what's in there, before doing a risky change? This is absurd, Joel.

[JoelSpeed]

Now you're conflating two different things, pre-flight space checks and retention. What should be retained on a hostPath that you just want to have for a one-time backup?

So I shall randomly delete things under a hostPath that we don't know what's in there, before doing a risky change? This is absurd, Joel.

How does this work today? I am not an expert on the backup mechanics, but I would assume at least that when you create a backup, there is a file (or folder?) with a well known name? You would delete only the things that look like they were created by a backup, based on the naming

I'm thinking here about how traditional VM backup software works. In at least 3 backup softwares I know of, I can configure a job, as a manual run. But then I can trigger that multiple times, and the retention settings are still shared across those runs.

In a parallel here, if I were an admin, and I didn't want to schedule my backups, but I re-used the folder over and over, would I expect over time to cap the size of that backup? Would I expect it to eventually start pruning the oldest of my manual backups? Perhaps this isn't supported today since there isn't a pruning/retention configuration in the API?

Are we going off on a tangent here? Perhaps we should refocus the conversation on the question "Would it be preferable long term to base all types of backup on a single implementation (the Job based one) assuming no major hurdles?" and if the answer to that is Yes, "What of the existing open questions/discussions around the Job based approach would be considered to big to overcome?".

[hexfusion]

nit: etcd is always lowercase :)

[hexfusion]

question: from a support perspective does this mean it is a bug if the backup is invalid or corrupt? are there any legal concerns with Red Hat "owning" backups?

[hexfusion]

Would it also be a goal to ensure that the backup is valid? Is there a periodic job that ensures this or is the cluster/customer trusting it will work?

[hexfusion]

I believe the size of the etcd db is no longer capped. Is this a concern for taking snapshots?

[hexfusion]

question: would a failure to create a backup block upgrades?

[hexfusion]

question: it has been a while but doesn't a snapshot request always go to the leader?

[JoelSpeed]

So how does retention actually work for the hostpath based backup in the DS approach?

[JoelSpeed]

I don't think this is accurate based on the latest testing and discussion is it?

[JoelSpeed]

Reviewing now, and discussing sync with @Elbehery, I think we at a high level are here:

• A "create only" default Backup CR will be created
  • Users can modify this resource if they disagree with the default configuration
  • If a user deletes this configuration, it will be recreated
• The PVC configuration is omitted on the Backup CR, which implicitly means that a hostpath is used
  • This is bad from an API perspective, we need a required storage choice discriminated union to ensure an explicit choice from the user
  • Current implementation ignores the PVC name if the Backup CR has a special name - this is confusing UX and doesn't allow user to update the configuration
  • Current implementation applies various special logical choices based on the CR name only, this should be within the API and an explicit choice
• HostPath no-config approach is a branch from the main periodic backup logic
  • It creates a DS with pods on each of the master nodes
  • Pods run continuously, and handle backups on cron schedule
  • Completely separate implementation to the existing cronjob implementation
  • This means implementing configuration of Backup CR twice for two backup paths - will want testing to ensure all options work across both implementations, perhaps some refactoring could deduplicate here to avoid this scenario
  • Enhancement needs to be updated to explain how the pruning works for this implementation (as it differs from the normal scenario)
• The sidecar container approach has been dismissed and needs to be moved to the alternatives
• A third path of using the existing CronJob approach has been suggested (alternatives section is outdated wrt this)
  • Recently proven to work (over shutdown), with parallelism and completion configuration
  • Re-uses existing cronjob logic, but modifies some parameters when "hostpath" is configured (TBD, needs an API change)
  • Potentially easier to maintain in the future?
  • Open questions about changes that might be required around backup pruning here

Based on this, I think the next steps should probably be:

• Remove special reliances on the CR name, and move explicit choice of storage (PVC vs hostpath) to the Backup API
• Ensure that when the ^ API configures hostpath, the appropriate controller handles this correctly
• Ensure that any user change to the default Backup CR is observed and enacted
• Write up the parallelism/job based approach that was tested over shutdown, exploring pros and cons as with the other two approaches
• Resolve open questions about DS and Job based approaches to a point where we could discuss and make a decision on which to move forward with
• Discuss/make a decision on cronjob vs DS approach based on each of their pros and cons, move one to alternatives
• Move sidecar approach to alternatives

I don't have a strong opinion on the DS vs Job based approach, both seem to have pros and cons. I would generally lean toward having 1 implementation of the backup mechanics rather than two though, since I expect over time having two implementations will lead to "this feature got implemented but only works in PVC mode" kind of bugs coming up. Having reviewed the implementations of both in POC form, neither is super complex though IMO

[openshift-bot]

Inactive enhancement proposals go stale after 28d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Mark the proposal as fresh by commenting /remove-lifecycle stale.
Stale proposals rot after an additional 7d of inactivity and eventually close.
Exclude this proposal from closing by commenting /lifecycle frozen.

If this proposal is safe to close now please do so with /close.

/lifecycle stale

[Elbehery]

/remove-lifecycle stale