RFE: use learner mode for joining etcd members

[fabriziopandini]

Growing a local etcd cluster is a complex operation, and in the past, we already faced some issues like e.g. kubernetes-sigs/kind#588

Now that the implementation of the etcd learner mode is progressing, we should start considering if to use it in kubeadm in order to make join --control-plane implementation more robust.

at a high level what we would like to achieve is:

• a new etcd member should be created as a learner and became a voting member only after the etcd data are fully aligned.
  ideally
• we should also prevent the api-server to read from a learner node

Ref docs:

• https://etcd.io/docs/v3.3.12/learning/learner/
• https://etcd.io/docs/v3.4.0/learning/design-learner/

---

(edit by neolit123)

1.26:

• initial KEP draft
  https://hackmd.io/@DAKGcrh_RpC5vlt8w5bf8A/r1qoLh9zj
• KEP PR + tracking issue at k/e:
  sig-cl/kubeadm: add provisional KEP for etcd learner mode enhancements#3615
  sig-cl/kubeadm: Use etcd's learner mode enhancements#3614

1.27(alpha):

• k/k alpha feature gate code
  kubeadm: enable etcd's learner mode when joining etcd members kubernetes#113318
  kubeadm: fix etcd learner join kubernetes#115038
• CI added
  add e2e test for etcd learner mode #2807
  kubeadm: add e2e test for etcd learner mode test-infra#28574

1.29(beta):

• update etcd learner mode status of kubeadm implementation enhancements#4172
• kubeadm: promote EtcdLearnerMode to beta kubernetes#120228
• kubeadm: EtcdLearnerMode is beta in v1.29 website#42801
• add a new blog for etcd learner mode usage in kubeadm join website#43095

1.32(GA):

• [1.32]kubeadm: promote etcd learner mode to GA  kubernetes#126374
• kinder: remove learner-mode workflow #3096
• kubeadm: remove learner-mode test job test-infra#33316
• kubeadm-init: mark EtcdLearnerMode as GA website#47563
• kubeadm: check whether the peer URL for the added etcd member already exists when the MemberAddAsLearner/MemberAdd fails kubernetes#127491 and kubeadm: fix a bug where the RemoveMember function did not return the correct member list when the member to be removed did not exist kubernetes#127650 (Cherry-picked to v1.28-1.31)

1.33:

• kubeadm: remove the GA EtcdLearnerMode FG kubernetes#129589
• kubeadm: remove the EtcdLearnerMode FG website#49410

[SataQiu]

/cc

[rosti]

We have to be careful, but we certainly need to act upon it. The plan is that from etcd 3.5 new members will be joined only as learners.
In etcd 3.5 it will be possible to use a learner node for reading, but still the problem with writing continues. And, as LBs are out of the scope of kubeadm, things might become a bit difficult.
We probably need to direct API servers to healthy leaders and possibly do that via an etcd LB. Another possibility is to not expose the API servers, that have a local learner etcd node from the API LB (not sure if this would actually work though).
In short, we need to experiment a bit with this to find what's viable and easy for use.

[RA489]

/assign