You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -31,9 +33,9 @@ Provide a user-configurable interface to support encryption of data stored in et
31
33
32
34
## Motivation
33
35
34
-
Today, we support local AES encryption at the datastore layer which protects against etcd data leaks in the event of a etcd backup compromise. However, aescbc and aesgcm which are supported ecncryption technologies today available in OpenShift do not protect against online host compromise i.e. in such cases attacker can decrypt encrypted data from etcd using local keys, KMS managed keys protects against such scenarios.
36
+
Today, we support local AES encryption at the datastore layer. It protects against etcd data leaks in the event of a etcd backup compromise. However, aescbc and aesgcm, which are supported encryption technologies today available in OpenShift do not protect against online host compromise i.e. in such cases, attackers can decrypt encrypted data from etcd using local keys, KMS managed keys protects against such scenarios.
35
37
36
-
Users of OpenShift would like the encrypt secret data in etcd using self-managed KMSbacked keys.
38
+
Users of OpenShift would like to encrypt secret data in etcd using self-managed KMS-backed keys.
37
39
-https://issues.redhat.com/browse/OCPSTRAT-108
38
40
39
41
### User Stories
@@ -43,8 +45,8 @@ As an OpenShift administrator, I want to encrypt secrets in my cluster at rest u
43
45
As an OpenShift administrator, I want to let the KMS provider manage the lifecycle of the encryption keys.
44
46
45
47
As an OpenShift user, I enable encryption by setting apiserver.spec.encryption.type to kms.
46
-
- After some time passes, user makes a backup of etcd.
47
-
- The user confirms that the secret values are encrypted by checking to see if they have the related kms prefix.
48
+
- After some time passes, a user makes a backup of etcd.
49
+
- The user confirms that the secret values are encrypted by checking to see if they have the related KMS prefix.
48
50
49
51
### Goals
50
52
@@ -58,13 +60,48 @@ As an OpenShift user, I enable encryption by setting apiserver.spec.encryption.t
58
60
### Non-Goals
59
61
60
62
1. Allow the user to force key rotation
61
-
2. Support for the complete lifecyle of KMSmanaged keys directly within OpenShift controlplane
63
+
2. Support for the complete lifecycle of KMS-managed keys directly within OpenShift control-plane
62
64
3. Support for hardware security modules
63
-
4. The user has in-depth understanding of each phase of the encryption process
65
+
4. The user has an in-depth understanding of each phase of the encryption process
64
66
5. Completely recover the cluster in the event of the KMS instance itself going down or keys getting lost
65
67
6. Allow users to configure which resources will be encrypted
66
68
67
-
### Proposal
69
+
## Proposal
70
+
71
+
To support KMS encryption in OpenShift, we will be able to leverage the work that was done in [upstream Kubernetes](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/). However, we will need to extend and adapt the encryption workflow in OpenShift to support new constraints introduced by the externalization of encryption keys in a KMS. Because OpenShift will not own the keys from the KMS, we will also need to provide tools to the users to detect KMS-related failures and take action toward recovering their clusters whenever possible.
72
+
73
+
There exist two versions of the KMS API upstream today. In OpenShift, we specifically want to use KMS v2 only as it is an optimized version of the first API that is more production-ready than its predecessor, which was infamous for its impact on performance and the pressure it puts on the KMS.
74
+
75
+
In OpenShift, we will add a new encryption type to the list in the apiserver API as `kms`, similar to the list of possible encryption providers from the apiserver's [EncryptionConfig](https://github.com/kubernetes/apiserver/blob/cccad306d649184bf2a0e319ba830c53f65c445c/pkg/apis/apiserver/types_encryption.go#L89-L101).
76
+
77
+
Unlike the other providers, KMS will require additional configuration from the users. Because of that, we will add a new configuration to the API to allow users to configure the connection to their KMS. In this new API, users will be able to configure some options specific to the KMS provider they want to use. There will be dedicated APIs for each KMS supported by OpenShift.
78
+
79
+
From a UX perspective, these are the only changes the KMS feature will introduce. It is intentionally minimal to reduce the burden on the users and the potential for errors.
80
+
81
+
In practice, this feature will re-use as much of the existing encryption logic as possible. It will leverage the existing encryption and migration workflow introduced for AES-CBC and AES-GCM. However, unlike the aescbc and aesgcm providers, the encryption keys for KMS are not managed by the apiserver operators, so we have to extend the existing controllers to support that new workflow. On top of that, the operators used to be in charge of rotating the keys, but it will now be in the hands of the users to rotate the keys. We will need to introduce a new workflow to react when keys are rotated in the external KMS to make sure that the encrypted data is migrated to use the new key.
82
+
83
+
### KMS plugins
84
+
85
+
One aspect of the upstream feature that wasn't mentioned yet is that it requires a third-party application called a KMS plugin to bridge between the apiservers and the KMS. In OpenShift, these plugins will be configured and managed by the kube-apiserver-operator. There are multiple reasons behind this choice:
86
+
87
+
1. Reduces the complexity for the users that want to use the KMS feature
88
+
2. Simplifies key rotation when users manually rotate the key because it requires creating a second instance of the plugin that would use the new key while the old plugin would still allow using the old key as a read key
89
+
3. It is cheap to maintain the plugins as we can either leverage the upstream communities or the vendors
90
+
4. Have more trust and guarantees towards the plugins that will be running in the platform
91
+
92
+
Plugins available in the open will be forked and maintained downstream. The images for these plugins will be published and distributed on the official Red Hat registry in the same way as they are today for HyperShift.
+// APIServerEncryption is used to encrypt sensitive resources on the cluster.
83
-
+// +openshift:validation:FeatureGateAwareXValidation:featureGate=KMSEncryptionProvider,rule="has(self.type) && self.type == 'KMS' ? has(self.kms) : !has(self.kms)",message="kms config is required when encryption type is KMS, and forbidden otherwise"
84
-
+// +union
85
-
type APIServerEncryption struct {
86
-
// type defines what encryption type should be used to encrypt resources at the datastore layer.
87
-
// When this field is unset (i.e. when it is set to the empty string), identity is implied.
88
-
@@ -191,9 +194,23 @@ type APIServerEncryption struct {
89
-
// +unionDiscriminator
90
-
// +optional
91
-
Type EncryptionType `json:"type,omitempty"`
92
-
+
93
-
+ // kms defines the configuration for the external KMS instance that manages the encryption keys,
94
-
+ // when KMS encryption is enabled sensitive resources will be encrypted using keys managed by an
95
-
+ // externally configured KMS instance.
96
-
+ //
97
-
+ // The Key Management Service (KMS) instance provides symmetric encryption and is responsible for
98
-
+ // managing the lifecyle of the encryption keys outside of the control plane.
99
-
+ // This allows integration with an external provider to manage the data encryption keys securely.
// aesgcm refers to a type where AES-GCM with random nonce and a 32-byte key
115
-
// is used to perform encryption at the datastore layer.
116
-
EncryptionTypeAESGCM EncryptionType = "aesgcm"
116
+
// aesgcm refers to a type where AES-GCM with random nonce and a 32-byte key
117
+
// is used to perform encryption at the datastore layer.
118
+
EncryptionTypeAESGCM EncryptionType = "aesgcm"
117
119
+
118
-
+// kms refers to a type of encryption where the encryption keys are managed
119
-
+// outside the control plane in a Key Management Service instance,
120
-
+// encryption is still performed at the datastore layer.
121
-
+EncryptionTypeKMS EncryptionType = "KMS"
120
+
+// kms refers to a type of encryption where the encryption keys are managed
121
+
+// outside the control plane in a Key Management Service instance,
122
+
+// encryption is still performed at the datastore layer.
123
+
+EncryptionTypeKMS EncryptionType = "KMS"
122
124
)
123
-
124
-
type APIServerStatus struct {
125
+
```
126
+
127
+
The default value today is an empty string, which implies identity and that no encryption is used in the cluster by default. Other possible local encryption schemes include `aescbc` and `aesgcm`, which will remain as-is. Similar to how local AES encryption works, the apiserver operators will observe this config and apply the KMS EncryptionProvider to the EncryptionConfig.
128
+
129
+
```diff
130
+
@@ -191,9 +194,23 @@ type APIServerEncryption struct {
131
+
// +unionDiscriminator
132
+
// +optional
133
+
Type EncryptionType `json:"type,omitempty"`
134
+
+
135
+
+ // kms defines the configuration for the external KMS instance that manages the encryption keys,
136
+
+ // when KMS encryption is enabled sensitive resources will be encrypted using keys managed by an
137
+
+ // externally configured KMS instance.
138
+
+ //
139
+
+ // The Key Management Service (KMS) instance provides symmetric encryption and is responsible for
140
+
+ // managing the lifecyle of the encryption keys outside of the control plane.
141
+
+ // This allows integration with an external provider to manage the data encryption keys securely.
+// KMSConfig defines the configuration for the KMS instance
134
161
+// that will be used with KMSEncryptionProvider encryption
135
162
+// +kubebuilder:validation:XValidation:rule="has(self.type) && self.type == 'AWS' ? has(self.aws) : !has(self.aws)",message="aws config is required when kms provider type is AWS, and forbidden otherwise"
136
163
+// +union
137
164
+type KMSConfig struct {
138
-
+ // type defines the kind of platform for the KMS provider.
139
-
+ // Available provider types are AWS only.
140
-
+ //
141
-
+ // +unionDiscriminator
142
-
+ // +kubebuilder:validation:Required
143
-
+ Type KMSProviderType `json:"type"`
165
+
+ // type defines the kind of platform for the KMS provider
166
+
+ //
167
+
+ // +unionDiscriminator
168
+
+ // +kubebuilder:validation:Required
169
+
+ Type KMSProviderType `json:"type"`
144
170
+
145
-
+// aws defines the key config for using an AWS KMS instance
146
-
+// for the encryption. The AWS KMS instance is managed
147
-
+// by the user outside the purview of the control plane.
148
-
+//
149
-
+// +unionMember
150
-
+// +optional
151
-
+AWS *AWSKMSConfig `json:"aws,omitempty"`
171
+
+// aws defines the key config for using an AWS KMS instance
172
+
+// for the encryption. The AWS KMS instance is managed
173
+
+// by the user outside the purview of the control plane.
174
+
+//
175
+
+// +unionMember
176
+
+// +optional
177
+
+AWS *AWSKMSConfig `json:"aws,omitempty"`
152
178
+}
179
+
180
+
+// KMSProviderType is a specific supported KMS provider
181
+
+// +kubebuilder:validation:Enum="";AWS
182
+
+type KMSProviderType string
153
183
+
184
+
+const (
185
+
+ // AWSKMSProvider represents a supported KMS provider for use with AWS KMS
186
+
+ AWSKMSProvider KMSProviderType = "AWS"
187
+
+)
188
+
```
189
+
190
+
This configuration will also include an enum of the various KMS supported by OCP. At first, it will only have the `AWS` type, but we will add more as we progress on the feature. This enum is essential to avoid potential ambiguities that might arise with future KMS.
191
+
192
+
Each KMS type will have a dedicated configuration that will be reflected on the plugin when installed. It will only be a partial representation of the plugin's configuration because most fields are irrelevant to the end users.
193
+
194
+
At first, this configuration will only include the `AWSKMSConfig`, but more KMS-specifc configs will be added as we include new KMS.
195
+
196
+
```diff
154
197
+// AWSKMSConfig defines the KMS config specific to AWS KMS provider
155
198
+type AWSKMSConfig struct {
156
199
+ // keyARN specifies the Amazon Resource Name (ARN) of the AWS KMS key used for encryption.
@@ -169,18 +212,11 @@ index 000000000..575affae6
169
212
+ // +kubebuilder:validation:XValidation:rule="self.matches('^[a-z]{2}-[a-z]+-[0-9]+$') && self.size() <= 64",message="region must be a valid AWS region"
170
213
+ Region string `json:"region"`
171
214
+}
172
-
+
173
-
+// KMSProviderType is a specific supported KMS provider
174
-
+// +kubebuilder:validation:Enum="";AWS
175
-
+type KMSProviderType string
176
-
+
177
-
+const (
178
-
+ // AWSKMSProvider represents a supported KMS provider for use with AWS KMS
179
-
+ AWSKMSProvider KMSProviderType = "AWS"
180
-
+)
181
215
```
182
216
183
-
The default value today is an empty string which implies identity and that no encryption is used in the cluster by default. Other possible local encryption schemes include `aescbc` and `aesgcm` which will remain as-is. Similar to how local AES encryption works, the kube-apiserver operator and openshift-apiserver operator will observe this config to apply the KMS Encryption Provider config onto kube-apiserver(s) and openshift-apiserver(s) respectively.
217
+
From a very high-level, the figure below shows how the new APIs will be used by the various components.
0 commit comments