Skip to content
This repository was archived by the owner on Mar 16, 2024. It is now read-only.

Commit a485a3e

Browse files
g-linvilleg-linville
authored
Redesign NetworkPolicies (#456) (#1352)
Signed-off-by: Grant Linville <[email protected]>
1 parent 9bc0286 commit a485a3e

File tree

26 files changed

+848
-10
lines changed

26 files changed

+848
-10
lines changed

‎docs/docs/100-reference/01-command-line/acorn_install.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ acorn install
2121
```
2222
--acorn-dns string enabled|disabled|auto. If enabled, containers created by Acorn will get public FQDNs. Auto functions as disabled if a custom clusterDomain has been supplied (default auto)
2323
--acorn-dns-endpoint string The URL to access the Acorn DNS service
24+
--allow-traffic-from-namespace strings Namespaces that are allowed to send network traffic to all Acorn apps
2425
--allow-user-annotation strings Allow these annotations to propagate to dependent objects, no effect if --ignore-user-labels-and-annotations not true
2526
--allow-user-label strings Allow these labels to propagate to dependent objects, no effect if --ignore-user-labels-and-annotations not true
2627
--api-server-replicas int acorn-api deployment replica count
@@ -33,12 +34,14 @@ acorn install
3334
--ignore-user-labels-and-annotations Don't propagate user-defined labels and annotations to dependent objects
3435
--image string Override the default image used for the deployment
3536
--ingress-class-name string The ingress class name to assign to all created ingress resources (default '')
37+
--ingress-controller-namespace string The namespace where the ingress controller runs - used to secure published HTTP ports with NetworkPolicies.
3638
--internal-cluster-domain string The Kubernetes internal cluster domain (default svc.cluster.local)
3739
--internal-registry-prefix string The image prefix to use when pushing internal images (example ghcr.io/my-org/)
3840
--lets-encrypt string enabled|disabled|staging. If enabled, acorn generated endpoints will be secured using TLS certificate from Let's Encrypt. Staging uses Let's Encrypt's staging environment. (default disabled)
3941
--lets-encrypt-email string Required if --lets-encrypt=enabled. The email address to use for Let's Encrypt registration(default '')
4042
--lets-encrypt-tos-agree Required if --lets-encrypt=enabled. If true, you agree to the Let's Encrypt terms of service (default false)
4143
--manage-volume-classes Manually manage volume classes rather than sync with storage classes, setting to 'true' will delete Acorn-created volume classes
44+
--network-policies Create Kubernetes NetworkPolicies which block cross-project network traffic (default true)
4245
-o, --output string Output manifests instead of applying them (json, yaml)
4346
--pod-security-enforce-profile string The name of the PodSecurity profile to set (default baseline)
4447
--propagate-project-annotation strings The list of keys of annotations to propagate from acorn project to app namespaces

‎docs/docs/30-installation/02-options.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,13 @@ The default installation of Acorn will automatically create and sync any storage
9494

9595
If an admin would rather manually manage the volume classes and not have these generated ones, then the `--manage-volume-classes` installation flag is available. The generated volume classes are not generated if this flag is used, and are deleted when the flag is set on an existing Acorn installation. If the flag is again switched off with `--manage-volume-classes=false`, then the volume classes will be generated again.
9696

97+
## Kubernetes NetworkPolicies
98+
By default, Acorn will automatically create and manage Kubernetes [NetworkPolicies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) to isolate Acorn projects on the network level. This behavior can be disabled by passing `--network-policies=false` to `acorn install`, and can later be re-enabled by passing `--network-policies=true`.
99+
100+
By default, Acorn workloads that publish ports that use HTTP will be allowed to receive traffic from internal (other pods in the cluster) and external (through the cluster's ingress) sources. To secure this further, you can require all traffic to Acorn workloads flow through your ingress by specifying the `--ingress-controller-namespace` parameter during installation.
101+
102+
To allow traffic from a specific namespace to all Acorn apps in the cluster, use `--allow-traffic-from-namespace=<namespace>`. This is useful if there is a monitoring namespace, for example, that needs to be able to connect to all the pods created by Acorn in order to scrape metrics.
103+
97104
## Changing install options
98105
If you want to change your installation options after the initial installation, just rerun `acorn install` with the new options. This will update the existing install dynamically.
99106

‎integration/run/run_test.go‎

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,14 +17,17 @@ import (
1717
kclient "github.com/acorn-io/acorn/pkg/k8sclient"
1818
"github.com/acorn-io/acorn/pkg/labels"
1919
"github.com/acorn-io/acorn/pkg/run"
20+
"github.com/acorn-io/acorn/pkg/scheme"
2021
"github.com/acorn-io/acorn/pkg/tolerations"
22+
"github.com/acorn-io/baaah/pkg/restconfig"
2123
"github.com/acorn-io/baaah/pkg/router"
2224
"github.com/stretchr/testify/assert"
2325
corev1 "k8s.io/api/core/v1"
2426
storagev1 "k8s.io/api/storage/v1"
2527
apierrors "k8s.io/apimachinery/pkg/api/errors"
2628
"k8s.io/apimachinery/pkg/api/resource"
2729
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
30+
k8slabels "k8s.io/apimachinery/pkg/labels"
2831
crClient "sigs.k8s.io/controller-runtime/pkg/client"
2932
)
3033

@@ -1135,3 +1138,181 @@ func getStorageClassName(t *testing.T, storageClasses *storagev1.StorageClassLis
11351138
}
11361139
return storageClassName
11371140
}
1141+
1142+
func TestCrossProjectNetworkConnection(t *testing.T) {
1143+
helper.StartController(t)
1144+
1145+
rc, err := restconfig.New(scheme.Scheme)
1146+
if err != nil {
1147+
t.Fatal("error while getting rest config:", err)
1148+
}
1149+
ctx := helper.GetCTX(t)
1150+
c, _ := helper.ClientAndNamespace(t)
1151+
kc := helper.MustReturn(kclient.Default)
1152+
1153+
// create two separate projects in which to run two Nginx apps
1154+
proj1, err := c.ProjectCreate(ctx, "proj1", "local")
1155+
if err != nil {
1156+
t.Fatal("error while creating project:", err)
1157+
}
1158+
proj1Client, err := client.New(rc, proj1.Name, proj1.Status.Namespace)
1159+
if err != nil {
1160+
t.Fatal("error creating client for proj1:", err)
1161+
}
1162+
1163+
proj2, err := c.ProjectCreate(ctx, "proj2", "local")
1164+
if err != nil {
1165+
t.Fatal("error while creating project:", err)
1166+
}
1167+
proj2Client, err := client.New(rc, proj2.Name, proj2.Status.Namespace)
1168+
if err != nil {
1169+
t.Fatal("error creating client for proj2:", err)
1170+
}
1171+
1172+
t.Cleanup(func() {
1173+
// clean up projects
1174+
_, err := proj1Client.ProjectDelete(ctx, "proj1")
1175+
if err != nil {
1176+
t.Log("failed to delete project 'proj1':", err)
1177+
}
1178+
_, err = proj2Client.ProjectDelete(ctx, "proj2")
1179+
if err != nil {
1180+
t.Log("failed to delete project 'proj2':", err)
1181+
}
1182+
})
1183+
1184+
// create both apps, one in proj1 and the other in proj2
1185+
// app "foo" in proj1 does not publish any ports
1186+
// app "bar" in proj2 publishes port 80
1187+
fooImage, err := proj1Client.AcornImageBuild(ctx, "testdata/networkpolicy/Acornfile", nil)
1188+
if err != nil {
1189+
t.Fatal("error while building image:", err)
1190+
}
1191+
fooApp, err := proj1Client.AppRun(ctx, fooImage.ID, &client.AppRunOptions{
1192+
Name: "foo",
1193+
TargetNamespace: proj1.Namespace,
1194+
})
1195+
if err != nil {
1196+
t.Fatal("error while running app:", err)
1197+
}
1198+
1199+
barImage, err := proj2Client.AcornImageBuild(ctx, "testdata/networkpolicy/publish.Acornfile", nil)
1200+
if err != nil {
1201+
t.Fatal("error while building image:", err)
1202+
}
1203+
barApp, err := proj2Client.AppRun(ctx, barImage.ID, &client.AppRunOptions{
1204+
Name: "bar",
1205+
TargetNamespace: proj2.Namespace,
1206+
})
1207+
if err != nil {
1208+
t.Fatal("error while running app:", err)
1209+
}
1210+
1211+
// wait for both apps to be ready
1212+
helper.WaitForObject(t, helper.Watcher(t, proj1Client), &apiv1.AppList{}, fooApp, func(obj *apiv1.App) bool {
1213+
return obj.Status.Condition(v1.AppInstanceConditionReady).Success
1214+
})
1215+
helper.WaitForObject(t, helper.Watcher(t, proj2Client), &apiv1.AppList{}, barApp, func(obj *apiv1.App) bool {
1216+
return obj.Status.Condition(v1.AppInstanceConditionReady).Success
1217+
})
1218+
1219+
// determine pod IPs so we can test network connections
1220+
fooIP := getPodIPFromAppName(t, ctx, &kc, fooApp.Name, fooApp.Status.Namespace)
1221+
barIP := getPodIPFromAppName(t, ctx, &kc, barApp.Name, barApp.Status.Namespace)
1222+
1223+
// build an Acorn that just runs a job with the official curl container
1224+
curlImage1, err := proj1Client.AcornImageBuild(ctx, "testdata/networkpolicy/curl.Acornfile", nil)
1225+
if err != nil {
1226+
t.Fatal("error while building image:", err)
1227+
}
1228+
curlImage2, err := proj2Client.AcornImageBuild(ctx, "testdata/networkpolicy/curl.Acornfile", nil)
1229+
if err != nil {
1230+
t.Fatal("error while building image:", err)
1231+
}
1232+
1233+
checks := []struct {
1234+
name string
1235+
client client.Client
1236+
podIP string
1237+
imageID string
1238+
expectFailure bool
1239+
}{
1240+
{
1241+
name: "curl-foo-proj1",
1242+
client: proj1Client,
1243+
podIP: fooIP,
1244+
imageID: curlImage1.ID,
1245+
expectFailure: false,
1246+
},
1247+
{
1248+
name: "curl-bar-proj1",
1249+
client: proj1Client,
1250+
podIP: barIP,
1251+
imageID: curlImage1.ID,
1252+
// even though bar has port 80 published, it should not be reachable from anything outside
1253+
// proj2 except for the ingress controller, so failure is expected here
1254+
expectFailure: true,
1255+
},
1256+
{
1257+
name: "curl-foo-proj2",
1258+
client: proj2Client,
1259+
podIP: fooIP,
1260+
imageID: curlImage2.ID,
1261+
expectFailure: true,
1262+
},
1263+
{
1264+
name: "curl-bar-proj2",
1265+
client: proj2Client,
1266+
podIP: barIP,
1267+
imageID: curlImage2.ID,
1268+
expectFailure: false,
1269+
},
1270+
}
1271+
for _, check := range checks {
1272+
t.Run(check.name, func(t *testing.T) {
1273+
// run curl to test the connection
1274+
app, err := check.client.AppRun(ctx, check.imageID, &client.AppRunOptions{
1275+
Name: check.name,
1276+
DeployArgs: map[string]any{
1277+
"address": check.podIP,
1278+
},
1279+
})
1280+
if err != nil {
1281+
t.Fatal("error while running app:", err)
1282+
}
1283+
1284+
appInstance := helper.WaitForObject(t, helper.Watcher(t, check.client), &apiv1.AppList{}, app, func(obj *apiv1.App) bool {
1285+
return obj.Status.JobsStatus["curl"].Failed || obj.Status.JobsStatus["curl"].Succeed
1286+
})
1287+
1288+
if check.expectFailure {
1289+
assert.Equal(t, true, appInstance.Status.JobsStatus["curl"].Failed)
1290+
} else {
1291+
assert.Equal(t, true, appInstance.Status.JobsStatus["curl"].Succeed)
1292+
}
1293+
})
1294+
}
1295+
}
1296+
1297+
func getPodIPFromAppName(t *testing.T, ctx context.Context, kc *crClient.WithWatch, appName, namespace string) string {
1298+
t.Helper()
1299+
selector, err := k8slabels.Parse(fmt.Sprintf("%s=%s", labels.AcornAppName, appName))
1300+
if err != nil {
1301+
t.Fatal("error creating k8s label selector:", err)
1302+
}
1303+
1304+
var podList corev1.PodList
1305+
podIP := ""
1306+
for podIP == "" {
1307+
err = (*kc).List(ctx, &podList, &kclient.ListOptions{
1308+
LabelSelector: selector,
1309+
Namespace: namespace,
1310+
})
1311+
if err != nil {
1312+
t.Fatal("error listing pods:", err)
1313+
}
1314+
podIP = podList.Items[0].Status.PodIP
1315+
}
1316+
1317+
return podIP
1318+
}

‎integration/run/testdata/networkpolicy/Acornfile‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
containers: {
2+
foo: {
3+
image: "public.ecr.aws/docker/library/nginx:latest"
4+
ports: "80/http"
5+
}
6+
}

‎integration/run/testdata/networkpolicy/curl.Acornfile‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
args: {
2+
address: ""
3+
}
4+
5+
jobs: {
6+
curl: {
7+
image: "curlimages/curl:latest"
8+
command: ["sh", "-c", "sleep 5 && curl " + args.address]
9+
}
10+
}

‎integration/run/testdata/networkpolicy/publish.Acornfile‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
containers: {
2+
bar: {
3+
image: "ghcr.io/acorn-io/images-mirror/nginx:latest"
4+
ports: publish: "80/http"
5+
}
6+
}

‎pkg/apis/api.acorn.io/v1/types.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -346,6 +346,9 @@ type Config struct {
346346
PropagateProjectAnnotations []string `json:"propagateProjectAnnotations" name:"propagate-project-annotation" usage:"The list of keys of annotations to propagate from acorn project to app namespaces"`
347347
PropagateProjectLabels []string `json:"propagateProjectLabels" name:"propagate-project-label" usage:"The list of keys of labels to propagate from acorn project to app namespaces"`
348348
ManageVolumeClasses *bool `json:"manageVolumeClasses" name:"manage-volume-classes" usage:"Manually manage volume classes rather than sync with storage classes, setting to 'true' will delete Acorn-created volume classes"`
349+
NetworkPolicies *bool `json:"networkPolicies" name:"network-policies" usage:"Create Kubernetes NetworkPolicies which block cross-project network traffic (default true)"`
350+
IngressControllerNamespace *string `json:"ingressControllerNamespace" name:"ingress-controller-namespace" usage:"The namespace where the ingress controller runs - used to secure published HTTP ports with NetworkPolicies."`
351+
AllowTrafficFromNamespace []string `json:"allowTrafficFromNamespace" name:"allow-traffic-from-namespace" usage:"Namespaces that are allowed to send network traffic to all Acorn apps"`
349352
}
350353

351354
type EncryptionKey struct {

‎pkg/apis/api.acorn.io/v1/zz_generated.deepcopy.go‎

Lines changed: 15 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pkg/cli/testdata/info/info_test-a.txt‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ projects:
1010
config:
1111
acornDNS: null
1212
acornDNSEndpoint: null
13+
allowTrafficFromNamespace: null
1314
allowUserAnnotations: null
1415
allowUserLabels: null
1516
autoUpgradeInterval: null
@@ -18,12 +19,14 @@ projects:
1819
httpEndpointPattern: null
1920
ignoreUserLabelsAndAnnotations: null
2021
ingressClassName: null
22+
ingressControllerNamespace: null
2123
internalClusterDomain: ""
2224
internalRegistryPrefix: null
2325
letsEncrypt: null
2426
letsEncryptEmail: ""
2527
letsEncryptTOSAgree: null
2628
manageVolumeClasses: null
29+
networkPolicies: null
2730
podSecurityEnforceProfile: ""
2831
propagateProjectAnnotations: null
2932
propagateProjectLabels: null
@@ -40,6 +43,7 @@ projects:
4043
userConfig:
4144
acornDNS: null
4245
acornDNSEndpoint: null
46+
allowTrafficFromNamespace: null
4347
allowUserAnnotations: null
4448
allowUserLabels: null
4549
autoUpgradeInterval: null
@@ -48,12 +52,14 @@ projects:
4852
httpEndpointPattern: null
4953
ignoreUserLabelsAndAnnotations: null
5054
ingressClassName: null
55+
ingressControllerNamespace: null
5156
internalClusterDomain: ""
5257
internalRegistryPrefix: null
5358
letsEncrypt: null
5459
letsEncryptEmail: ""
5560
letsEncryptTOSAgree: null
5661
manageVolumeClasses: null
62+
networkPolicies: null
5763
podSecurityEnforceProfile: ""
5864
propagateProjectAnnotations: null
5965
propagateProjectLabels: null
@@ -68,6 +74,7 @@ projects:
6874
config:
6975
acornDNS: null
7076
acornDNSEndpoint: null
77+
allowTrafficFromNamespace: null
7178
allowUserAnnotations: null
7279
allowUserLabels: null
7380
autoUpgradeInterval: null
@@ -76,12 +83,14 @@ projects:
7683
httpEndpointPattern: null
7784
ignoreUserLabelsAndAnnotations: null
7885
ingressClassName: null
86+
ingressControllerNamespace: null
7987
internalClusterDomain: ""
8088
internalRegistryPrefix: null
8189
letsEncrypt: null
8290
letsEncryptEmail: ""
8391
letsEncryptTOSAgree: null
8492
manageVolumeClasses: null
93+
networkPolicies: null
8594
podSecurityEnforceProfile: ""
8695
propagateProjectAnnotations: null
8796
propagateProjectLabels: null
@@ -98,6 +107,7 @@ projects:
98107
userConfig:
99108
acornDNS: null
100109
acornDNSEndpoint: null
110+
allowTrafficFromNamespace: null
101111
allowUserAnnotations: null
102112
allowUserLabels: null
103113
autoUpgradeInterval: null
@@ -106,12 +116,14 @@ projects:
106116
httpEndpointPattern: null
107117
ignoreUserLabelsAndAnnotations: null
108118
ingressClassName: null
119+
ingressControllerNamespace: null
109120
internalClusterDomain: ""
110121
internalRegistryPrefix: null
111122
letsEncrypt: null
112123
letsEncryptEmail: ""
113124
letsEncryptTOSAgree: null
114125
manageVolumeClasses: null
126+
networkPolicies: null
115127
podSecurityEnforceProfile: ""
116128
propagateProjectAnnotations: null
117129
propagateProjectLabels: null

0 commit comments

Comments
 (0)