Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support nacos storage #796

Draft
wants to merge 43 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
bd58388
feat: support nacos storage
sjcsjc123 Jan 22, 2024
625c61d
recover something
sjcsjc123 Jan 22, 2024
d848bb8
recover something
sjcsjc123 Jan 22, 2024
e8888cb
recover something
sjcsjc123 Jan 22, 2024
d01a220
simple
sjcsjc123 Jan 22, 2024
d8bb8d3
fix bug
sjcsjc123 Jan 23, 2024
5aed7f4
modify nacos image
sjcsjc123 Jan 23, 2024
0ffcc30
update nacos image in Makefile
sjcsjc123 Jan 23, 2024
c0c4e38
add nacos client
sjcsjc123 Jan 23, 2024
8eb8f5b
add ci for nacos storage
sjcsjc123 Jan 23, 2024
9bb4920
delete something
sjcsjc123 Jan 23, 2024
31ca385
fix bug with start fail
sjcsjc123 Jan 23, 2024
b5fd8a7
rename make
sjcsjc123 Jan 23, 2024
85b8274
opt import
sjcsjc123 Jan 23, 2024
370d386
update e2e
sjcsjc123 Jan 24, 2024
01c08fe
opt import
sjcsjc123 Jan 24, 2024
06651de
recover something
sjcsjc123 Jan 24, 2024
35c5db1
modify default falg value
sjcsjc123 Jan 24, 2024
9449862
make ci print simple
sjcsjc123 Jan 24, 2024
ecb2672
update apply
sjcsjc123 Jan 24, 2024
62c421a
update apply
sjcsjc123 Jan 24, 2024
2e72259
add wasm+nacos ci
sjcsjc123 Jan 24, 2024
0c83d5f
update wasm+nacos ci
sjcsjc123 Jan 24, 2024
0715626
fix ci
sjcsjc123 Jan 24, 2024
62cb1dd
fix bug with wasm plugin
sjcsjc123 Jan 24, 2024
8ce8df9
Merge branch 'main' into helm/nacos-storage
sjcsjc123 Jan 27, 2024
cd3527e
Merge branch 'main' into helm/nacos-storage
sjcsjc123 Jan 31, 2024
4b5cd88
fix ci
sjcsjc123 Feb 1, 2024
a3ecec7
Merge branch 'main' into helm/nacos-storage
sjcsjc123 Feb 1, 2024
9349772
fix ci
sjcsjc123 Feb 1, 2024
a3b114b
fix ci
sjcsjc123 Feb 1, 2024
41bd16a
Merge branch 'main' into helm/nacos-storage
johnlanni Feb 3, 2024
598aaf0
Merge branch 'main' into helm/nacos-storage
johnlanni Feb 20, 2024
a8af197
Merge branch 'main' into helm/nacos-storage
sjcsjc123 Feb 28, 2024
6cd64b4
update nacos image
sjcsjc123 Feb 29, 2024
5575512
update nacos image
sjcsjc123 Feb 29, 2024
81810c0
fix ci
sjcsjc123 Feb 29, 2024
daba6c7
fix ci
sjcsjc123 Feb 29, 2024
32a6786
fix ci
sjcsjc123 Mar 1, 2024
0a425dd
fix ci
sjcsjc123 Mar 1, 2024
0f270ce
Merge branch 'main' into helm/nacos-storage
sjcsjc123 Mar 27, 2024
b306f61
Merge branch 'main' into helm/nacos-storage
johnlanni Apr 1, 2024
688442b
Merge branch 'main' into helm/nacos-storage
johnlanni Apr 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .github/workflows/build-and-test-plugin.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,16 @@ jobs:
steps:
- uses: actions/checkout@v3

- name: Free Up GitHub Actions Ubuntu Runner Disk Space 🔧
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
android: true
dotnet: true
haskell: true
large-packages: true
swap-storage: true

- name: "Setup Go"
uses: actions/setup-go@v3
with:
Expand Down
20 changes: 20 additions & 0 deletions .github/workflows/build-and-test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,16 @@ jobs:
needs: [build]
steps:
- uses: actions/checkout@v3

- name: Free Up GitHub Actions Ubuntu Runner Disk Space 🔧
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
android: true
dotnet: true
haskell: true
large-packages: true
swap-storage: true

- name: "Setup Go"
uses: actions/setup-go@v3
Expand Down Expand Up @@ -141,6 +151,16 @@ jobs:
steps:
- uses: actions/checkout@v3

- name: Free Up GitHub Actions Ubuntu Runner Disk Space 🔧
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
android: true
dotnet: true
haskell: true
large-packages: true
swap-storage: true

- name: "Setup Go"
uses: actions/setup-go@v3
with:
Expand Down
28 changes: 14 additions & 14 deletions Makefile.core.mk
Original file line number Diff line number Diff line change
Expand Up @@ -185,13 +185,11 @@ install-dev: pre-install
install-dev-wasmplugin: build-wasmplugins pre-install
helm install higress helm/core -n higress-system --create-namespace --set 'controller.tag=$(TAG)' --set 'gateway.replicas=1' --set 'pilot.tag=$(ISTIO_LATEST_IMAGE_TAG)' --set 'gateway.tag=$(ENVOY_LATEST_IMAGE_TAG)' --set 'global.local=true' --set 'global.volumeWasmPlugins=true' --set 'global.onlyPushRouteCluster=false'
install-dev-nacos: pre-install
kubectl apply -f tools/hack/conf/nacos.yaml
tools/hack/gen-secret-configmap.sh
helm install higress helm/core -n higress-system --set 'controller.tag=$(TAG)' --set 'gateway.replicas=1' --set 'pilot.tag=$(ISTIO_LATEST_IMAGE_TAG)' --set 'gateway.tag=$(ENVOY_LATEST_IMAGE_TAG)' --set 'global.local=true' --set 'apiserver.enabled=true' --set 'apiserver.serverAddr="http://nacos-service.higress-system.svc.cluster.local:8848"'
helm install higress helm/core -n higress-system --set 'controller.tag=$(TAG)' --set 'gateway.replicas=1' --set 'pilot.tag=$(ISTIO_LATEST_IMAGE_TAG)' --set 'gateway.tag=$(ENVOY_LATEST_IMAGE_TAG)' --set 'global.local=true' --set 'apiserver.enabled=true' --set 'nacos.enabled=true'
install-dev-wasmplugin-nacos: build-wasmplugins pre-install
kubectl apply -f tools/hack/conf/nacos.yaml
tools/hack/gen-secret-configmap.sh
helm install higress helm/core -n higress-system --set 'controller.tag=$(TAG)' --set 'gateway.replicas=1' --set 'pilot.tag=$(ISTIO_LATEST_IMAGE_TAG)' --set 'gateway.tag=$(ENVOY_LATEST_IMAGE_TAG)' --set 'global.local=true' --set 'apiserver.enabled=true' --set 'apiserver.serverAddr="http://nacos-service.higress-system.svc.cluster.local:8848"' --set 'global.volumeWasmPlugins=true' --set 'global.onlyPushRouteCluster=false'
helm install higress helm/core -n higress-system --set 'controller.tag=$(TAG)' --set 'gateway.replicas=1' --set 'pilot.tag=$(ISTIO_LATEST_IMAGE_TAG)' --set 'gateway.tag=$(ENVOY_LATEST_IMAGE_TAG)' --set 'global.local=true' --set 'apiserver.enabled=true' --set 'nacos.enabled=true' --set 'global.volumeWasmPlugins=true' --set 'global.onlyPushRouteCluster=false'

uninstall:
helm uninstall higress -n higress-system
Expand Down Expand Up @@ -275,8 +273,8 @@ higress-wasmplugin-test-nacos: $(tools/kind) delete-cluster create-cluster docke

.PHONY: port-forward-nacos
port-forward-nacos:
kubectl wait --timeout=10m -n higress-system deploy/nacos --for=condition=Available
kubectl port-forward -n higress-system svc/nacos-service 8848:8848 9848:9848 &
kubectl wait --timeout=10m -n higress-system --for=condition=Ready pod -l app.kubernetes.io/name=higress-nacos,app.kubernetes.io/instance=higress
kubectl port-forward -n higress-system svc/higress-nacos-service 8848:8848 9848:9848 &

# create-cluster creates a kube cluster with kind.
.PHONY: create-cluster
Expand All @@ -296,13 +294,15 @@ delete-cluster: $(tools/kind) ## Delete kind cluster.
kube-load-image: $(tools/kind) ## Install the Higress image to a kind cluster using the provided $IMAGE and $TAG.
tools/hack/kind-load-image.sh higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/higress $(TAG)
tools/hack/docker-pull-image.sh higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/dubbo-provider-demo 0.0.3-x86
tools/hack/docker-pull-image.sh docker.io/swsk33/nacos-standalone 2.2.3
tools/hack/docker-pull-image.sh docker.io/alihigress/nacos-standlone-rc3 1.0.0-RC3
tools/hack/docker-pull-image.sh docker.io/nacos/nacos-server latest
tools/hack/docker-pull-image.sh docker.io/hashicorp/consul 1.16.0
tools/hack/docker-pull-image.sh docker.io/charlie1380/eureka-registry-provider v0.3.0
tools/hack/docker-pull-image.sh docker.io/bitinit/eureka latest
tools/hack/docker-pull-image.sh docker.io/alihigress/httpbin 1.0.2
tools/hack/kind-load-image.sh higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/dubbo-provider-demo 0.0.3-x86
tools/hack/kind-load-image.sh docker.io/swsk33/nacos-standalone 2.2.3
tools/hack/kind-load-image.sh docker.io/alihigress/nacos-standlone-rc3 1.0.0-RC3
tools/hack/kind-load-image.sh docker.io/nacos/nacos-server latest
tools/hack/kind-load-image.sh docker.io/hashicorp/consul 1.16.0
tools/hack/kind-load-image.sh docker.io/alihigress/httpbin 1.0.2
tools/hack/kind-load-image.sh docker.io/charlie1380/eureka-registry-provider v0.3.0
Expand Down Expand Up @@ -367,8 +367,8 @@ run-higress-e2e-test-wasmplugin:
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
go test -v -tags conformance ./test/e2e/e2e_test.go -isWasmPluginTest=true -wasmPluginType=$(PLUGIN_TYPE) -wasmPluginName=$(PLUGIN_NAME) --ingress-class=higress --debug=true --test-area=all
# run-higress-e2e-test-wasmplugin-run starts to run ingress e2e conformance tests.

# run-higress-e2e-test-wasmplugin-run starts to run ingress e2e conformance tests.
.PHONY: run-higress-e2e-test-wasmplugin-run
run-higress-e2e-test-wasmplugin-run:
@echo -e "\n\033[36mRunning higress conformance tests...\033[0m"
Expand All @@ -377,8 +377,8 @@ run-higress-e2e-test-wasmplugin-run:
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
go test -v -tags conformance ./test/e2e/e2e_test.go -isWasmPluginTest=true -wasmPluginType=$(PLUGIN_TYPE) -wasmPluginName=$(PLUGIN_NAME) --ingress-class=higress --debug=true --test-area=run
# run-higress-e2e-test-wasmplugin-clean starts to clean ingress e2e tests.

# run-higress-e2e-test-wasmplugin-clean starts to clean ingress e2e tests.
.PHONY: run-higress-e2e-test-wasmplugin-clean
run-higress-e2e-test-wasmplugin-clean:
@echo -e "\n\033[36mRunning higress conformance tests...\033[0m"
Expand All @@ -387,7 +387,7 @@ run-higress-e2e-test-wasmplugin-clean:
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
go test -v -tags conformance ./test/e2e/e2e_test.go -isWasmPluginTest=true -wasmPluginType=$(PLUGIN_TYPE) -wasmPluginName=$(PLUGIN_NAME) --ingress-class=higress --debug=true --test-area=clean

.PHONY: run-higress-e2e-test-nacos
run-higress-e2e-test-nacos:
@echo -e "\n\033[36mRunning higress conformance tests...\033[0m"
Expand All @@ -396,7 +396,7 @@ run-higress-e2e-test-nacos:
@echo -e "\n\033[36mWaiting higress-gateway to be ready...\033[0m\n"
kubectl wait --timeout=10m -n higress-system deployment/higress-gateway --for=condition=Available
go test -v -tags conformance ./test/e2e/e2e_test.go --ingress-class=higress --enableApiServer=true

.PHONY: run-higress-e2e-test-wasmplugin-nacos
run-higress-e2e-test-wasmplugin-nacos:
@echo -e "\n\033[36mRunning higress conformance tests...\033[0m"
Expand Down
30 changes: 30 additions & 0 deletions helm/core/templates/nacos-service.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
{{- if .Values.nacos.enabled }}
apiVersion: v1
kind: Service
metadata:
name: {{ .Values.nacos.name }}-service
namespace: {{ .Release.Namespace }}
spec:
type: {{ .Values.nacos.service.type }}
ports:
- port: {{ .Values.nacos.service.port }}
targetPort: {{ .Values.nacos.service.port }}
protocol: TCP
name: http
- port: {{ add .Values.nacos.service.port 1000}}
name: client-rpc
targetPort: {{add .Values.nacos.service.port 1000}}
- port: {{add .Values.nacos.service.port 1001}}
name: raft-rpc
targetPort: {{add .Values.nacos.service.port 1001}}
- port: 7848
name: old-raft-rpc
targetPort: 7848
protocol: TCP
{{- if eq .Values.nacos.service.type "NodePort" }}
nodePort: {{ .Values.nacos.service.nodePort }}
{{- end }}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里格式处理一下吧。另外,什么情况下需要用到 NodePort 呢,还只有 7848?

selector:
app.kubernetes.io/name: {{ .Values.nacos.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end}}
104 changes: 104 additions & 0 deletions helm/core/templates/nacos-statefulset.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
{{- if .Values.nacos.enabled }}
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ .Values.nacos.name }}
namespace: {{ .Release.Namespace }}
annotations:
{{- toYaml .Values.nacos.annotations | indent 4 }}
spec:
serviceName: {{ .Values.nacos.name }}-service
replicas: {{ .Values.nacos.replica }}
{{- if .Values.nacos.podManagementPolicy }}
podManagementPolicy: {{ .Values.nacos.podManagementPolicy }}
{{- else}}
podManagementPolicy: OrderedReady
{{- end }}
selector:
matchLabels:
app.kubernetes.io/name: {{ .Values.nacos.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
template:
metadata:
labels:
app.kubernetes.io/name: {{ .Values.nacos.name }}
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
{{- with .Values.nacos.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.nacos.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.nacos.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: {{ .Values.nacos.name }}
image: "{{ .Values.nacos.hub }}:{{ .Values.nacos.tag }}"
imagePullPolicy: {{ .Values.nacos.pullPolicy }}
startupProbe:
initialDelaySeconds: 180
periodSeconds: 5
timeoutSeconds: 10
httpGet:
scheme: HTTP
port: {{ .Values.nacos.service.port }}
path: /nacos/v1/console/health/readiness
livenessProbe:
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 10
httpGet:
scheme: HTTP
port: {{ .Values.nacos.service.port }}
path: /nacos/v1/console/health/liveness
ports:
- name: http
containerPort: {{ .Values.nacos.service.port }}
protocol: TCP
- containerPort: {{ add .Values.nacos.service.port 1000}}
name: client-rpc
- containerPort: {{ add .Values.nacos.service.port 1001 }}
name: raft-rpc
- containerPort: 7848
name: old-raft-rpc
resources:
{{- toYaml .Values.nacos.resources | nindent 12 }}
env:
- name: NACOS_SERVER_PORT
value: {{ .Values.nacos.service.port | quote }}
- name: NACOS_APPLICATION_PORT
value: {{ .Values.nacos.service.port | quote }}
- name: PREFER_HOST_MODE
value: {{ .Values.nacos.preferHostMode | quote }}
- name: MODE
value: "standalone"
- name: EMBEDDED_STORAGE
value: embedded
volumeMounts:
- name: data
mountPath: /home/nacos/plugins/peer-finder
subPath: peer-finder
- name: data
mountPath: /home/nacos/data
subPath: data
- name: data
mountPath: /home/nacos/logs
subPath: logs
{{- if not .Values.nacos.persistence.enabled }}
volumes:
- name: data
emptyDir: { }
{{- end }}
{{- if .Values.nacos.persistence.enabled }}
volumeClaimTemplates:
- metadata:
name: data
spec:
{{- toYaml .Values.nacos.persistence.data | nindent 8 }}
{{- end }}
{{- end }}
34 changes: 33 additions & 1 deletion helm/core/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -651,7 +651,7 @@ apiserver:
imagePullPolicy: IfNotPresent
securePort: 8443
storage: nacos
serverAddr: http://127.0.0.1:8848
serverAddr: http://higress-nacos-service.higress-system.svc.cluster.local:8848
Copy link
Collaborator

@johnlanni johnlanni Mar 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

serverAddr 建议改名为 storageAddr,否则容易误解。
此外能否不要让用户配置 http:// 这个协议头呢,在helpers里根据storage和storageAddr来生成最终的配置参数
@CH3CHO 一起看看

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我觉得可以的。之前也考虑过能不能让 nacos 直接支持这种配置方式。短时间可以在生成配置的时候处理一下。

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

只让用户配置higress-nacos-service.higress-system.svc.cluster.local:8848这个storageAddr参数吗,通过helper.tpl给处理一下http://

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

或者直接让用户配置 nacos://higress-nacos-service.higress-system.svc.cluster.local:8848?这样 storage 也不需要单独配置了。

username: ""
password: ""
namespaceID: ""
Expand All @@ -663,3 +663,35 @@ skywalking:
service:
address: ~
port: 11800

# nacos config settings
nacos:
name: "higress-nacos"
enabled: false
hub: nacos/nacos-server
tag: latest
pullPolicy: IfNotPresent
replica: 1
podManagementPolicy: Parallel
nodeSelector: { }
affinity: { }
tolerations: [ ]
persistence:
enabled: false
data:
accessModes:
- ReadWriteOnce
storageClassName: manual
resources:
requests:
storage: 5Gi
service:
port: 8848
type: NodePort
nodePort: 30000
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

默认需要开NodePort吗?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不需要的,也可以改成clusterip,我给改一下

annotations: {}
preferHostMode: hostname
resources:
requests:
cpu: 500m
memory: 2Gi
5 changes: 1 addition & 4 deletions test/e2e/conformance/base/nacos.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,9 @@ metadata:
spec:
containers:
- name: nacos-standlone-rc3
image: swsk33/nacos-standalone:2.2.3
image: registry.cn-hangzhou.aliyuncs.com/hinsteny/nacos-standlone-rc3:1.0.0-RC3
CH3CHO marked this conversation as resolved.
Show resolved Hide resolved
ports:
- containerPort: 8848
- containerPort: 9848
---
apiVersion: v1
kind: Service
Expand All @@ -48,5 +47,3 @@ spec:
ports:
- name: foo # name is not required for single-port Services
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

现在已经不是单端口服务了,name 还是好好取一个吧

port: 8848
- name: bar
port: 9848
2 changes: 1 addition & 1 deletion test/e2e/conformance/utils/flags/flags.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,5 +30,5 @@ var (
TestArea = flag.String("test-area", "all", "Test area to run, like all to run setup/run/clean, setup to prepare test environment, run to run test cases, clean to clean test environment")
EnableApiServer = flag.Bool("enableApiServer", false, "Determine if enable api server")
// TODO when apiserver support other mode, we can use string field like: nacos://username:password@ip:port/namespace
Storage = flag.String("storage", "http://127.0.0.1:8848", "Define storage")
ApiServerStorage = flag.String("storage", "http://127.0.0.1:8848", "Define storage")
)
6 changes: 3 additions & 3 deletions test/e2e/conformance/utils/kubernetes/apply.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ import (
"k8s.io/apimachinery/pkg/types"
"k8s.io/apimachinery/pkg/util/yaml"
"sigs.k8s.io/controller-runtime/pkg/client"
yamlFormK8s "sigs.k8s.io/yaml"
yamlFromK8s "sigs.k8s.io/yaml"

"github.com/alibaba/higress/test/e2e/conformance/utils/config"
)
Expand Down Expand Up @@ -316,7 +316,7 @@ func (a Applier) MustApplyConfigmapDataWithYaml(t *testing.T, cc cc.Storage, c c
err := c.Get(ctx, client.ObjectKey{Namespace: namespace, Name: name}, cm)
require.NoError(t, err)

y, err := yamlFormK8s.Marshal(val)
y, err := yamlFromK8s.Marshal(val)
require.NoError(t, err)
data := string(y)

Expand All @@ -328,7 +328,7 @@ func (a Applier) MustApplyConfigmapDataWithYaml(t *testing.T, cc cc.Storage, c c
t.Logf("🏗 Updating %s %s", name, namespace)

if enableApiServer {
marshal, err := yamlFormK8s.Marshal(cm)
marshal, err := yamlFromK8s.Marshal(cm)
require.NoError(t, err)
err = cc.PublishConfig("configmap", cm.GetName(), cm.GetNamespace(), string(marshal))
require.NoError(t, err)
Expand Down
Loading
Loading