Skip to content

Commit f5b4a98

Browse files
marquezikanschad
marquezika
and
nschad
authored
Feat Query-Scheduler (grafana#268)
* docs: add query-scheduler Signed-off-by: Joao Marques <[email protected]> * improve apiVersion for pdb Signed-off-by: nschad <[email protected]> * fix changelog Signed-off-by: nschad <[email protected]> * add query-scheduler to test-deployment Signed-off-by: nschad <[email protected]> * fix query-scheduler to test-deployment Signed-off-by: nschad <[email protected]> Co-authored-by: nschad <[email protected]>
1 parent 6a47771 commit f5b4a98

12 files changed

+401
-0
lines changed

‎CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
* [ENHANCEMENT] Allow StoreGateway podManagementPolicy to be changed #332
66
* [BUGFIX] Correct a typo in enabling distribtuor HPA #334
7+
* [FEATURE] Add Query-Scheduler #268
78

89
## 1.4.0 / 2022-03-08
910

‎README.md

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -622,6 +622,50 @@ Kubernetes: `^1.19.0-0`
622622
| query_frontend.&ZeroWidthSpace;strategy.&ZeroWidthSpace;type | string | `"RollingUpdate"` | |
623623
| query_frontend.&ZeroWidthSpace;terminationGracePeriodSeconds | int | `180` | |
624624
| query_frontend.&ZeroWidthSpace;tolerations | list | `[]` | |
625+
| query_scheduler.&ZeroWidthSpace;affinity.&ZeroWidthSpace;podAntiAffinity.&ZeroWidthSpace;preferredDuringSchedulingIgnoredDuringExecution[0].&ZeroWidthSpace;podAffinityTerm.&ZeroWidthSpace;labelSelector.&ZeroWidthSpace;matchExpressions[0].&ZeroWidthSpace;key | string | `"app.kubernetes.io/component"` | |
626+
| query_scheduler.&ZeroWidthSpace;affinity.&ZeroWidthSpace;podAntiAffinity.&ZeroWidthSpace;preferredDuringSchedulingIgnoredDuringExecution[0].&ZeroWidthSpace;podAffinityTerm.&ZeroWidthSpace;labelSelector.&ZeroWidthSpace;matchExpressions[0].&ZeroWidthSpace;operator | string | `"In"` | |
627+
| query_scheduler.&ZeroWidthSpace;affinity.&ZeroWidthSpace;podAntiAffinity.&ZeroWidthSpace;preferredDuringSchedulingIgnoredDuringExecution[0].&ZeroWidthSpace;podAffinityTerm.&ZeroWidthSpace;labelSelector.&ZeroWidthSpace;matchExpressions[0].&ZeroWidthSpace;values[0] | string | `"query-scheduler"` | |
628+
| query_scheduler.&ZeroWidthSpace;affinity.&ZeroWidthSpace;podAntiAffinity.&ZeroWidthSpace;preferredDuringSchedulingIgnoredDuringExecution[0].&ZeroWidthSpace;podAffinityTerm.&ZeroWidthSpace;topologyKey | string | `"kubernetes.io/hostname"` | |
629+
| query_scheduler.&ZeroWidthSpace;affinity.&ZeroWidthSpace;podAntiAffinity.&ZeroWidthSpace;preferredDuringSchedulingIgnoredDuringExecution[0].&ZeroWidthSpace;weight | int | `100` | |
630+
| query_scheduler.&ZeroWidthSpace;annotations | object | `{}` | |
631+
| query_scheduler.&ZeroWidthSpace;containerSecurityContext.&ZeroWidthSpace;enabled | bool | `true` | |
632+
| query_scheduler.&ZeroWidthSpace;containerSecurityContext.&ZeroWidthSpace;readOnlyRootFilesystem | bool | `true` | |
633+
| query_scheduler.&ZeroWidthSpace;enabled | bool | `false` | If true, querier and query-frontend will connect to it (requires Cortex v1.6.0+) https://cortexmetrics.io/docs/operations/scaling-query-frontend/#query-scheduler |
634+
| query_scheduler.&ZeroWidthSpace;env | list | `[]` | |
635+
| query_scheduler.&ZeroWidthSpace;extraArgs | object | `{}` | Additional Cortex container arguments, e.g. log.level (debug, info, warn, error) |
636+
| query_scheduler.&ZeroWidthSpace;extraContainers | list | `[]` | |
637+
| query_scheduler.&ZeroWidthSpace;extraPorts | list | `[]` | |
638+
| query_scheduler.&ZeroWidthSpace;extraVolumeMounts | list | `[]` | |
639+
| query_scheduler.&ZeroWidthSpace;extraVolumes | list | `[]` | |
640+
| query_scheduler.&ZeroWidthSpace;initContainers | list | `[]` | |
641+
| query_scheduler.&ZeroWidthSpace;lifecycle | object | `{}` | |
642+
| query_scheduler.&ZeroWidthSpace;livenessProbe.&ZeroWidthSpace;httpGet.&ZeroWidthSpace;path | string | `"/ready"` | |
643+
| query_scheduler.&ZeroWidthSpace;livenessProbe.&ZeroWidthSpace;httpGet.&ZeroWidthSpace;port | string | `"http-metrics"` | |
644+
| query_scheduler.&ZeroWidthSpace;nodeSelector | object | `{}` | |
645+
| query_scheduler.&ZeroWidthSpace;persistentVolume.&ZeroWidthSpace;subPath | string | `nil` | |
646+
| query_scheduler.&ZeroWidthSpace;podAnnotations | object | `{"prometheus.io/port":"http-metrics","prometheus.io/scrape":"true"}` | Pod Annotations |
647+
| query_scheduler.&ZeroWidthSpace;podDisruptionBudget.&ZeroWidthSpace;maxUnavailable | int | `1` | |
648+
| query_scheduler.&ZeroWidthSpace;podLabels | object | `{}` | Pod Labels |
649+
| query_scheduler.&ZeroWidthSpace;readinessProbe.&ZeroWidthSpace;httpGet.&ZeroWidthSpace;path | string | `"/ready"` | |
650+
| query_scheduler.&ZeroWidthSpace;readinessProbe.&ZeroWidthSpace;httpGet.&ZeroWidthSpace;port | string | `"http-metrics"` | |
651+
| query_scheduler.&ZeroWidthSpace;replicas | int | `2` | |
652+
| query_scheduler.&ZeroWidthSpace;resources | object | `{}` | |
653+
| query_scheduler.&ZeroWidthSpace;securityContext | object | `{}` | |
654+
| query_scheduler.&ZeroWidthSpace;service.&ZeroWidthSpace;annotations | object | `{}` | |
655+
| query_scheduler.&ZeroWidthSpace;service.&ZeroWidthSpace;labels | object | `{}` | |
656+
| query_scheduler.&ZeroWidthSpace;serviceMonitor.&ZeroWidthSpace;additionalLabels | object | `{}` | |
657+
| query_scheduler.&ZeroWidthSpace;serviceMonitor.&ZeroWidthSpace;enabled | bool | `false` | |
658+
| query_scheduler.&ZeroWidthSpace;serviceMonitor.&ZeroWidthSpace;extraEndpointSpec | object | `{}` | Additional endpoint configuration https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#endpoint |
659+
| query_scheduler.&ZeroWidthSpace;serviceMonitor.&ZeroWidthSpace;metricRelabelings | list | `[]` | |
660+
| query_scheduler.&ZeroWidthSpace;serviceMonitor.&ZeroWidthSpace;relabelings | list | `[]` | |
661+
| query_scheduler.&ZeroWidthSpace;startupProbe.&ZeroWidthSpace;failureThreshold | int | `10` | |
662+
| query_scheduler.&ZeroWidthSpace;startupProbe.&ZeroWidthSpace;httpGet.&ZeroWidthSpace;path | string | `"/ready"` | |
663+
| query_scheduler.&ZeroWidthSpace;startupProbe.&ZeroWidthSpace;httpGet.&ZeroWidthSpace;port | string | `"http-metrics"` | |
664+
| query_scheduler.&ZeroWidthSpace;strategy.&ZeroWidthSpace;rollingUpdate.&ZeroWidthSpace;maxSurge | int | `0` | |
665+
| query_scheduler.&ZeroWidthSpace;strategy.&ZeroWidthSpace;rollingUpdate.&ZeroWidthSpace;maxUnavailable | int | `1` | |
666+
| query_scheduler.&ZeroWidthSpace;strategy.&ZeroWidthSpace;type | string | `"RollingUpdate"` | |
667+
| query_scheduler.&ZeroWidthSpace;terminationGracePeriodSeconds | int | `180` | |
668+
| query_scheduler.&ZeroWidthSpace;tolerations | list | `[]` | |
625669
| ruler.&ZeroWidthSpace;affinity | object | `{}` | |
626670
| ruler.&ZeroWidthSpace;annotations | object | `{}` | |
627671
| ruler.&ZeroWidthSpace;containerSecurityContext.&ZeroWidthSpace;enabled | bool | `true` | |

‎ci/test-deployment-values.yaml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,12 @@ config:
4848
join_members:
4949
- '{{ include "cortex.fullname" $ }}-memberlist'
5050

51+
frontend:
52+
scheduler_address: 'dns+{{ include "cortex.querySchedulerFullname" $ }}-headless:9095'
53+
54+
frontend_worker:
55+
scheduler_address: 'dns+{{ include "cortex.querySchedulerFullname" $ }}-headless:9095'
56+
5157
ingester:
5258
replicas: 1
5359
autoscaling:
@@ -98,6 +104,15 @@ query_frontend:
98104
extraVolumeMounts:
99105
- name: tmp-test
100106
mountPath: /tmp-test
107+
query_scheduler:
108+
replicas: 1
109+
enabled: true
110+
extraVolumes:
111+
- name: tmp-test
112+
emptyDir: {}
113+
extraVolumeMounts:
114+
- name: tmp-test
115+
mountPath: /tmp-test
101116
nginx:
102117
replicas: 1
103118
autoscaling:

‎templates/querier/querier-dep.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,11 @@ spec:
5252
args:
5353
- "-target=querier"
5454
- "-config.file=/etc/cortex/cortex.yaml"
55+
{{- if .Values.query_scheduler.enabled }}
56+
- "-querier.scheduler-address={{ template "cortex.fullname" . }}-query-scheduler-headless.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}:{{ .Values.config.server.grpc_listen_port }}"
57+
{{- else }}
5558
- "-querier.frontend-address={{ template "cortex.fullname" . }}-query-frontend-headless.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}:{{ .Values.config.server.grpc_listen_port }}"
59+
{{- end }}
5660
{{- include "cortex.memcached" . | nindent 12}}
5761
{{- range $key, $value := .Values.querier.extraArgs }}
5862
- "-{{ $key }}={{ $value }}"

‎templates/query-frontend/query-frontend-dep.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,9 @@ spec:
5151
- "-target=query-frontend"
5252
- "-config.file=/etc/cortex/cortex.yaml"
5353
{{- include "cortex.frontend-memcached" . | nindent 12 }}
54+
{{- if .Values.query_scheduler.enabled }}
55+
- "-frontend.scheduler-address={{ template "cortex.fullname" . }}-query-scheduler-headless.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}:{{ .Values.config.server.grpc_listen_port }}"
56+
{{- end }}
5457
{{- range $key, $value := .Values.query_frontend.extraArgs }}
5558
- "-{{ $key }}={{ $value }}"
5659
{{- end }}

‎templates/query-scheduler/_helpers-query-scheduler.tpl

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
2+
{{/*
3+
query-scheduler fullname
4+
*/}}
5+
{{- define "cortex.querySchedulerFullname" -}}
6+
{{ include "cortex.fullname" . }}-query-scheduler
7+
{{- end }}
8+
9+
{{/*
10+
query-scheduler common labels
11+
*/}}
12+
{{- define "cortex.querySchedulerLabels" -}}
13+
{{ include "cortex.labels" . }}
14+
app.kubernetes.io/component: query-scheduler
15+
{{- end }}
16+
17+
{{/*
18+
query-scheduler selector labels
19+
*/}}
20+
{{- define "cortex.querySchedulerSelectorLabels" -}}
21+
{{ include "cortex.selectorLabels" . }}
22+
app.kubernetes.io/component: query-scheduler
23+
{{- end }}

‎templates/query-scheduler/query-scheduler-dep.yaml

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
{{- if .Values.query_scheduler.enabled }}
2+
apiVersion: apps/v1
3+
kind: Deployment
4+
metadata:
5+
name: {{ include "cortex.querySchedulerFullname" . }}
6+
namespace: {{ .Release.Namespace }}
7+
labels:
8+
{{- include "cortex.querySchedulerLabels" . | nindent 4 }}
9+
annotations:
10+
{{- toYaml .Values.query_scheduler.annotations | nindent 4 }}
11+
spec:
12+
replicas: {{ .Values.query_scheduler.replicas }}
13+
selector:
14+
matchLabels:
15+
{{- include "cortex.querySchedulerSelectorLabels" . | nindent 6 }}
16+
strategy:
17+
{{- toYaml .Values.query_scheduler.strategy | nindent 4 }}
18+
template:
19+
metadata:
20+
labels:
21+
{{- include "cortex.querySchedulerLabels" . | nindent 8 }}
22+
{{- with .Values.query_scheduler.podLabels }}
23+
{{- toYaml . | nindent 8 }}
24+
{{- end }}
25+
annotations:
26+
{{- if .Values.useExternalConfig }}
27+
checksum/config: {{ .Values.externalConfigVersion }}
28+
{{- else }}
29+
checksum/config: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}
30+
{{- end }}
31+
{{- with .Values.query_scheduler.podAnnotations }}
32+
{{- toYaml . | nindent 8 }}
33+
{{- end }}
34+
spec:
35+
serviceAccountName: {{ template "cortex.serviceAccountName" . }}
36+
{{- if .Values.query_scheduler.priorityClassName }}
37+
priorityClassName: {{ .Values.query_scheduler.priorityClassName }}
38+
{{- end }}
39+
{{- if .Values.query_scheduler.securityContext.enabled }}
40+
securityContext: {{- omit .Values.query_scheduler.securityContext "enabled" | toYaml | nindent 8 }}
41+
{{- end }}
42+
initContainers:
43+
{{- toYaml .Values.query_scheduler.initContainers | nindent 8 }}
44+
{{- if .Values.image.pullSecrets }}
45+
imagePullSecrets:
46+
{{- range .Values.image.pullSecrets }}
47+
- name: {{ . }}
48+
{{- end }}
49+
{{- end }}
50+
containers:
51+
- name: query-scheduler
52+
image: "{{ .Values.image.repository }}:{{ default .Chart.AppVersion .Values.image.tag }}"
53+
imagePullPolicy: {{ .Values.image.pullPolicy }}
54+
args:
55+
- "-target=query-scheduler"
56+
- "-config.file=/etc/cortex/cortex.yaml"
57+
{{- range $key, $value := .Values.query_scheduler.extraArgs }}
58+
- "-{{ $key }}={{ $value }}"
59+
{{- end }}
60+
volumeMounts:
61+
{{- if .Values.query_scheduler.extraVolumeMounts }}
62+
{{- toYaml .Values.query_scheduler.extraVolumeMounts | nindent 12 }}
63+
{{- end }}
64+
- name: config
65+
mountPath: /etc/cortex
66+
- name: runtime-config
67+
mountPath: /etc/cortex-runtime-config
68+
ports:
69+
- name: http-metrics
70+
containerPort: {{ .Values.config.server.http_listen_port }}
71+
protocol: TCP
72+
- name: grpc
73+
containerPort: {{ .Values.config.server.grpc_listen_port }}
74+
protocol: TCP
75+
startupProbe:
76+
{{- toYaml .Values.query_scheduler.startupProbe | nindent 12 }}
77+
livenessProbe:
78+
{{- toYaml .Values.query_scheduler.livenessProbe | nindent 12 }}
79+
readinessProbe:
80+
{{- toYaml .Values.query_scheduler.readinessProbe | nindent 12 }}
81+
resources:
82+
{{- toYaml .Values.query_scheduler.resources | nindent 12 }}
83+
{{- if .Values.query_scheduler.containerSecurityContext.enabled }}
84+
securityContext: {{- omit .Values.query_scheduler.containerSecurityContext "enabled" | toYaml | nindent 12 }}
85+
{{- end }}
86+
{{- if .Values.query_scheduler.env }}
87+
env:
88+
{{- toYaml .Values.query_scheduler.env | nindent 12 }}
89+
{{- end }}
90+
{{- with .Values.query_scheduler.lifecycle }}
91+
lifecycle:
92+
{{- toYaml . | nindent 12 }}
93+
{{- end }}
94+
{{- if .Values.query_scheduler.extraContainers }}
95+
{{- toYaml .Values.query_scheduler.extraContainers | nindent 8 }}
96+
{{- end }}
97+
nodeSelector:
98+
{{- toYaml .Values.query_scheduler.nodeSelector | nindent 8 }}
99+
affinity:
100+
{{- toYaml .Values.query_scheduler.affinity | nindent 8 }}
101+
tolerations:
102+
{{- toYaml .Values.query_scheduler.tolerations | nindent 8 }}
103+
terminationGracePeriodSeconds: {{ .Values.query_scheduler.terminationGracePeriodSeconds }}
104+
volumes:
105+
- name: config
106+
secret:
107+
{{- if .Values.useExternalConfig }}
108+
secretName: {{ .Values.externalConfigSecretName }}
109+
{{- else }}
110+
secretName: {{ template "cortex.fullname" . }}
111+
{{- end }}
112+
- name: runtime-config
113+
configMap:
114+
name: {{ template "cortex.fullname" . }}-runtime-config
115+
{{- if .Values.query_scheduler.extraVolumes }}
116+
{{- toYaml .Values.query_scheduler.extraVolumes | nindent 8 }}
117+
{{- end }}
118+
{{- end }}

‎templates/query-scheduler/query-scheduler-poddisruptionbudget.yaml

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{{- if and (.Values.query_scheduler.enabled) (gt (int .Values.query_scheduler.replicas) 1) (.Values.query_scheduler.podDisruptionBudget) }}
2+
apiVersion: {{ include "cortex.pdbVersion" . }}
3+
kind: PodDisruptionBudget
4+
metadata:
5+
name: {{ include "cortex.querySchedulerFullname" . }}
6+
namespace: {{ .Release.Namespace }}
7+
labels:
8+
{{- include "cortex.querySchedulerLabels" . | nindent 4 }}
9+
spec:
10+
selector:
11+
matchLabels:
12+
{{- include "cortex.querySchedulerSelectorLabels" . | nindent 6 }}
13+
{{- toYaml .Values.query_scheduler.podDisruptionBudget | nindent 2 }}
14+
{{- end }}

‎templates/query-scheduler/query-scheduler-servicemonitor.yaml

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
{{- if and .Values.query_scheduler.enabled .Values.query_scheduler.serviceMonitor.enabled }}
2+
apiVersion: monitoring.coreos.com/v1
3+
kind: ServiceMonitor
4+
metadata:
5+
name: {{ include "cortex.querySchedulerFullname" . }}
6+
namespace: {{ .Release.Namespace }}
7+
labels:
8+
{{- include "cortex.querySchedulerLabels" . | nindent 4 }}
9+
{{- if .Values.query_scheduler.serviceMonitor.additionalLabels }}
10+
{{ toYaml .Values.query_scheduler.serviceMonitor.additionalLabels | indent 4 }}
11+
{{- end }}
12+
{{- if .Values.query_scheduler.serviceMonitor.annotations }}
13+
annotations:
14+
{{ toYaml .Values.query_scheduler.serviceMonitor.annotations | indent 4 }}
15+
{{- end }}
16+
spec:
17+
selector:
18+
matchLabels:
19+
{{- include "cortex.querySchedulerSelectorLabels" . | nindent 6 }}
20+
namespaceSelector:
21+
matchNames:
22+
- {{ .Release.Namespace | quote }}
23+
endpoints:
24+
- port: http-metrics
25+
{{- if .Values.query_scheduler.serviceMonitor.interval }}
26+
interval: {{ .Values.query_scheduler.serviceMonitor.interval }}
27+
{{- end }}
28+
{{- if .Values.query_scheduler.serviceMonitor.scrapeTimeout }}
29+
scrapeTimeout: {{ .Values.query_scheduler.serviceMonitor.scrapeTimeout }}
30+
{{- end }}
31+
{{- if .Values.query_scheduler.serviceMonitor.relabelings }}
32+
relabelings:
33+
{{- toYaml .Values.query_scheduler.serviceMonitor.relabelings | nindent 4 }}
34+
{{- end }}
35+
{{- if .Values.query_scheduler.serviceMonitor.metricRelabelings }}
36+
metricRelabelings:
37+
{{- toYaml .Values.query_scheduler.serviceMonitor.metricRelabelings | nindent 4 }}
38+
{{- end }}
39+
{{- with .Values.query_scheduler.serviceMonitor.extraEndpointSpec }}
40+
{{- toYaml . | nindent 4 }}
41+
{{- end }}
42+
{{- end }}

‎templates/query-scheduler/query-scheduler-svc-headless.yaml

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
{{- if .Values.query_scheduler.enabled }}
2+
apiVersion: v1
3+
kind: Service
4+
metadata:
5+
name: {{ include "cortex.querySchedulerFullname" . }}-headless
6+
namespace: {{ .Release.Namespace }}
7+
labels:
8+
{{- include "cortex.querySchedulerLabels" . | nindent 4 }}
9+
{{- with .Values.query_scheduler.service.labels }}
10+
{{- toYaml . | nindent 4 }}
11+
{{- end }}
12+
annotations:
13+
{{- toYaml .Values.query_scheduler.service.annotations | nindent 4 }}
14+
spec:
15+
type: ClusterIP
16+
clusterIP: None
17+
publishNotReadyAddresses: true
18+
ports:
19+
- port: {{ .Values.config.server.grpc_listen_port }}
20+
protocol: TCP
21+
name: grpc
22+
targetPort: grpc
23+
selector:
24+
{{- include "cortex.querySchedulerSelectorLabels" . | nindent 4 }}
25+
{{- end }}

‎templates/query-scheduler/query-scheduler-svc.yaml

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
{{- if .Values.query_scheduler.enabled }}
2+
apiVersion: v1
3+
kind: Service
4+
metadata:
5+
name: {{ include "cortex.querySchedulerFullname" . }}
6+
namespace: {{ .Release.Namespace }}
7+
labels:
8+
{{- include "cortex.querySchedulerLabels" . | nindent 4 }}
9+
{{- with .Values.query_scheduler.service.labels }}
10+
{{- toYaml . | nindent 4 }}
11+
{{- end }}
12+
annotations:
13+
{{- toYaml .Values.query_scheduler.service.annotations | nindent 4 }}
14+
spec:
15+
type: ClusterIP
16+
ports:
17+
- port: {{ .Values.config.server.http_listen_port }}
18+
protocol: TCP
19+
name: http-metrics
20+
targetPort: http-metrics
21+
selector:
22+
{{- include "cortex.querySchedulerSelectorLabels" . | nindent 4 }}
23+
{{- end }}

0 commit comments

Comments
 (0)