Skip to content

Add how to debug Memgraph under k8s #1272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
130 changes: 130 additions & 0 deletions pages/database-management/debugging.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -250,3 +250,133 @@ hotspot perf.data
you should be able to see a similar flamegraph like in the picture below.

![](/pages/database-management/debugging/perf.png)

## Debugging Memgraph under Kubernetes (k8s)

### General Commands

Managing [nodes](https://kubernetes.io/docs/concepts/architecture/nodes/):
```
kubectl get nodes --show-labels # Show all nodes and their labels.
kubectl top nodes # Get the current memory usage.
```

Managing [pods](https://kubernetes.io/docs/concepts/workloads/pods/):
```
kubectl get pods --show-labels # Show all pods and their labels.
kubectl get pods -o wide # Inspect how pods get scheduled.
kubectl describe pod <pod-name> # Inspect pod config (args, envs, ...).
kubectl exec -it <pod-name> -- /bin/bash # Login to a runnning pod.
kubectl logs <pod-name> # Get logs for a running pod.
kubectl logs memgraph-data-0-0 | tail -n 100 # Filter last logs from a running pod.
kubectl logs --previous <pod-name> # Get logs from a crashed pod.
kubectl cp <pod-name>:<pod-path> . # Copy logs from a running pod.
```

[Events](https://kubernetes.io/docs/reference/kubernetes-api/cluster-resources/event-v1/):
```
kubectl get events --all-namespaces --sort-by='.metadata.creationTimestamp' # List all events by creation time.
kubectl get events --namespace <namespace-name> # List all events in the given namespace.
```

[StatefulSets](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/):
```
kubectl get statefulsets # Show all StatefulSets.
kubectl get pvc -l app=<statefulset-name> # Get the PersistentVolumeClaims for the StatefulSet.
```

### Debugging Running Pods

### Creating the Debugging Memgraph Pod

The main issue with debugging under k8s (running gdb) is that the container
needs to be in the privileged mode. To run any given container in the
privileged mode, the k8s cluster itself needs to have an appropriate
configuration.

Below is an example on how to start the privileged `kind` cluster. First create
a new config `debug-cluster.yaml` file:
```
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
image: kindest/node:v1.31.0
extraPortMappings:
- containerPort: 80
hostPort: 8080
protocol: TCP
kubeadmConfigPatches:
- |
kind: ClusterConfiguration
kubeletConfiguration:
extraArgs:
allow-privileged: "true"
# To inspect the cluster run `kubectl get pods -n kube-system`.
# If some of the pods is in the CrashLoopBackOff status, try runnig `kubectl
# logs <pod-name> -n kube-system` to get the error message.
```
To start the cluster, you have to execute:
```
kind create cluster --name <cluster-name> --config debug-cluster.yaml
```

Once cluster is up and running, create a new `debug-pod.yaml` file with the
following content:
```
apiVersion: v1
kind: Pod
metadata:
name: debug-pod
spec:
containers:
- name: my-container
image: memgraph/memgraph:3.2.0-relwithdebinfo # Use the latest, but make sure it's the relwithdebinfo one!
securityContext:
runAsUser: 0 # Runs the container as root.
privileged: true
capabilities:
add: ["SYS_PTRACE"]
allowPrivilegeEscalation: true
command: ["sleep"]
args: ["infinity"]
stdin: true
tty: true
```

To get the pod up and running and open a shell inside it run:
```
kubectl apply -f debug-pod.yaml
kubectl exec -it debug-pod -- bash
```

Once you are in the pod execute:
```
apt-get update && apt-get install -y gdb
su memgraph
gdb --args ./memgraph <memgraph-flags>
run
```

Once you have memgraph up and running under `gdb`, run your workload (insert
data, write or queries…). When you manage to recreate the issue, use the [gdb
commands](/database-management/debugging#list-of-useful-commands-when-in-gdb)
to pin point the exact issue.

To delete the debug pod run:
```
kubectl delete pod debug-pod
```

k8s official documentation on how to [debug running
pods](https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/)
is quite detailed.

### Specific Cloud Provider Instructions

* [AWS](https://github.com/memgraph/helm-charts/tree/main/charts/memgraph-high-availability/aws)
* [Azure](https://github.com/memgraph/helm-charts/blob/main/charts/memgraph-high-availability/aks)

The [k8s quick
reference](https://kubernetes.io/docs/reference/kubectl/quick-reference/) is an
amazing set of commands!