Skip to content

Commit 85a3659

Browse files
committed
docs: document ingest storage jsonnet setup
Signed-off-by: Vladimir Varankin <[email protected]>
1 parent 45d4f5b commit 85a3659

File tree

1 file changed

+73
-0
lines changed

1 file changed

+73
-0
lines changed
Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
---
2+
aliases:
3+
- ../../operators-guide/deploy-grafana-mimir/jsonnet/configure-ingest-storage/
4+
description: Learn how to configure Grafana Mimir ingest storage architecture when using Jsonnet.
5+
menuTitle: Configure ingest storage architecture
6+
title: Configure ingest storage architecture
7+
weight: 20
8+
---
9+
10+
# Configure ingest storage architecture
11+
12+
The [ingest storage](https://grafana.com/docs/mimir/<MIMIR_VERSION>/get-started/about-grafana-mimir-architecture/) is the next generation architecture of Grafana Mimir. It decouples the Mimir read and write paths using Apache Kafka or a Kafka-compatible backend.
13+
14+
To enable ingest storage, set the following Jsonnet:
15+
16+
```jsonnet
17+
{
18+
_config+:: {
19+
ingest_storage_enabled: true,
20+
}
21+
}
22+
```
23+
24+
{{< admonition type="note" >}}
25+
The ingest storage architecture requires a production-grade Apache Kafka cluster or Kafka-compatible backend.
26+
{{< /admonition >}}
27+
28+
{{< admonition type="warning" >}}
29+
The ingest storage architecture requires the ruler remote evaluation to be enabled for the Mimir instance.
30+
Refer to the [Configure the Grafana Mimir ruler with Jsonnet](https://grafana.com/docs/mimir/<MIMIR_VERSION>/set-up/jsonnet/configure-ruler/) documentation for details about the ruler's operational modes.
31+
{{< /admonition >}}
32+
33+
## Kafka connection options
34+
35+
The jsonnet configures some Kafka connection parameters by default:
36+
37+
- **Kafka topic**: `ingest`
38+
- **Auto-created partitions**: 1000 (when topic doesn't exist)
39+
- **Kafka addresses**: `kafka.<namespace>.svc.<cluster_domain>:9092`
40+
41+
To customize the Kafka connection configuration, override the default settings:
42+
43+
```jsonnet
44+
{
45+
_config+:: {
46+
ingest_storage_enabled: true,
47+
},
48+
49+
// Override producer and consumer addresses for external Kafka
50+
ingest_storage_kafka_producer_address:: 'kafka-broker-1:9092,kafka-broker-2:9092',
51+
ingest_storage_kafka_consumer_address:: 'kafka-broker-1:9092,kafka-broker-2:9092',
52+
53+
// Override Kafka client configuration
54+
ingest_storage_kafka_client_args+:: {
55+
'ingest-storage.kafka.topic': 'mimir-ingest',
56+
'ingest-storage.kafka.auto-create-topic-default-partitions': 500,
57+
},
58+
}
59+
```
60+
61+
### Kafka partition sizing
62+
63+
The configured topic must have at least as many partitions as the number of ingesters in one zone.
64+
65+
The [Configure the Grafana Mimir Kafka backend](https://grafana.com/docs/mimir/<MIMIR_VERSION>/configure/configure-kafka-backend/) documentation provides more details about Kafka configurations.
66+
67+
## Migration considerations
68+
69+
{{< admonition type="warning" >}}
70+
Migration of a live Grafana Mimir cluster from the classic architecture to the ingest storage is a complex multi-step process, that requires careful planning and execution.
71+
72+
For use-cases that allow some downtime, it's simpler to deploy a new Grafana Mimir cluster, that uses ingest storage, and migrate the data from the old cluster to the new one.
73+
{{< /admonition >}}

0 commit comments

Comments
 (0)