Skip to content

Commit 5510db0

Browse files
dannyzakendannyzaken
committed
added doc noobaa-cnpg-integration.md
Signed-off-by: Danny Zaken <[email protected]> doc changes
1 parent 22f1cc3 commit 5510db0

File tree

1 file changed

+250
-0
lines changed

1 file changed

+250
-0
lines changed

‎doc/noobaa-cnpg-integration.md‎

Lines changed: 250 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,250 @@
1+
# NooBaa CloudNative-PG Integration
2+
3+
[NooBaa Operator](../README.md) /
4+
5+
## Introduction
6+
7+
NooBaa operator integrates with [CloudNative-PG (CNPG)](https://cloudnative-pg.io/) to provide high availability PostgreSQL database deployments for NooBaa metadata storage. This integration eliminates the single point of failure that existed with the previous standalone PostgreSQL StatefulSet approach.
8+
9+
CloudNative-PG is an open-source Kubernetes operator designed to manage PostgreSQL workloads on any supported Kubernetes cluster. It provides automated failover, self-healing capabilities, and declarative management of PostgreSQL configurations, making it an ideal solution for NooBaa's database requirements.
10+
11+
### Key Benefits
12+
13+
- **High Availability**: Automated failover and self-healing capabilities
14+
- **Zero Data Loss**: Optional synchronous replication support
15+
- **Declarative Management**: Kubernetes-native database configuration
16+
- **Scalability**: Support for read replicas and cluster scaling
17+
- **Monitoring**: Prometheus-compatible metrics export
18+
19+
## High-level CloudNative-PG Concepts
20+
21+
### Core Components
22+
23+
CloudNative-PG introduces several custom Kubernetes resources:
24+
25+
- **[Cluster](https://cloudnative-pg.io/documentation/1.27/cloudnative-pg.v1/#postgresql-cnpg-io-v1-Cluster)**: Represents a PostgreSQL cluster with one primary instance and optional replicas
26+
- **[ImageCatalog](https://cloudnative-pg.io/documentation/1.27/cloudnative-pg.v1/#postgresql-cnpg-io-v1-ImageCatalog)**: Maps PostgreSQL major versions to specific container images
27+
- **Services**: Automatically created read-write and read-only services for database access
28+
- **Secrets**: Automatically managed database credentials and certificates
29+
30+
### Key Features
31+
32+
- **Asynchronous Replication**: Default replication mode for better performance
33+
- **Synchronous Replication**: Optional zero data loss guarantee (RPO=0)
34+
- **Automated Failover**: Promotes the most aligned replica when primary fails
35+
- **Self-healing**: Automatic recreation of failed replicas
36+
- **Planned Switchover**: Manual promotion of selected replicas
37+
- **Monitoring**: Built-in Prometheus metrics export
38+
- **TLS Support**: Secure connections with client certificate authentication
39+
40+
### Architecture
41+
42+
CloudNative-PG manages PostgreSQL clusters directly without using StatefulSets, instead directly managing Persistent Volume Claims (PVCs). This provides more flexibility and better integration with Kubernetes storage systems.
43+
44+
## High-level Integration Description
45+
46+
### CLI Integration
47+
48+
The NooBaa CLI provides comprehensive CloudNative-PG management commands:
49+
50+
```bash
51+
noobaa cnpg --help
52+
```
53+
54+
Available commands:
55+
- `install` - Install CloudNative-PG operator
56+
- `uninstall` - Uninstall CloudNative-PG operator
57+
- `yaml` - Show bundled CloudNative-PG YAML manifests
58+
- `status` - Check CloudNative-PG operator deployment status
59+
60+
The CLI automatically installs CloudNative-PG when creating a new NooBaa system unless the `--use-standalone-db` flag is specified. When installing via OLM, the CloudNative-PG operator deployment and resources are included in the ClusterServiceVersion (CSV).
61+
62+
### Operator Reconciliation Loop
63+
64+
The NooBaa operator integrates CloudNative-PG reconciliation into its main reconciliation phases:
65+
66+
#### Phase 2: Creating
67+
During the "Creating" phase, the operator:
68+
1. **Creates ImageCatalog**: Maps PostgreSQL major versions to container images
69+
2. **Creates Cluster**: Establishes the PostgreSQL cluster with appropriate configuration
70+
3. **Handles Database Import**: For upgrades, imports data from existing standalone databases
71+
72+
#### Database Reconciliation Logic
73+
The operator handles three main scenarios:
74+
75+
1. **Fresh Install**: Creates a new empty CNPG cluster
76+
2. **Upgrade from Standalone DB**: Imports data from existing StatefulSet-based PostgreSQL
77+
3. **Existing CNPG Cluster**: Updates configuration and manages cluster lifecycle
78+
79+
### OLM and CSV Generation
80+
81+
CloudNative-PG resources are integrated into the Operator Lifecycle Manager (OLM) deployment:
82+
83+
#### CSV Generation
84+
- CNPG resources are included in the ClusterServiceVersion (CSV) when `--include-cnpg` flag is used
85+
- CNPG CRDs are bundled with NooBaa operator CRDs
86+
- Security contexts are modified for OLM compliance
87+
- Related images include CNPG operator image
88+
89+
#### OLM Integration
90+
- CNPG operator is deployed alongside NooBaa operator
91+
- RBAC permissions are properly configured for namespace-scoped operations
92+
- Webhook configurations are included for admission control
93+
94+
### API Group Customization
95+
96+
To avoid conflicts with upstream CloudNative-PG installations, NooBaa uses a custom API group:
97+
98+
- **Default**: `postgresql.cnpg.noobaa.io/v1`
99+
- **Upstream**: `postgresql.cnpg.io/v1` (when `USE_CNPG_API_GROUP=true`)
100+
101+
This customization is handled through environment variables and YAML manifest modifications.
102+
103+
## Low-level Code Description
104+
105+
### Core Integration Files
106+
107+
#### `pkg/cnpg/cnpg.go`
108+
Main CloudNative-PG integration package containing:
109+
110+
- **Resource Management**: Loading and modifying CNPG operator manifests
111+
- **CLI Commands**: Implementation of `noobaa cnpg` commands
112+
- **API Group Handling**: Custom API group configuration
113+
- **RBAC Configuration**: Namespace-scoped permissions setup
114+
115+
Key functions:
116+
- `LoadCnpgResources()`: Loads CNPG operator manifests from embedded files
117+
- `modifyResources()`: Customizes resources for NooBaa deployment
118+
- `getCnpgAPIGroup()`: Returns appropriate API group based on configuration
119+
120+
#### `pkg/system/db_reconciler.go`
121+
Database reconciliation logic handling:
122+
123+
- **Cluster Reconciliation**: Main reconciliation function for CNPG clusters
124+
- **Image Catalog Management**: Handles PostgreSQL version and image mapping
125+
- **Import Logic**: Manages data import from standalone databases
126+
- **Status Updates**: Tracks cluster readiness and status
127+
128+
Key functions:
129+
- `ReconcileCNPGCluster()`: Main reconciliation entry point
130+
- `reconcileCNPGImageCatalog()`: Manages PostgreSQL image versions
131+
- `reconcileDBCluster()`: Handles cluster creation and updates
132+
- `reconcileClusterCreateOrImport()`: Manages database import process
133+
134+
#### `pkg/apis/noobaa/v1alpha1/noobaa_types.go`
135+
API definitions for database configuration:
136+
137+
- **NooBaaDBSpec**: Database specification in NooBaa CR
138+
- **NooBaaDBStatus**: Database status reporting
139+
140+
Key fields:
141+
- `DBSpec`: Optional database specification for managed PostgreSQL
142+
- `DBImage`: PostgreSQL container image override
143+
- `PostgresMajorVersion`: PostgreSQL major version specification
144+
- `Instances`: Number of database instances
145+
- `DBResources`: Resource requirements for database pods
146+
- `DBMinVolumeSize`: Minimum volume size for database storage
147+
- `DBStorageClass`: Storage class for database volumes
148+
- `DBConf`: PostgreSQL configuration overrides
149+
150+
#### `pkg/apis/noobaa/v1alpha1/cnpg_types.go`
151+
CloudNative-PG type registrations:
152+
153+
- Registers CNPG types with custom API group
154+
- Handles both upstream and custom API group variants
155+
- Integrates with Kubernetes scheme for type recognition
156+
157+
### Database Configuration
158+
159+
#### Default Configuration
160+
The operator sets optimized PostgreSQL parameters:
161+
162+
```go
163+
overrideParameters := map[string]string{
164+
"huge_pages": "off",
165+
"max_connections": "600",
166+
"effective_cache_size": "3GB",
167+
"maintenance_work_mem": "256MB",
168+
"checkpoint_completion_target": "0.9",
169+
"wal_buffers": "16MB",
170+
"default_statistics_target": "100",
171+
"random_page_cost": "1.1",
172+
"effective_io_concurrency": "300",
173+
"work_mem": "1747kB",
174+
"min_wal_size": "2GB",
175+
"max_wal_size": "8GB",
176+
"pg_stat_statements.track": "all",
177+
}
178+
```
179+
180+
#### Resource Management
181+
- **CPU and Memory**: Configurable through `DBSpec.DBResources`
182+
- **Storage**: Configurable through `DBSpec.DBMinVolumeSize` and `DBSpec.DBStorageClass`
183+
- **Instances**: Default 2 instances (1 primary + 1 replica)
184+
185+
### Import Process
186+
187+
For upgrades from standalone PostgreSQL deployments:
188+
189+
1. **Pod Shutdown**: Stops NooBaa core and endpoint pods
190+
2. **Volume Analysis**: Checks existing database volume usage
191+
3. **Size Calculation**: Doubles volume size if usage > 33%
192+
4. **Import Configuration**: Sets up external cluster connection
193+
5. **Data Migration**: Uses CNPG's microservice import type
194+
6. **Cleanup**: Scales down old StatefulSet to 0 replicas
195+
196+
### Monitoring and Status
197+
198+
#### Status Tracking
199+
The operator tracks database status through `NooBaaDBStatus`:
200+
201+
- `DBClusterStatus`: Current cluster state (None, Creating, Importing, Ready, Updating, Failed)
202+
- `DBCurrentImage`: Current PostgreSQL image
203+
- `CurrentPgMajorVersion`: PostgreSQL major version
204+
- `ActualVolumeSize`: Actual volume size
205+
206+
#### Monitoring Configuration
207+
- **Default**: Prometheus monitoring enabled
208+
- **Disable**: Set annotation `noobaa.io/disable_db_default_monitoring=true`
209+
- **Metrics**: Exported on port 9187
210+
211+
212+
### Security Considerations
213+
214+
#### RBAC Configuration
215+
- **Namespace-scoped**: Most permissions limited to NooBaa namespace
216+
- **Cluster-scoped**: Only nodes and clusterimagecatalogs require cluster permissions
217+
- **Webhook Permissions**: Separate cluster role for admission webhooks
218+
219+
#### Network Security
220+
- **Internal Communication**: Database pods communicate within cluster network
221+
- **Service Discovery**: Uses Kubernetes services for database access
222+
- **TLS Support**: Optional TLS encryption for database connections
223+
224+
### Troubleshooting
225+
226+
#### Common Issues
227+
1. **Import Failures**: Check volume space and permissions
228+
2. **Cluster Not Ready**: Verify CNPG operator deployment
229+
3. **Resource Constraints**: Ensure adequate CPU/memory allocation
230+
4. **Storage Issues**: Verify storage class and volume provisioning
231+
232+
#### Debugging Commands
233+
```bash
234+
# Check CNPG operator status
235+
noobaa cnpg status
236+
237+
# View cluster status
238+
kubectl get clusters.postgresql.cnpg.noobaa.io
239+
240+
# Check cluster events
241+
kubectl describe cluster <cluster-name>
242+
243+
# View CNPG operator logs
244+
kubectl logs -n <namespace> deployment/cnpg-controller-manager
245+
```
246+
247+
## References
248+
249+
- [CloudNative-PG Documentation v1.27](https://cloudnative-pg.io/documentation/1.27/)
250+
- [PostgreSQL High Availability](https://www.postgresql.org/docs/current/high-availability.html)

0 commit comments

Comments
 (0)