Skip to content

Commit bc13331

Browse files
cmilesbcmilesb
authored
Merge pull request #1707 from redis/rdi-aurora-instructions
Cherry-pick RDI with RDS/Aurora instructions from preview-rc-rdi to main
2 parents 95ba474 + 57d0e20 commit bc13331

File tree

6 files changed

+345
-59
lines changed

6 files changed

+345
-59
lines changed

‎content/embeds/aur-rds-mysql-create-debezium-user.md

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
## Create Debezium user
2+
3+
The Debezium connector needs a user account to connect to MySQL. This
4+
user must have appropriate permissions on all databases where you want Debezium
5+
to capture changes.
6+
7+
1. Connect to your database as an admin user and create a new user for the connector:
8+
9+
```sql
10+
CREATE USER '<username>'@'%' IDENTIFIED BY '<password>';
11+
```
12+
13+
Replace `<username>` and `<password>` with a username and password for the new user.
14+
15+
The `%` means that the user can connect from any client. If you want to restrict the user to connect only from the RDI host, replace `%` with the IP address of the RDI host.
16+
17+
1. Grant the user the necessary permissions:
18+
19+
```sql
20+
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT, LOCK TABLES ON *.* TO '<username>'@'%';
21+
```
22+
23+
Replace `<username>` with the username of the Debezium user.
24+
25+
You can also grant SELECT permissions for specific tables only. The other permissions are global and cannot be restricted to specific tables.
26+
27+
```sql
28+
GRANT RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT, LOCK TABLES ON *.* TO '<username>'@'%';
29+
GRANT SELECT ON <database>.<table> TO '<username>'@'%';
30+
```
31+
32+
1. Finalize the user's permissions:
33+
34+
```sql
35+
FLUSH PRIVILEGES;
36+
```

‎content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aur-pgsql.md

Lines changed: 0 additions & 59 deletions
This file was deleted.

‎content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/_index.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
---
2+
Title: Prepare AWS RDS and Aurora databases for RDI
3+
aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/
4+
alwaysopen: false
5+
categories:
6+
- docs
7+
- integrate
8+
- rs
9+
- rdi
10+
description: Learn how to prepare AWS RDS and Aurora databases for RDI.
11+
group: di
12+
linkTitle: Prepare AWS RDS and Aurora
13+
summary: Prepare AWS Aurora and AWS RDS databases to work with Redis Data Integration.
14+
hideListLinks: false
15+
type: integration
16+
weight: 5
17+
---
18+
19+
You can use RDI with databases on [AWS Relational Database Service (RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) and [AWS Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraOverview.html).
20+
21+
The pages in this section give detailed instructions to get your source
22+
database ready for Debezium to use:

‎content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
---
2+
Title: Prepare AWS Aurora MySQL/AWS RDS MySQL for RDI
3+
aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql/
4+
alwaysopen: false
5+
categories:
6+
- docs
7+
- integrate
8+
- rs
9+
- rdi
10+
description: Enable CDC features in your source databases
11+
group: di
12+
hideListLinks: false
13+
linkTitle: Prepare AWS Aurora/RDS MySQL
14+
summary: Prepare AWS Aurora MySQL and AWS RDS MySQL databases to work with Redis Data Integration.
15+
type: integration
16+
weight: 2
17+
---
18+
19+
Follow the steps in the sections below to prepare an [AWS Aurora MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.Aurora.html) or [AWS RDS MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.MySQL.html) database.
20+
database to work with RDI.
21+
22+
Select the steps for your database type.
23+
24+
{{< multitabs id="rds-aur-mysql"
25+
tab1="AWS Aurora MySQL"
26+
tab2="AWS RDS MySQL" >}}
27+
28+
## Add an Aurora reader node
29+
30+
RDI requires that your Aurora MySQL database has at least one replica or reader node.
31+
32+
To add a reader node to an existing database, select **Add reader** from the **Actions** menu of the database and [add a reader node](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-replicas-adding.html).
33+
34+
You can also create one during database creation by selecting **Create an Aurora Replica or Reader node in a different AZ (recommended for scaled availability)** under **Availability & durability > Multi-AZ deployment**.
35+
36+
## Create and apply parameter group
37+
38+
RDI requires some changes to database parameters. On AWS Aurora, you change these parameters via a parameter group.
39+
40+
1. In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/),navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) with the following settings:
41+
42+
| Name | Value |
43+
| :-- | :-- |
44+
| **Parameter group name** | Enter a suitable parameter group name, like `rdi-mysql` |
45+
| **Description** | (Optional) Enter a description for the parameter group |
46+
| **Engine Type** | Choose **Aurora MySQL**. |
47+
| **Parameter group family** | Choose **aurora-mysql8.0**. |
48+
| **Type** | Select **DB Parameter Group**. |
49+
50+
Select **Create** to create the parameter group.
51+
52+
1. Navigate to **Parameter groups** in the console. Select the parameter group you have just created and then select **Edit**. Change the following parameters:
53+
54+
| Name | Value |
55+
| :-- | :-- |
56+
| `binlog_format` | `ROW` |
57+
| `binlog_row_image` | `FULL` |
58+
| `gtid_mode` | `ON` |
59+
| `enforce_gtid_consistency` | `ON` |
60+
61+
Select **Save Changes** to apply the changes to the parameter group.
62+
63+
1. Go back to your target database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created.
64+
65+
Select **Save changes** to apply the parameter group to the new database.
66+
67+
{{< embed-md "aur-rds-mysql-create-debezium-user.md" >}}
68+
69+
-tab-sep-
70+
71+
## Create and apply parameter group
72+
73+
RDI requires some changes to database parameters. On AWS RDS, you change these parameters via a parameter group.
74+
75+
1. In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/),navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) with the following settings:
76+
77+
| Name | Value |
78+
| :-- | :-- |
79+
| **Parameter group name** | Enter a suitable parameter group name, like `rdi-mysql` |
80+
| **Description** | (Optional) Enter a description for the parameter group |
81+
| **Engine Type** | Choose **MySQL Community**. |
82+
| **Parameter group family** | Choose **mysql8.0**. |
83+
84+
Select **Create** to create the parameter group.
85+
86+
1. Navigate to **Parameter groups** in the console. Select the parameter group you have just created and then select **Edit**. Change the following parameters:
87+
88+
| Name | Value |
89+
| :-- | :-- |
90+
| `binlog_format` | `ROW` |
91+
| `binlog_row_image` | `FULL` |
92+
93+
Select **Save Changes** to apply the changes to the parameter group.
94+
95+
1. Go back to your target database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created.
96+
97+
Select **Save changes** to apply the parameter group to the new database.
98+
99+
{{< embed-md "aur-rds-mysql-create-debezium-user.md" >}}
100+
101+
{{< /multitabs >}}

‎content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
---
2+
Title: Prepare AWS Aurora PostgreSQL/AWS RDS PostgreSQL for RDI
3+
aliases:
4+
- /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aur-pgsql/
5+
- /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql/
6+
- /integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aur-pgsql/
7+
alwaysopen: false
8+
categories:
9+
- docs
10+
- integrate
11+
- rs
12+
- rc
13+
- rdi
14+
description: Prepare AWS Aurora PostgreSQL databases to work with RDI
15+
group: di
16+
linkTitle: Prepare AWS Aurora PostgreSQL
17+
summary: Prepare AWS Aurora PostgreSQL databases to work with Redis Data Integration.
18+
type: integration
19+
weight: 1
20+
---
21+
22+
Follow the steps in the sections below to prepare an
23+
[AWS Aurora PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.AuroraPostgreSQL.html) or [AWS RDS PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.PostgreSQL.html)
24+
database to work with RDI.
25+
26+
## Create and apply parameter group
27+
28+
RDI requires some changes to database parameters. On AWS RDS and AWS Aurora, you change these parameters via a parameter group.
29+
30+
1. In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/), navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) with the following settings:
31+
32+
| Name | Value |
33+
| :-- | :-- |
34+
| **Parameter group name** | Enter a suitable parameter group name, like `rdi-aurora-pg` or `rdi-rds-pg` |
35+
| **Description** | (Optional) Enter a description for the parameter group |
36+
| **Engine Type** | Choose **Aurora PostgreSQL** for Aurora PostgreSQL or **PostgreSQL** for AWS RDS PostgreSQL. |
37+
| **Parameter group family** | Choose **aurora-postgresql15** for Aurora PostgreSQL or **postgresql13** for AWS RDS PostgreSQL. |
38+
39+
Select **Create** to create the parameter group.
40+
41+
1. Navigate to **Parameter groups** in the console. Select the group you have just created and then select **Edit**. Change the following parameters:
42+
43+
| Name | Value |
44+
| :-- | :-- |
45+
| `rds.logical_replication` | `1` |
46+
47+
Select **Save Changes** to apply the changes to the parameter group.
48+
49+
1. Go back to your database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created.
50+
51+
Select **Save changes** to apply the parameter group to your database.
52+
53+
## Create Debezium user
54+
55+
The Debezium connector needs a user account to connect to PostgreSQL. This
56+
user must have appropriate permissions on all databases where you want Debezium
57+
to capture changes.
58+
59+
1. Connect to PostgreSQL as the `postgres` user and create a new user for the connector:
60+
61+
```sql
62+
CREATE ROLE <username> WITH LOGIN PASSWORD '<password>' VALID UNTIL 'infinity';
63+
```
64+
65+
Replace `<username>` and `<password>` with a username and password for the new user.
66+
67+
1. Grant the user the necessary replication permissions:
68+
69+
```sql
70+
GRANT rds_replication TO <username>;
71+
```
72+
73+
Replace `<username>` with the username of the Debezium user.
74+
75+
1. Connect to your database as the `postgres` user and grant the new user access to one or more schemas in the database:
76+
77+
```sql
78+
GRANT SELECT ON ALL TABLES IN SCHEMA <schema> TO <username>;
79+
```
80+
81+
Replace `<username>` with the username of the Debezium user and `<schema>` with the schema name.
82+

0 commit comments

Comments
 (0)