Skip to content

Commit 71ba182

Browse files
iamKunalGuptaiamKunalGupta
authored
refactor(clickpipes): reverse private link for postgres (#3696)
* refactor(clickpipes): reverse private link for postgres * chore: add to aspell-dict * fix: info box
1 parent 625b89b commit 71ba182

File tree

6 files changed

+32
-19
lines changed

6 files changed

+32
-19
lines changed

‎docs/integrations/data-ingestion/clickpipes/aws-privatelink.md

+4-2
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ that will be accessible in a ClickPipes VPC.
7777
It's a preferred choice for:
7878

7979
- Any on-premise Kafka setup that requires private DNS support
80-
- Cross-region connectivity for Postgres CDC
80+
- [Cross-region connectivity for Postgres CDC](/knowledgebase/aws-privatelink-setup-for-clickpipes)
8181
- Cross-region connectivity for MSK cluster. Please reach out to the ClickHouse support team for assistance.
8282

8383
See the [getting started](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html) guide for more details.
@@ -113,8 +113,10 @@ can be configured for ClickPipes. Add [your ClickPipe region](#aws-privatelink-r
113113
5. Select any of existing reverse private endpoints or create a new one.
114114

115115
:::info
116-
For RDS, the process to create the VPC Endpoint service is a bit more involved and
116+
If cross-region access is required for RDS, you need to create a VPC endpoint service and
117117
[this guide should provide](/knowledgebase/aws-privatelink-setup-for-clickpipes) a good starting point to set it up.
118+
119+
For same-region access, creating a VPC Resource is the recommended approach.
118120
:::
119121

120122
<Image img={cp_rpe_step1} alt="Select reverse private endpoint" size="lg" border/>

‎docs/integrations/data-ingestion/clickpipes/index.md

+15-12
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ import S3svg from '@site/static/images/integrations/logos/amazon_s3_logo.svg';
1414
import Amazonkinesis from '@site/static/images/integrations/logos/amazon_kinesis_logo.svg';
1515
import Gcssvg from '@site/static/images/integrations/logos/gcs.svg';
1616
import Postgressvg from '@site/static/images/integrations/logos/postgresql.svg';
17+
import Mysqlsvg from '@site/static/images/integrations/logos/mysql.svg';
1718
import redpanda_logo from '@site/static/images/integrations/logos/logo_redpanda.png';
1819
import clickpipes_stack from '@site/static/images/integrations/data-ingestion/clickpipes/clickpipes_stack.png';
1920
import cp_custom_role from '@site/static/images/integrations/data-ingestion/clickpipes/cp_custom_role.png';
@@ -29,18 +30,20 @@ import Image from '@theme/IdealImage';
2930

3031
## Supported Data Sources {#supported-data-sources}
3132

32-
| Name |Logo|Type| Status | Description |
33-
|----------------------|----|----|-----------------|------------------------------------------------------------------------------------------------------|
34-
| Apache Kafka |<Kafkasvg class="image" alt="Apache Kafka logo" style={{width: '3rem', 'height': '3rem'}}/>|Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Apache Kafka into ClickHouse Cloud. |
35-
| Confluent Cloud |<Confluentsvg class="image" alt="Confluent Cloud logo" style={{width: '3rem'}}/>|Streaming| Stable | Unlock the combined power of Confluent and ClickHouse Cloud through our direct integration. |
36-
| Redpanda |<Image img={redpanda_logo} size="logo" alt="Redpanda logo"/> |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Redpanda into ClickHouse Cloud. |
37-
| AWS MSK |<Msksvg class="image" alt="AWS MSK logo" style={{width: '3rem', 'height': '3rem'}}/>|Streaming| Stable | Configure ClickPipes and start ingesting streaming data from AWS MSK into ClickHouse Cloud. |
38-
| Azure Event Hubs |<Azureeventhubssvg class="image" alt="Azure Event Hubs logo" style={{width: '3rem'}}/>|Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Azure Event Hubs into ClickHouse Cloud. |
39-
| WarpStream |<Warpstreamsvg class="image" alt="WarpStream logo" style={{width: '3rem'}}/>|Streaming| Stable | Configure ClickPipes and start ingesting streaming data from WarpStream into ClickHouse Cloud. |
40-
| Amazon S3 |<S3svg class="image" alt="Amazon S3 logo" style={{width: '3rem', height: 'auto'}}/>|Object Storage| Stable | Configure ClickPipes to ingest large volumes of data from object storage. |
41-
| Google Cloud Storage |<Gcssvg class="image" alt="Google Cloud Storage logo" style={{width: '3rem', height: 'auto'}}/>|Object Storage| Stable | Configure ClickPipes to ingest large volumes of data from object storage. |
42-
| Amazon Kinesis |<Amazonkinesis class="image" alt="Amazon Kenesis logo" style={{width: '3rem', height: 'auto'}}/>|Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Amazon Kinesis into ClickHouse cloud. |
43-
| Postgres |<Postgressvg class="image" alt="Postgres logo" style={{width: '3rem', height: 'auto'}}/>|DBMS| Public Beta | Configure ClickPipes and start ingesting data from Postgres into ClickHouse Cloud. |
33+
| Name | Logo |Type| Status | Description |
34+
|----------------------|--------------------------------------------------------------------------------------------------|----|------------------|------------------------------------------------------------------------------------------------------|
35+
| Apache Kafka | <Kafkasvg class="image" alt="Apache Kafka logo" style={{width: '3rem', 'height': '3rem'}}/> |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Apache Kafka into ClickHouse Cloud. |
36+
| Confluent Cloud | <Confluentsvg class="image" alt="Confluent Cloud logo" style={{width: '3rem'}}/> |Streaming| Stable | Unlock the combined power of Confluent and ClickHouse Cloud through our direct integration. |
37+
| Redpanda | <Image img={redpanda_logo} size="logo" alt="Redpanda logo"/> |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Redpanda into ClickHouse Cloud. |
38+
| AWS MSK | <Msksvg class="image" alt="AWS MSK logo" style={{width: '3rem', 'height': '3rem'}}/> |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from AWS MSK into ClickHouse Cloud. |
39+
| Azure Event Hubs | <Azureeventhubssvg class="image" alt="Azure Event Hubs logo" style={{width: '3rem'}}/> |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Azure Event Hubs into ClickHouse Cloud. |
40+
| WarpStream | <Warpstreamsvg class="image" alt="WarpStream logo" style={{width: '3rem'}}/> |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from WarpStream into ClickHouse Cloud. |
41+
| Amazon S3 | <S3svg class="image" alt="Amazon S3 logo" style={{width: '3rem', height: 'auto'}}/> |Object Storage| Stable | Configure ClickPipes to ingest large volumes of data from object storage. |
42+
| Google Cloud Storage | <Gcssvg class="image" alt="Google Cloud Storage logo" style={{width: '3rem', height: 'auto'}}/> |Object Storage| Stable | Configure ClickPipes to ingest large volumes of data from object storage. |
43+
| Amazon Kinesis | <Amazonkinesis class="image" alt="Amazon Kenesis logo" style={{width: '3rem', height: 'auto'}}/> |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Amazon Kinesis into ClickHouse cloud. |
44+
| Postgres | <Postgressvg class="image" alt="Postgres logo" style={{width: '3rem', height: 'auto'}}/> |DBMS| Public Beta | Configure ClickPipes and start ingesting data from Postgres into ClickHouse Cloud. |
45+
| MySQL | <Mysqlsvg class="image" alt="MySQL logo" style={{width: '3rem', height: 'auto'}}/> |DBMS| Private Beta | Configure ClickPipes and start ingesting data from MySQL into ClickHouse Cloud. |
46+
4447

4548
More connectors will get added to ClickPipes, you can find out more by [contacting us](https://clickhouse.com/company/contact?loc=clickpipes).
4649

‎docs/integrations/data-ingestion/clickpipes/postgres/index.md

+6
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,12 @@ Make sure you are logged in to your ClickHouse Cloud account. If you don't have
8787

8888
<Image img={postgres_connection_details} alt="Fill in connection details" size="lg" border/>
8989

90+
#### (Optional) Setting up AWS Private Link {#optional-setting-up-aws-private-link}
91+
92+
You can use AWS Private Link to connect to your source Postgres database if it is hosted on AWS. This is useful if you
93+
want to keep your data transfer private.
94+
You can follow the [setup guide to set up the connection](/integrations/clickpipes/aws-privatelink).
95+
9096
#### (Optional) Setting up SSH Tunneling {#optional-setting-up-ssh-tunneling}
9197

9298
You can specify SSH tunneling details if your source Postgres database is not publicly accessible.

‎knowledgebase/aws-privatelink-setup-for-clickpipes.mdx

+6-5
Original file line numberDiff line numberDiff line change
@@ -11,12 +11,13 @@ keywords: ['AWS PrivateLink', 'Private RDS', 'ClickPipes']
1111

1212
## AWS PrivateLink setup to expose private RDS for ClickPipes {#aws-privatelink-setup-to-expose-private-rds-for-clickpipes}
1313

14-
Setup steps to expose a private RDS via AWS PrivateLink to ClickPipes.
14+
Setup steps to expose a private RDS via AWS PrivateLink to ClickPipes, also supporting cross-region access.
1515

16-
## Requirements {#requirements}
17-
18-
The VPC must be located in one of our ClickPipes regions: us-east-1, us-east-2 or eu-central-1, additionally the
19-
ClickHouse Instance must be in the same region.
16+
:::info
17+
This uses AWS Endpoint Service and is mainly suitable for Cross-Region Access. If you require same-region access,
18+
[creating a VPC Resource](/integrations/clickpipes/aws-privatelink#vpc-resource) is
19+
recommended instead.
20+
:::
2021

2122
## Private link creation {#private-link-creation}
2223

‎scripts/aspell-dict-file.txt

+1
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,7 @@ ConfluentSVG
119119
GCSSVG
120120
KafkaSVG
121121
MskSVG
122+
Mysqlsvg
122123
PostgresSVG
123124
WarpStreamSVG
124125
clickpipes

‎static/images/integrations/data-ingestion/clickpipes/postgres/postgres-connection-details.jpg

2.5 KB
Loading

0 commit comments

Comments
 (0)