From 42e05abc4d69dd73ad86fe2b3bc5f97024bf2b8e Mon Sep 17 00:00:00 2001 From: philkra Date: Thu, 4 Sep 2025 17:05:45 +0200 Subject: [PATCH 1/7] remove tiger lake limitation --- use-timescale/tigerlake.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/use-timescale/tigerlake.md b/use-timescale/tigerlake.md index cd33516912..d628a63534 100644 --- a/use-timescale/tigerlake.md +++ b/use-timescale/tigerlake.md @@ -227,7 +227,9 @@ statement: ```sql ALTER TABLE SET ( tigerlake.iceberg_sync = true | false, - tigerlake.iceberg_partitionby = '' + tigerlake.iceberg_partitionby = '', + tigerlake.iceberg_namespace = '', + tigerlake.iceberg_table = '' ) ``` @@ -236,6 +238,8 @@ ALTER TABLE SET ( * `tigerlake.iceberg_partitionby`: optional property to define a partition specification in Iceberg. By default the Iceberg table is partitioned as `day()`. This default behavior is only applicable to $HYPERTABLEs. For more information, see [partitioning][partitioning]. +* `tigerlake.iceberg_namespace`: optional property to set a namespace, the default is `timescaledb`. +* `tigerlake.iceberg_table`: optional property to specify a different table name. If no name is specified the $PG table name is used. ### Partitioning intervals @@ -317,16 +321,14 @@ data lake: ## Limitations -* Only $PG 17.4 is supported. Services running $PG 17.5 are downgraded to 17.4. +* Only $PG 17.6 and above is supported. * Consistent ingestion rates of over 50000 records / second can lead to a lost replication slot. * [Amazon S3 Tables Iceberg REST][aws-s3-tables] catalog only is supported. * In order to collect deletes made to data in the columstore, certain columnstore optimizations are disabled for $HYPERTABLEs. * The `TRUNCATE` statement is not supported, and does not truncate data in the corresponding Iceberg table. * Data in a $HYPERTABLE that has been moved to the [low-cost object storage tier][data-tiering] is not synced. -* Renaming a table in $PG stops the sync to Iceberg and causes unexpected behavior. * Writing to the same S3 table bucket from multiple services is not supported, bucket-to-service mapping is one-to-one. * Iceberg snapshots are pruned automatically if the amount exceeds 2500. -* The Iceberg namespace is hard coded to `timescaledb`, a custom namespace value is work in progress. [cmc]: https://console.aws.amazon.com/cloudformation/ [aws-athena]: https://aws.amazon.com/athena/ From 98929987ee89c0326bb1a67bce8f9c1a1b4023c5 Mon Sep 17 00:00:00 2001 From: philkra Date: Mon, 22 Sep 2025 08:37:49 +0200 Subject: [PATCH 2/7] iam compat policy --- use-timescale/tigerlake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/use-timescale/tigerlake.md b/use-timescale/tigerlake.md index d628a63534..f4c6f5d5d4 100644 --- a/use-timescale/tigerlake.md +++ b/use-timescale/tigerlake.md @@ -89,7 +89,7 @@ To connect a $SERVICE_LONG to your data lake: ```shell aws cloudformation create-stack \ - --capabilities CapabilityIAM \ + --capabilities CAPABILITY_IAM \ --template-url https://tigerlake.s3.us-east-1.amazonaws.com/tigerlake-connect-cloudformation.yaml \ --region \ --stack-name \ From 7dad45cb5764aeb9faf84f2458a43765584498c4 Mon Sep 17 00:00:00 2001 From: philkra Date: Fri, 26 Sep 2025 17:08:16 +0200 Subject: [PATCH 3/7] restart no longer required --- use-timescale/tigerlake.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/use-timescale/tigerlake.md b/use-timescale/tigerlake.md index f4c6f5d5d4..35edcd32ac 100644 --- a/use-timescale/tigerlake.md +++ b/use-timescale/tigerlake.md @@ -67,7 +67,7 @@ To connect a $SERVICE_LONG to your data lake: - ARN of the S3Table bucket - ARN of a role with permissions to write to the table bucket - Provisioning takes a couple of minutes, during this time the $SERVICE_SHORT is restarted. + Provisioning takes a couple of minutes. @@ -111,7 +111,7 @@ To connect a $SERVICE_LONG to your data lake: - ARN of the S3Table bucket - ARN of a role with permissions to write to the table bucket - Provisioning takes a couple of minutes, during this time the $SERVICE_SHORT is restarted. + Provisioning takes a couple of minutes. @@ -204,7 +204,7 @@ To connect a $SERVICE_LONG to your data lake: - ARN of the S3Table bucket - ARN of a role with permissions to write to the table bucket - Provisioning takes a couple of minutes, during this time the $SERVICE_SHORT is restarted. + Provisioning takes a couple of minutes. From aafc514af003e5903576572f3b3ceb4d25a25423 Mon Sep 17 00:00:00 2001 From: philkra Date: Mon, 6 Oct 2025 08:35:08 +0200 Subject: [PATCH 4/7] something --- use-timescale/tigerlake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/use-timescale/tigerlake.md b/use-timescale/tigerlake.md index 35edcd32ac..0792a6110b 100644 --- a/use-timescale/tigerlake.md +++ b/use-timescale/tigerlake.md @@ -321,7 +321,7 @@ data lake: ## Limitations -* Only $PG 17.6 and above is supported. +* Service requires $PG 17.6 and above is supported. * Consistent ingestion rates of over 50000 records / second can lead to a lost replication slot. * [Amazon S3 Tables Iceberg REST][aws-s3-tables] catalog only is supported. * In order to collect deletes made to data in the columstore, certain columnstore optimizations are disabled for $HYPERTABLEs. From 4ff55c42bb20a394ca6e393b1ffa7ccde230c86a Mon Sep 17 00:00:00 2001 From: philkra Date: Sun, 12 Oct 2025 10:05:14 +0200 Subject: [PATCH 5/7] add table name and namespace API --- use-timescale/tigerlake.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/use-timescale/tigerlake.md b/use-timescale/tigerlake.md index 0792a6110b..0c27348adb 100644 --- a/use-timescale/tigerlake.md +++ b/use-timescale/tigerlake.md @@ -319,6 +319,30 @@ data lake: When you change the partition, you **do not** have to pause the sync to Iceberg. Apache Iceberg handles the partitioning operation in function of the internal implementation. +**Specify a different namespace** + + Tables are created, by default in the the `timescaledb` namespace, use the `tigerlake.iceberg_namespace` property to specify a different namespace when starting the sync through the following statement: + + ```sql + ALTER TABLE my_hypertable SET ( + tigerlake.iceberg_sync = true, + tigerlake.iceberg_namespace = 'my_namespace' + ); + ``` + +**Specify a different Iceberg table name** + + The table name in Iceberg is the same as the source table in $CLOUD_LONG. + Some services do not allow mixed case or have other constraints for table names. + With the `tigerlake.iceberg_table` property a different table name for the Iceberg table can be defined on sync start. + + ```sql + ALTER TABLE Mixed_CASE_TableNAME SET ( + tigerlake.iceberg_sync = true, + tigerlake.iceberg_table = 'my_table_name' + ); + ``` + ## Limitations * Service requires $PG 17.6 and above is supported. From 725aa92f24f0e98a4efcc19dbea8285d8766a818 Mon Sep 17 00:00:00 2001 From: philkra Date: Sun, 12 Oct 2025 10:08:59 +0200 Subject: [PATCH 6/7] amend limitations --- use-timescale/tigerlake.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/use-timescale/tigerlake.md b/use-timescale/tigerlake.md index 0c27348adb..f98781c415 100644 --- a/use-timescale/tigerlake.md +++ b/use-timescale/tigerlake.md @@ -346,9 +346,10 @@ data lake: ## Limitations * Service requires $PG 17.6 and above is supported. -* Consistent ingestion rates of over 50000 records / second can lead to a lost replication slot. +* Consistent ingestion rates of over 30000 records / second can lead to a lost replication slot. Burst can be feathered out over time. * [Amazon S3 Tables Iceberg REST][aws-s3-tables] catalog only is supported. * In order to collect deletes made to data in the columstore, certain columnstore optimizations are disabled for $HYPERTABLEs. +* [Direct Compress][direct-compress] is not supported. * The `TRUNCATE` statement is not supported, and does not truncate data in the corresponding Iceberg table. * Data in a $HYPERTABLE that has been moved to the [low-cost object storage tier][data-tiering] is not synced. * Writing to the same S3 table bucket from multiple services is not supported, bucket-to-service mapping is one-to-one. @@ -372,3 +373,4 @@ data lake: [services-portal]: https://console.cloud.timescale.com/dashboard/services [aws-s3-tables]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-open-source.html [data-tiering]: /use-timescale/:currentVersion:/data-tiering/ +[direct-compress]: /use-timescale/:currentVersion:/hypertables/hypertable-crud/#speed-up-data-ingestion \ No newline at end of file From db33d3ef1aff30bc54a53e163da6ab3a4c64c7b4 Mon Sep 17 00:00:00 2001 From: Philip Krauss <35487337+philkra@users.noreply.github.com> Date: Mon, 13 Oct 2025 12:48:32 +0200 Subject: [PATCH 7/7] Apply suggestions from code review Co-authored-by: Iain Cox Signed-off-by: Philip Krauss <35487337+philkra@users.noreply.github.com> --- use-timescale/tigerlake.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/use-timescale/tigerlake.md b/use-timescale/tigerlake.md index f98781c415..615a2a1014 100644 --- a/use-timescale/tigerlake.md +++ b/use-timescale/tigerlake.md @@ -321,7 +321,7 @@ data lake: **Specify a different namespace** - Tables are created, by default in the the `timescaledb` namespace, use the `tigerlake.iceberg_namespace` property to specify a different namespace when starting the sync through the following statement: + By default, tables are created in the the `timescaledb` namespace. To specify a different namespace when you start the sync, use the `tigerlake.iceberg_namespace` property. For example: ```sql ALTER TABLE my_hypertable SET ( @@ -333,8 +333,8 @@ data lake: **Specify a different Iceberg table name** The table name in Iceberg is the same as the source table in $CLOUD_LONG. - Some services do not allow mixed case or have other constraints for table names. - With the `tigerlake.iceberg_table` property a different table name for the Iceberg table can be defined on sync start. + Some services do not allow mixed case, or have other constraints for table names. + To define a different table name for the Iceberg table at sync start, use the `tigerlake.iceberg_table` property. For example: ```sql ALTER TABLE Mixed_CASE_TableNAME SET (