Skip to content

Commit ae4d721

Browse files
jbeemsterjbeemster
committed
Add ability to use Confluent Cloud Kafka Cluster in Azure (closes #79)
1 parent 0c6444f commit ae4d721

File tree

6 files changed

+181
-45
lines changed

6 files changed

+181
-45
lines changed

‎terraform/azure/pipeline/README.md

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,18 +14,18 @@ No providers.
1414
| Name | Source | Version |
1515
|------|--------|---------|
1616
| <a name="module_bad_1_eh_topic"></a> [bad\_1\_eh\_topic](#module\_bad\_1\_eh\_topic) | snowplow-devops/event-hub/azurerm | 0.1.1 |
17-
| <a name="module_collector_eh"></a> [collector\_eh](#module\_collector\_eh) | snowplow-devops/collector-event-hub-vmss/azurerm | 0.1.1 |
17+
| <a name="module_collector_eh"></a> [collector\_eh](#module\_collector\_eh) | snowplow-devops/collector-event-hub-vmss/azurerm | 0.2.1 |
1818
| <a name="module_collector_lb"></a> [collector\_lb](#module\_collector\_lb) | snowplow-devops/lb/azurerm | 0.2.0 |
1919
| <a name="module_eh_namespace"></a> [eh\_namespace](#module\_eh\_namespace) | snowplow-devops/event-hub-namespace/azurerm | 0.1.1 |
20-
| <a name="module_enrich_eh"></a> [enrich\_eh](#module\_enrich\_eh) | snowplow-devops/enrich-event-hub-vmss/azurerm | 0.1.2 |
20+
| <a name="module_enrich_eh"></a> [enrich\_eh](#module\_enrich\_eh) | snowplow-devops/enrich-event-hub-vmss/azurerm | 0.2.1 |
2121
| <a name="module_enriched_eh_topic"></a> [enriched\_eh\_topic](#module\_enriched\_eh\_topic) | snowplow-devops/event-hub/azurerm | 0.1.1 |
22-
| <a name="module_lake_loader"></a> [lake\_loader](#module\_lake\_loader) | snowplow-devops/lake-loader-vmss/azurerm | 0.1.1 |
22+
| <a name="module_lake_loader"></a> [lake\_loader](#module\_lake\_loader) | snowplow-devops/lake-loader-vmss/azurerm | 0.2.1 |
2323
| <a name="module_lake_storage_container"></a> [lake\_storage\_container](#module\_lake\_storage\_container) | snowplow-devops/storage-container/azurerm | 0.1.1 |
2424
| <a name="module_raw_eh_topic"></a> [raw\_eh\_topic](#module\_raw\_eh\_topic) | snowplow-devops/event-hub/azurerm | 0.1.1 |
25-
| <a name="module_sf_loader"></a> [sf\_loader](#module\_sf\_loader) | snowplow-devops/snowflake-loader-vmss/azurerm | 0.1.1 |
25+
| <a name="module_sf_loader"></a> [sf\_loader](#module\_sf\_loader) | snowplow-devops/snowflake-loader-vmss/azurerm | 0.2.1 |
2626
| <a name="module_sf_message_queue_eh_topic"></a> [sf\_message\_queue\_eh\_topic](#module\_sf\_message\_queue\_eh\_topic) | snowplow-devops/event-hub/azurerm | 0.1.1 |
2727
| <a name="module_sf_transformer_storage_container"></a> [sf\_transformer\_storage\_container](#module\_sf\_transformer\_storage\_container) | snowplow-devops/storage-container/azurerm | 0.1.1 |
28-
| <a name="module_sf_transformer_wrj"></a> [sf\_transformer\_wrj](#module\_sf\_transformer\_wrj) | snowplow-devops/transformer-event-hub-vmss/azurerm | 0.1.1 |
28+
| <a name="module_sf_transformer_wrj"></a> [sf\_transformer\_wrj](#module\_sf\_transformer\_wrj) | snowplow-devops/transformer-event-hub-vmss/azurerm | 0.2.1 |
2929
| <a name="module_storage_account"></a> [storage\_account](#module\_storage\_account) | snowplow-devops/storage-account/azurerm | 0.1.2 |
3030

3131
## Resources
@@ -45,6 +45,13 @@ No resources.
4545
| <a name="input_storage_account_name"></a> [storage\_account\_name](#input\_storage\_account\_name) | The name of the Storage Account the data will be loaded into | `string` | n/a | yes |
4646
| <a name="input_subnet_id_lb"></a> [subnet\_id\_lb](#input\_subnet\_id\_lb) | The ID of the subnet to deploy the load balancer into (e.g. collector-agw1) | `string` | n/a | yes |
4747
| <a name="input_subnet_id_servers"></a> [subnet\_id\_servers](#input\_subnet\_id\_servers) | The ID of the subnet to deploy the servers into (e.g. pipeline1) | `string` | n/a | yes |
48+
| <a name="input_confluent_cloud_api_key"></a> [confluent\_cloud\_api\_key](#input\_confluent\_cloud\_api\_key) | Confluent Cloud API Key | `string` | `""` | no |
49+
| <a name="input_confluent_cloud_api_secret"></a> [confluent\_cloud\_api\_secret](#input\_confluent\_cloud\_api\_secret) | Confluent Cloud API Secret | `string` | `""` | no |
50+
| <a name="input_confluent_cloud_bad_1_topic_name"></a> [confluent\_cloud\_bad\_1\_topic\_name](#input\_confluent\_cloud\_bad\_1\_topic\_name) | Confluent Cloud 'bad-1' topic name | `string` | `"bad-1"` | no |
51+
| <a name="input_confluent_cloud_bootstrap_server"></a> [confluent\_cloud\_bootstrap\_server](#input\_confluent\_cloud\_bootstrap\_server) | Confluent Cloud cluster bootstrap server | `string` | `""` | no |
52+
| <a name="input_confluent_cloud_enriched_topic_name"></a> [confluent\_cloud\_enriched\_topic\_name](#input\_confluent\_cloud\_enriched\_topic\_name) | Confluent Cloud 'enriched' topic name | `string` | `"enriched"` | no |
53+
| <a name="input_confluent_cloud_raw_topic_name"></a> [confluent\_cloud\_raw\_topic\_name](#input\_confluent\_cloud\_raw\_topic\_name) | Confluent Cloud 'raw' topic name | `string` | `"raw"` | no |
54+
| <a name="input_confluent_cloud_snowflake_loader_topic_name"></a> [confluent\_cloud\_snowflake\_loader\_topic\_name](#input\_confluent\_cloud\_snowflake\_loader\_topic\_name) | Confluent Cloud 'snowflake-loader' topic name | `string` | `"snowflake-loader"` | no |
4855
| <a name="input_lake_enabled"></a> [lake\_enabled](#input\_lake\_enabled) | Whether to load all data into a Storage Container to build a data-lake based on Delta format | `bool` | `false` | no |
4956
| <a name="input_snowflake_account"></a> [snowflake\_account](#input\_snowflake\_account) | Snowflake account to use | `string` | `""` | no |
5057
| <a name="input_snowflake_database"></a> [snowflake\_database](#input\_snowflake\_database) | Snowflake database name | `string` | `""` | no |
@@ -57,6 +64,7 @@ No resources.
5764
| <a name="input_snowflake_warehouse"></a> [snowflake\_warehouse](#input\_snowflake\_warehouse) | Snowflake warehouse name | `string` | `""` | no |
5865
| <a name="input_ssl_information"></a> [ssl\_information](#input\_ssl\_information) | SSL certificate information to optionally bind to the load balancer | <pre>object({<br> enabled = bool<br> data = string<br> password = string<br> })</pre> | <pre>{<br> "data": "",<br> "enabled": false,<br> "password": ""<br>}</pre> | no |
5966
| <a name="input_storage_account_deploy"></a> [storage\_account\_deploy](#input\_storage\_account\_deploy) | Whether this module should create a new storage account with the specified name - if the account already exists set this to false | `bool` | `true` | no |
67+
| <a name="input_stream_type"></a> [stream\_type](#input\_stream\_type) | The stream type to use as the Kafka Cluster between components (options: azure\_event\_hubs, confluent\_cloud) | `string` | `"azure_event_hubs"` | no |
6068
| <a name="input_tags"></a> [tags](#input\_tags) | The tags to append to the resources in this module | `map(string)` | `{}` | no |
6169
| <a name="input_telemetry_enabled"></a> [telemetry\_enabled](#input\_telemetry\_enabled) | Whether or not to send telemetry information back to Snowplow Analytics Ltd | `bool` | `true` | no |
6270
| <a name="input_user_provided_id"></a> [user\_provided\_id](#input\_user\_provided\_id) | An optional unique identifier to identify the telemetry events emitted by this stack | `string` | `""` | no |

‎terraform/azure/pipeline/main.tf

Lines changed: 54 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -24,13 +24,21 @@ module "storage_account" {
2424

2525
locals {
2626
storage_account_name = var.storage_account_deploy ? join("", module.storage_account.*.name) : var.storage_account_name
27+
28+
# Note: as the options are only EventHubs or Confluent Cloud we want to default to EventHubs
29+
# unless Confluent Cloud is *explictly selected*.
30+
#
31+
# This logic will need to change should we support multiple streaming options.
32+
use_azure_event_hubs = var.stream_type != "confluent_cloud"
2733
}
2834

2935
# 1. Deploy EventHubs topics
3036
module "eh_namespace" {
3137
source = "snowplow-devops/event-hub-namespace/azurerm"
3238
version = "0.1.1"
3339

40+
count = local.use_azure_event_hubs ? 1 : 0
41+
3442
name = "${var.prefix}-namespace"
3543
resource_group_name = var.resource_group_name
3644

@@ -41,30 +49,49 @@ module "raw_eh_topic" {
4149
source = "snowplow-devops/event-hub/azurerm"
4250
version = "0.1.1"
4351

52+
count = local.use_azure_event_hubs ? 1 : 0
53+
4454
name = "raw-topic"
45-
namespace_name = module.eh_namespace.name
55+
namespace_name = join("", module.eh_namespace.*.name)
4656
resource_group_name = var.resource_group_name
4757
}
4858

4959
module "bad_1_eh_topic" {
5060
source = "snowplow-devops/event-hub/azurerm"
5161
version = "0.1.1"
5262

63+
count = local.use_azure_event_hubs ? 1 : 0
64+
5365
name = "bad-1-topic"
54-
namespace_name = module.eh_namespace.name
66+
namespace_name = join("", module.eh_namespace.*.name)
5567
resource_group_name = var.resource_group_name
5668
}
5769

5870
module "enriched_eh_topic" {
5971
source = "snowplow-devops/event-hub/azurerm"
6072
version = "0.1.1"
6173

74+
count = local.use_azure_event_hubs ? 1 : 0
75+
6276
name = "enriched-topic"
63-
namespace_name = module.eh_namespace.name
77+
namespace_name = join("", module.eh_namespace.*.name)
6478
resource_group_name = var.resource_group_name
6579
}
6680

67-
# 2. Deploy Collector stack
81+
# 2. Figure out which Kafka Cluster to use
82+
83+
locals {
84+
kafka_brokers = local.use_azure_event_hubs ? join("", module.eh_namespace.*.broker) : var.confluent_cloud_bootstrap_server
85+
kafka_username = local.use_azure_event_hubs ? "$ConnectionString" : var.confluent_cloud_api_key
86+
87+
eh_namespace_name = local.use_azure_event_hubs ? join("", module.eh_namespace.*.name) : ""
88+
89+
raw_topic_name = local.use_azure_event_hubs ? join("", module.raw_eh_topic.*.name) : var.confluent_cloud_raw_topic_name
90+
bad_1_topic_name = local.use_azure_event_hubs ? join("", module.bad_1_eh_topic.*.name) : var.confluent_cloud_bad_1_topic_name
91+
enriched_topic_name = local.use_azure_event_hubs ? join("", module.enriched_eh_topic.*.name) : var.confluent_cloud_enriched_topic_name
92+
}
93+
94+
# 3. Deploy Collector stack
6895
module "collector_lb" {
6996
source = "snowplow-devops/lb/azurerm"
7097
version = "0.2.0"
@@ -84,7 +111,7 @@ module "collector_lb" {
84111

85112
module "collector_eh" {
86113
source = "snowplow-devops/collector-event-hub-vmss/azurerm"
87-
version = "0.1.1"
114+
version = "0.2.1"
88115

89116
name = "${var.prefix}-collector"
90117
resource_group_name = var.resource_group_name
@@ -97,21 +124,24 @@ module "collector_eh" {
97124
ssh_public_key = var.ssh_public_key
98125
ssh_ip_allowlist = var.ssh_ip_allowlist
99126

100-
good_topic_name = module.raw_eh_topic.name
101-
bad_topic_name = module.bad_1_eh_topic.name
102-
eh_namespace_broker = module.eh_namespace.broker
103-
eh_namespace_read_write_connection_string = module.eh_namespace.read_write_primary_connection_string
127+
good_topic_name = local.raw_topic_name
128+
bad_topic_name = local.bad_1_topic_name
129+
kafka_brokers = local.kafka_brokers
130+
kafka_username = local.kafka_username
131+
kafka_password = local.use_azure_event_hubs ? join("", module.eh_namespace.*.read_write_primary_connection_string) : var.confluent_cloud_api_secret
132+
133+
kafka_source = var.stream_type
104134

105135
telemetry_enabled = var.telemetry_enabled
106136
user_provided_id = var.user_provided_id
107137

108138
tags = var.tags
109139
}
110140

111-
# 3. Deploy Enrich stack
141+
# 4. Deploy Enrich stack
112142
module "enrich_eh" {
113143
source = "snowplow-devops/enrich-event-hub-vmss/azurerm"
114-
version = "0.1.2"
144+
version = "0.2.1"
115145

116146
name = "${var.prefix}-enrich"
117147
resource_group_name = var.resource_group_name
@@ -120,14 +150,19 @@ module "enrich_eh" {
120150
ssh_public_key = var.ssh_public_key
121151
ssh_ip_allowlist = var.ssh_ip_allowlist
122152

123-
raw_topic_name = module.raw_eh_topic.name
124-
raw_topic_connection_string = module.raw_eh_topic.read_only_primary_connection_string
125-
good_topic_name = module.enriched_eh_topic.name
126-
good_topic_connection_string = module.enriched_eh_topic.read_write_primary_connection_string
127-
bad_topic_name = module.bad_1_eh_topic.name
128-
bad_topic_connection_string = module.bad_1_eh_topic.read_write_primary_connection_string
129-
eh_namespace_name = module.eh_namespace.name
130-
eh_namespace_broker = module.eh_namespace.broker
153+
raw_topic_name = local.raw_topic_name
154+
raw_topic_kafka_username = local.kafka_username
155+
raw_topic_kafka_password = local.use_azure_event_hubs ? join("", module.raw_eh_topic.*.read_only_primary_connection_string) : var.confluent_cloud_api_secret
156+
good_topic_name = local.enriched_topic_name
157+
good_topic_kafka_username = local.kafka_username
158+
good_topic_kafka_password = local.use_azure_event_hubs ? join("", module.enriched_eh_topic.*.read_write_primary_connection_string) : var.confluent_cloud_api_secret
159+
bad_topic_name = local.bad_1_topic_name
160+
bad_topic_kafka_username = local.kafka_username
161+
bad_topic_kafka_password = local.use_azure_event_hubs ? join("", module.bad_1_eh_topic.*.read_write_primary_connection_string) : var.confluent_cloud_api_secret
162+
eh_namespace_name = local.eh_namespace_name
163+
kafka_brokers = local.kafka_brokers
164+
165+
kafka_source = var.stream_type
131166

132167
telemetry_enabled = var.telemetry_enabled
133168
user_provided_id = var.user_provided_id

‎terraform/azure/pipeline/target_lake_loader.tf

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -10,20 +10,24 @@ module "lake_storage_container" {
1010

1111
module "lake_loader" {
1212
source = "snowplow-devops/lake-loader-vmss/azurerm"
13-
version = "0.1.1"
13+
version = "0.2.1"
1414

1515
count = var.lake_enabled ? 1 : 0
1616

1717
name = "${var.prefix}-lake-loader"
1818
resource_group_name = var.resource_group_name
1919
subnet_id = var.subnet_id_servers
2020

21-
enriched_topic_name = module.enriched_eh_topic.name
22-
enriched_topic_connection_string = module.enriched_eh_topic.read_only_primary_connection_string
23-
bad_topic_name = module.bad_1_eh_topic.name
24-
bad_topic_connection_string = module.bad_1_eh_topic.read_write_primary_connection_string
25-
eh_namespace_name = module.eh_namespace.name
26-
eh_namespace_broker = module.eh_namespace.broker
21+
enriched_topic_name = local.enriched_topic_name
22+
enriched_topic_kafka_username = local.kafka_username
23+
enriched_topic_kafka_password = local.use_azure_event_hubs ? join("", module.enriched_eh_topic.*.read_only_primary_connection_string) : var.confluent_cloud_api_secret
24+
bad_topic_name = local.bad_1_topic_name
25+
bad_topic_kafka_username = local.kafka_username
26+
bad_topic_kafka_password = local.use_azure_event_hubs ? join("", module.bad_1_eh_topic.*.read_write_primary_connection_string) : var.confluent_cloud_api_secret
27+
eh_namespace_name = local.eh_namespace_name
28+
kafka_brokers = local.kafka_brokers
29+
30+
kafka_source = var.stream_type
2731

2832
storage_account_name = local.storage_account_name
2933
storage_container_name = module.lake_storage_container[0].name

‎terraform/azure/pipeline/target_snowflake.tf

Lines changed: 25 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,17 @@ module "sf_message_queue_eh_topic" {
22
source = "snowplow-devops/event-hub/azurerm"
33
version = "0.1.1"
44

5-
count = var.snowflake_enabled ? 1 : 0
5+
count = local.use_azure_event_hubs && var.snowflake_enabled ? 1 : 0
66

77
name = "snowflake-loader-topic"
8-
namespace_name = module.eh_namespace.name
8+
namespace_name = join("", module.eh_namespace.*.name)
99
resource_group_name = var.resource_group_name
1010
}
1111

12+
locals {
13+
snowflake_loader_topic_name = local.use_azure_event_hubs ? join("", module.sf_message_queue_eh_topic.*.name) : var.confluent_cloud_snowflake_loader_topic_name
14+
}
15+
1216
module "sf_transformer_storage_container" {
1317
source = "snowplow-devops/storage-container/azurerm"
1418
version = "0.1.1"
@@ -21,20 +25,24 @@ module "sf_transformer_storage_container" {
2125

2226
module "sf_transformer_wrj" {
2327
source = "snowplow-devops/transformer-event-hub-vmss/azurerm"
24-
version = "0.1.1"
28+
version = "0.2.1"
2529

2630
count = var.snowflake_enabled ? 1 : 0
2731

2832
name = "${var.prefix}-snowflake-transformer"
2933
resource_group_name = var.resource_group_name
3034
subnet_id = var.subnet_id_servers
3135

32-
enriched_topic_name = module.enriched_eh_topic.name
33-
enriched_topic_connection_string = module.enriched_eh_topic.read_only_primary_connection_string
34-
queue_topic_name = module.sf_message_queue_eh_topic[0].name
35-
queue_topic_connection_string = module.sf_message_queue_eh_topic[0].read_write_primary_connection_string
36-
eh_namespace_name = module.eh_namespace.name
37-
eh_namespace_broker = module.eh_namespace.broker
36+
enriched_topic_name = local.enriched_topic_name
37+
enriched_topic_kafka_username = local.kafka_username
38+
enriched_topic_kafka_password = local.use_azure_event_hubs ? join("", module.enriched_eh_topic.*.read_only_primary_connection_string) : var.confluent_cloud_api_secret
39+
queue_topic_name = local.snowflake_loader_topic_name
40+
queue_topic_kafka_username = local.kafka_username
41+
queue_topic_kafka_password = local.use_azure_event_hubs ? join("", module.sf_message_queue_eh_topic.*.read_write_primary_connection_string) : var.confluent_cloud_api_secret
42+
eh_namespace_name = local.eh_namespace_name
43+
kafka_brokers = local.kafka_brokers
44+
45+
kafka_source = var.stream_type
3846

3947
storage_account_name = local.storage_account_name
4048
storage_container_name = module.sf_transformer_storage_container[0].name
@@ -57,18 +65,21 @@ module "sf_transformer_wrj" {
5765

5866
module "sf_loader" {
5967
source = "snowplow-devops/snowflake-loader-vmss/azurerm"
60-
version = "0.1.1"
68+
version = "0.2.1"
6169

6270
count = var.snowflake_enabled ? 1 : 0
6371

6472
name = "${var.prefix}-snowflake-loader"
6573
resource_group_name = var.resource_group_name
6674
subnet_id = var.subnet_id_servers
6775

68-
queue_topic_name = module.sf_message_queue_eh_topic[0].name
69-
queue_topic_connection_string = module.sf_message_queue_eh_topic[0].read_only_primary_connection_string
70-
eh_namespace_name = module.eh_namespace.name
71-
eh_namespace_broker = module.eh_namespace.broker
76+
queue_topic_name = local.snowflake_loader_topic_name
77+
queue_topic_kafka_username = local.kafka_username
78+
queue_topic_kafka_password = local.use_azure_event_hubs ? join("", module.sf_message_queue_eh_topic.*.read_only_primary_connection_string) : var.confluent_cloud_api_secret
79+
eh_namespace_name = local.eh_namespace_name
80+
kafka_brokers = local.kafka_brokers
81+
82+
kafka_source = var.stream_type
7283

7384
storage_account_name = local.storage_account_name
7485
storage_container_name_for_transformer_output = module.sf_transformer_storage_container[0].name

‎terraform/azure/pipeline/terraform.tfvars

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,30 @@ ssl_information = {
3636
enabled = false
3737
}
3838

39+
# --- Stream Selection
40+
41+
# The stream type to use between deployed components:
42+
#
43+
# 1. azure_event_hubs: If selected will deploy a namespace and topics into the same resource
44+
# group as all other assets (no extra steps required)
45+
# 2. confluent_cloud: If selected you will need to manually deploy a Cluster and associated
46+
# topics for the applications to stream data into
47+
stream_type = "azure_event_hubs"
48+
49+
# --- Stream: Confluent Cloud
50+
# API Key details for your deployed cluster
51+
confluent_cloud_api_key = ""
52+
confluent_cloud_api_secret = ""
53+
54+
# Bootstrap server for your deployed cluster
55+
confluent_cloud_bootstrap_server = ""
56+
57+
# Names of the created topics within the deployed cluster
58+
confluent_cloud_raw_topic_name = "raw"
59+
confluent_cloud_enriched_topic_name = "enriched"
60+
confluent_cloud_bad_1_topic_name = "bad-1"
61+
confluent_cloud_snowflake_loader_topic_name = "snowflake-loader"
62+
3963
# --- Target: Snowflake
4064
# Follow the guide to get input values for the loader:
4165
# https://docs.snowplow.io/docs/getting-started-on-snowplow-open-source/quick-start/

0 commit comments

Comments
 (0)