Skip to content

Attempt to use startup script to add LS domain name to /etc/hosts for MWAA environment #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+21 −6
Open

Attempt to use startup script to add LS domain name to /etc/hosts for MWAA environment #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions airflow-dbt-transformation/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,18 @@ install: ## Install dependencies

init: ## Initialize the Airflow environment in LocalStack MWAA
awslocal s3 mb s3://snowflake-airflow
awslocal s3api put-bucket-versioning --bucket snowflake-airflow --versioning-configuration MFADelete=Disabled,Status=Enabled
awslocal s3 cp startup.sh s3://snowflake-airflow/startup.sh
@startup_version_id=$$(awslocal s3api list-object-versions --bucket snowflake-airflow --prefix startup.sh --query 'Versions[?IsLatest].[VersionId]' --output text) && \
awslocal mwaa create-environment --dag-s3-path /dags \
--execution-role-arn arn:aws:iam::000000000000:role/airflow-role \
--network-configuration {} \
--source-bucket-arn arn:aws:s3:::snowflake-airflow \
--airflow-configuration-options agent.code=007,agent.name=bond \
--airflow-configuration-options agent.code=007,agent.name=bond,env.LOCALSTACK_RESOLVE_IP_ADDRESS=172.20.0.2 \
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a fixed IP address for the MWAA emulator?

--name my-mwaa-env \
--endpoint-url http://localhost.localstack.cloud:4566
--endpoint-url http://localhost.localstack.cloud:4566 \
--startup-script-s3-path startup.sh \
--startup-script-s3-object-version $$startup_version_id

deploy: ## Deploy the DAG to the local Airflow instance
awslocal s3 cp requirements.txt s3://snowflake-airflow/
Expand Down
3 changes: 2 additions & 1 deletion airflow-dbt-transformation/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@ The code is based on the Snowflake Guide for [Data Engineering with Apache Airfl
Start the LocalStack Snowflake emulator using the following command:

```bash
DOCKER_FLAGS='-e SF_LOG=trace' \
docker network create --attachable --subnet 172.20.0.1/24 localstack
DOCKER_FLAGS='-e SF_LOG=trace --network localstack --ip 172.20.0.2' \
IMAGE_NAME=localstack/snowflake \
DEBUG=1 \
localstack start
Expand Down
6 changes: 3 additions & 3 deletions airflow-dbt-transformation/airflow_dag.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@

from airflow import settings
from airflow.models import Connection
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.empty import EmptyOperator
from airflow.decorators import dag, task
from cosmos import DbtTaskGroup, DbtDag, ProjectConfig, ProfileConfig, ExecutionConfig
from cosmos import DbtTaskGroup, ProjectConfig, ProfileConfig, ExecutionConfig
from cosmos.profiles.snowflake.user_pass import SnowflakeUserPasswordProfileMapping
from snowflake import connector

Expand Down Expand Up @@ -76,7 +76,7 @@ def dbt_snowpark_dag():
operator_args={"install_deps": True},
)

intermediate = DummyOperator(task_id='intermediate')
intermediate = EmptyOperator(task_id='intermediate')

@task
def query_result_data():
Expand Down
9 changes: 9 additions & 0 deletions airflow-dbt-transformation/startup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#!/bin/sh

if [ -z "${!LOCALSTACK_RESOLVE_IP_ADDRESS}" ]; then
echo "Resolving *.localhost.localstack.cloud to ${LOCALSTACK_RESOLVE_IP_ADDRESS}"
echo "${LOCALSTACK_RESOLVE_IP_ADDRESS} snowflake.localhost.localstack.cloud" >> /etc/hosts
echo "${LOCALSTACK_RESOLVE_IP_ADDRESS} localhost.localstack.cloud" >> /etc/hosts
else
echo "Skipping adding domain resolver to /etc/hosts"
fi