How do regional geographic locations in the US impact the frequency and severity of extreme weather events like heavy rain and heatwaves?
This project builds a data pipeline to analyze weather data across major U.S. climate regions for the years 2024–2025, enabling structured analytics and visualization.
✔ Highlights seasonal variation
✔ Shows temperature differences across climate regions
✔ Compares extreme precipitation events
✔ Identifies high-rainfall regions (e.g., Gulf Coast)
| Layer | Tools |
|---|---|
| Infrastructure | Terraform |
| Orchestration | Prefect |
| Ingestion | Python (Requests, Pandas) |
| Storage | Google Cloud Storage (GCS) |
| Data Warehouse | BigQuery |
| Transformation | dbt |
| Visualization | Looker Studio |
- Extracts data from NOAA GSOD API
- Processes 5 U.S. stations
- Stores data in GCS as Parquet files
-
The pipeline is made to run daily using Prefect deployment.
-
Currently, to keep things simple, the pipeline processes selected historical years (2024–2025), but it can be extended to support dynamic year selection so the batch processing makes more sense.
-
That will allow automated ingestion of data from NOAA GSOD API
-
Future enhancements can include incremental ingestion and backfill strategies.
-
I have configured my prefect deployment to run on a daily schedule as an example
- Staging → Data cleaning
- Intermediate → Standardization
- Marts → Business-ready models
Medallion Architecture
Outputs:
- Fact table
- Dimension table
- Enriched data table
- Looker Studio connected to BigQuery
- Dashboards built to answer problem statement
- Temperature metrics
- Precipitation
- Flags:
- Heatwave
- Freeze
- Heavy rain
- Station ID
- Climate region
- State
Follow the steps below to replicate the infrastructure, orchestration, and transformation layers.
- Python 3.9+ installed locally.
- Docker & Docker Compose (for Prefect Server).
- GCP Service Account with
Storage AdminandBigQuery Adminroles.
1️⃣ Clone Repository
git clone <this repo url>
cd US_Weather_Analysis
2️⃣ Infrastructure as Code (Terraform) Provision GCP resources including the GCS bucket and BigQuery warehouse.
cd terraform
terraform init
terraform apply
Resources created: GCS Bucket: Raw data landing zone. BigQuery Dataset: Warehouse for staging and production models.
3️⃣ Start Prefect Server Spin up the orchestration engine using Docker.
docker-compose up -d
👉 Prefect UI: http://localhost:4200
4️⃣ Configure Prefect Blocks To keep the pipeline secure and dynamic, you must configure two blocks in the Prefect UI:
🔐 GCP Credentials Block
Block Name: gcp-creds
Add your Service Account JSON. This allows Prefect to authenticate with your GCP project.
🪣 GCS Bucket Block
Block Name: weather-bucket
Create gcs bucket block
Link the gcp-creds block and specify your bucket name.
5️⃣ Run Ingestion Pipeline This script extracts data from the weather API, cleans it using Pandas, and persists it to GCS as Parquet files.
python3 ingest_weather.py
6️⃣ Pipeline Deployment (Batch Scheduling) Register the flow and apply the batch schedule via the deployment manifest.
prefect deployment apply deployment.yaml
7️⃣ Analytics Engineering (dbt) Transforming raw data into a production-ready Star Schema.
cd dbt_project
dbt run
dbt test
Staging: Schema enforcement and renaming.
Intermediate: Flagging "Extreme" events (Heatwaves/Heavy Rain).
Marts: Fact and Dimension tables optimized for BI.
8️⃣ Visualization (Looker Studio) Connect: Link Looker Studio to your BigQuery Marts.
Join: Link fct_weather_readings and dim_stations on station_id.
Visualize: Create Frequency and Severity tiles to analyze the regional "Climate Gap."