Covid-19 / Retail analysis with a modern data stack

This project builds a lambda data architecture using services and technologies from a modern data stack. Serverless or PaaS services are preferred over IaaS where possible, and UI-based data wrangling tools are favored over hand-written glue code when practical.

The pipeline ingests Iowa liquor retail sales and Covid-19 public datasets from BigQuery, transforms them with dbt into a dimensional model, orchestrates runs with Airflow, tracks lineage with Marquez, and visualizes results in Superset.

Table of contents

• Architecture
• Data model
• Prerequisites
• Quick start
• Running services individually
• Project structure
• Dashboards and notebooks

Architecture

The following diagram shows the local stack running in Docker:

Locally, the stack is available at:

Service	URL	Default credentials
Airflow	http://localhost:8080	airflow / airflow
Marquez	http://localhost:3000	—
Superset	http://localhost:8088	admin / admin

For a cloud deployment, the project targets GCP services together with Airbyte Cloud, the Astronomer platform, Datakin, and dbt Cloud:

Data model

Data is organized in a medallion-style layout inside the covid19retail BigQuery dataset:

Layer	dbt folder / schema	Materialization	Description
Silver	trusted	Incremental	Cleaned extracts from BigQuery public datasets
Gold	app	Table	Dimensional model (star schema) for analytics

Silver (trusted) tables

• iowa_liquor_sales — daily alcoholic beverage sales in Iowa, sourced from bigquery-public-data.iowa_liquor_sales.sales
• covid19_open_data — Covid-19 pandemic indicators for Iowa, sourced from bigquery-public-data.covid19_open_data.covid19_open_data

Gold (app) tables

The gold layer is a snowflake schema with dimension and fact tables:

Type	Tables
Dimensions	dim_dates, dim_counties, dim_stores, dim_vendors, dim_categories, dim_products
Facts	fact_sales, fact_events, fact_reports

dbt models are grouped and tested per table. Airflow DAGs are generated automatically from the dbt manifest:

OpenLineage integration surfaces the pipeline in Marquez:

Prerequisites

• Docker and Docker Compose
• dbt with the BigQuery adapter (dbt-bigquery), if you plan to compile models locally
• A GCP service account JSON key with BigQuery access

Place your service account key in the credentials/ directory. The Docker Compose files expect the file at:

credentials/covid19-retail-090522-773462c5c6bd.json

If you use a different filename, update GOOGLE_APPLICATION_CREDENTIALS in docker-compose.airflow.yaml and docker-compose.superset.yaml.

Quick start

1. Clone the repository and add your GCP credentials as described above.

2. Start the full stack (compiles dbt artifacts, then starts Airflow, Marquez, and Superset):

   ./quickstart.sh

3. Open the service URLs listed in the architecture section.

The default quickstart.sh run compiles dbt manifest.json and catalog.json before starting containers. Airflow DAGs depend on the manifest to build task dependencies.

Running services individually

# Airflow + Marquez only
./quickstart.sh airflow

# Compile dbt artifacts only (no Docker)
./quickstart.sh dbt

# Superset only
./quickstart.sh superset

# Stop all services
./quickstart.sh stop

# Show available commands
./quickstart.sh help

Running dbt manually

cd dbt
dbt run --profiles-dir profiles
dbt test --profiles-dir profiles

Airflow DAGs

Two DAGs are defined in dags/templates/covid19retail.yaml:

• covid19retail-monthly — runs on a @monthly schedule
• covid19retail-yearly — runs on a @yearly schedule

Both execute dbt models and their tests in dependency order using dbt-ol (dbt with OpenLineage).

Project structure

.
├── credentials/          # GCP service account keys (gitignored)
├── dags/                 # Airflow DAGs and YAML templates
├── dashboards/           # Superset dashboard and dataset exports
├── dbt/
│   ├── analyses/         # Ad-hoc analytical SQL
│   ├── models/
│   │   ├── trusted/      # Silver-layer incremental models
│   │   └── app/          # Gold-layer dimensional models
│   ├── profiles/         # dbt connection profiles
│   └── tests/            # Custom data tests
├── docker/               # Custom Airflow and Superset images
├── docs/                 # Architecture diagrams and screenshots
├── notebooks/          # Jupyter notebooks (e.g. sales forecasting)
├── quickstart.sh         # Local environment bootstrap script
├── docker-compose.airflow.yaml
├── docker-compose.marquez.yaml
└── docker-compose.superset.yaml

Dashboards and notebooks

Superset dashboards are pre-loaded on first startup from dashboards/:

The notebooks/forecast.ipynb notebook contains exploratory forecasting work on the gold-layer tables and can be opened in Google Colab.