Skip to content

CLAUDE.md

Latest commit

 

History

History
129 lines (98 loc) · 10 KB

CLAUDE.md

File metadata and controls

129 lines (98 loc) · 10 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

Mobility Feed API — a FastAPI service that serves open mobility data (GTFS, GTFS-RT, GBFS feeds) from across the world, backing mobilitydatabase.org.

Three main runtime components:

  • api/ — the FastAPI service (Feeds API + User Service), spec-first via OpenAPI Generator.
  • functions-python/ — Google Cloud Functions (Gen2) for background processing: batch dataset downloads, GBFS/GTFS validation, BigQuery ingestion, notifications, reverse geolocation, etc.
  • PostgreSQL + PostGIS, schema-migrated with Liquibase (liquibase/), geospatial support for location-based feed queries.

infra/ holds Terraform for GCP infra (dev/qa/prod). integration-tests/ is a separate Python suite that hits a live deployed API.

Ground Rules

  • Never commit, push, or comment on PRs unless explicitly asked. Prepare changes and leave them for review.
  • Preserve backward compatibility on the public surface. docs/DatabaseCatalogAPI.yaml and docs/UserServiceAPI.yaml are consumed by real external clients (see README's access-token flow), and the Liquibase-managed DB schema is relied on by both the API and functions-python/. If a change would break an existing endpoint, response shape, or column, flag it explicitly instead of making it silently — don't assume a breaking change is fine just because tests still pass.

Code Generation Split (critical)

This repo is spec-first. Never hand-edit generated code:

  • docs/DatabaseCatalogAPI.yaml → generates api/src/feeds_gen/ via scripts/api-gen.sh (config scripts/gen-config.yaml).
  • docs/UserServiceAPI.yaml → generates api/src/user_service_gen/ via scripts/api-user-service-gen.sh (config scripts/gen-user-service-config.yaml).
  • liquibase/changelog.xml (+ changes/) → generates api/src/shared/database_gen/sqlacodegen_models.py (SQLAlchemy ORM) via scripts/db-gen.sh.
  • liquibase/changelog_user.xml (+ changes_user/) → generates api/src/shared/users_database_gen/ via scripts/db-gen-user.sh.

Workflow when changing the API schema or DB schema:

  1. Edit the OpenAPI yaml (docs/DatabaseCatalogAPI.yaml or docs/UserServiceAPI.yaml) and/or add a Liquibase changeset.
  2. Re-run the matching scripts/*-gen.sh script.
  3. Implement/adjust business logic in feeds/impl/ or user_service/impl/ — never in the _gen folders.

Generated code (feeds_gen/, user_service_gen/, shared/database_gen/, shared/users_database_gen/) is excluded from black/flake8 (see api/pyproject.toml, api/.flake8) and should be treated as read-only.

Common Commands

All commands assume repo root unless noted.

# One-time setup
scripts/setup-openapi-generator.sh
cd api && pip3 install -r requirements.txt -r requirements_dev.txt

# Local Postgres + Liquibase migrations
docker-compose --env-file ./config/.env.local up -d --force-recreate

# Re-init local env after checking out a branch: rebuilds main+test DBs, regenerates
# SQLAlchemy models, FastAPI stubs (Feeds + User Service + Operations API), and
# re-symlinks all functions' shared code. Prefer this over the individual steps below
# whenever the branch you just checked out changed the schema/spec files.
scripts/init-local-folder.sh                  # migrations only, no data
scripts/init-local-folder.sh --populate-db    # also pulls latest CSV from GCS into the main DB

# Regenerate stubs individually (after schema changes)
scripts/api-gen.sh                # Feeds API -> feeds_gen
scripts/api-user-service-gen.sh    # User Service API -> user_service_gen
scripts/db-gen.sh                  # DB schema -> database_gen
scripts/db-gen-user.sh             # User DB schema -> users_database_gen

# Reset local DB and (re)populate from a catalog sources.csv (destructive — wipes local DB)
./scripts/populate-db.sh <path to sources.csv>
./scripts/docker-localdb-rebuild-data.sh --populate-db
./scripts/docker-localdb-rebuild-data.sh --populate-db --populate-test-data   # also loads dataset entities
./scripts/docker-localdb-rebuild-data.sh --use-test-db                       # reset the empty test DB before running tests

# Run the API locally (Swagger UI at http://localhost:8080/docs/)
scripts/api-start.sh

# Lint (flake8 + black) across api/, functions-python/, integration-tests/
scripts/lint-tests.sh
pre-commit install && pre-commit run --all-files   # same check, runs lint-tests.sh as a hook

# Tests
scripts/api-tests.sh                                          # all `api/` tests + coverage (branch coverage must be >= 80%)
scripts/api-tests.sh --test_file <path/to/test_file.py>
scripts/api-tests.sh --folder functions-python                # every function under functions-python/ with a tests/ dir
scripts/api-tests.sh --folder functions-python/<function_name> # a single Cloud Function
scripts/api-tests.sh --html_report                             # also emit HTML coverage under scripts/coverage_reports

# Run/test a single Cloud Function locally
scripts/function-python-setup.sh --function_name <name>   # symlinks shared code (see below) into src/shared and tests/test_shared
scripts/function-python-run.sh --function_name <name>     # installs deps into a venv, runs via functions-framework --debug
scripts/function-python-build.sh --function_name <name>   # zips into .dist/ for deploy

api-tests.sh copies config/.env.local to .env before running, so local env vars match the docker-compose Postgres instance.

Architecture Notes

api/src/ layout

  • feeds/impl/*_api_impl.py — one impl class per resource (FeedsApiImpl, DatasetsApiImpl, SearchApiImpl, MetadataApiImpl, LicensesApiImpl), each subclassing a generated Base*Api from feeds_gen. user_service/impl/*_api_impl.py mirrors this for UsersApiImpl, NotificationsApiImpl, SubscriptionsApiImpl.
  • main.py — a single FastAPI app mounts routers from both feeds_gen and user_service_gen; user-service routes are wrapped with offload_blocking_routes() (utils/route_offload.py) since those impls make blocking DB/Brevo HTTP calls. CORS is configured via raw Starlette middleware, not FastAPI's, due to a known FastAPI CORS bug — don't "simplify" this.
  • shared/ — code shared between the API and the Cloud Functions:
    • database_gen/sqlacodegen_models.py — generated ORM models; Feed is a polymorphic base with GtfsFeed, GbfsFeed, GtfsRTFeed subclasses.
    • db_models/ — hand-written business-logic wrappers per entity (feed_impl.py, gtfs_feed_impl.py, etc.) layered over the generated models.
    • database/ — Database, with_db_session decorator (use this for any DB-touching function), users_database.py.
    • feed_filters/ — query-parameter filter builders per feed type.
    • common/ — error_handling.py, logging_utils.py, config_reader.py, gcp_utils.py, rate_limiter.py, brevo.py (email), entity_type_enum.py.
    • notifications/ — notification event service + Brevo sender.
  • middleware/request_context.py — the de facto auth layer (no file is literally named auth.py): decodes the GCP IAP JWT (x-goog-iap-jwt-assertion header) via decode_jwt/resolve_google_public_keys, exposes extract_user_id, is_user_email_restricted(), and builds a RequestContext. request_context_middleware.py wires this in as ASGI middleware and logs API access.
  • scripts/ (under api/src/, distinct from repo-root scripts/) — DB population scripts (populate_db*.py), gbfs_utils/ (also defines the supported GBFS versions map referenced from the README).

Error handling convention

  • Raise shared.common.error_handling.InternalHTTPException internally; convert to HTTPException via feeds.impl.error_handling.convert_exception().
  • Keep error message strings as Finals in api/src/feeds/impl/error_handling.py so they're reusable and easy to locate.
  • Responses follow {"details": "the error message"}.

functions-python/ — Cloud Functions

Each function directory follows the same shape:

functions-python/<name>/
  src/main.py           # @functions_framework.http (or event-triggered) entry point
  src/shared/           # gitignored symlinks into api/src/shared/*, created by function-python-setup.sh
  tests/                # pytest; tests/test_shared/ is also symlinked (test_utils, etc.)
  function_config.json  # entry_point, runtime, memory, timeout, trigger, include_folders/include_api_folders, secrets, scaling
  requirements.txt / requirements_dev.txt

function_config.json's include_folders (from functions-python/) and include_api_folders (from api/src/shared/) tell function-python-setup.sh what to symlink in — that's how functions share DB models and helpers with the API without a package dependency. Non-deployed shared support code (e.g. helpers/, test_utils/) uses test_config.json instead.

Functions have no direct interdependencies on each other; shared logic lives in functions-python/helpers, functions-python/dataset_service, functions-python/test_utils, or api/src/shared/*.

Gotcha: functions-python/database_gen at the top level is a stray, gitignored, stale copy — it is not the symlink target and can be out of sync with the real source of truth, api/src/shared/database_gen/sqlacodegen_models.py. Don't treat it as canonical.

Database migrations

Liquibase changelogs live in liquibase/ (changelog.xml + changes/ for the main schema, changelog_user.xml + changes_user/ for the user-service schema). docker-compose.yaml runs liquibase/liquibase-test/liquibase-user/liquibase-user-test services to apply them to local Postgres containers before db-gen.sh/db-gen-user.sh regenerate the SQLAlchemy models.

Integration tests

integration-tests/ is a separate suite that runs against a live API URL (not the local unit-test DB). New tests: add a class under endpoints/ inheriting from the base IntegrationTests class, prefix methods with test_. Run via ./scripts/integration-tests.sh -u <API URL> -f <data file> [-c <ClassName1,ClassName2>].