Open
Conversation
- Use kinds={'dlt', 'iceberg'} for ingestion assets
- Add get_kinds() to CustomDbtTranslator returning {'dbt', 'trino'}
- Replace deprecated compute_kind with kinds parameter
Remove all glucose/github/nightscout specific code from src/phlo: - defs/ingestion/github/ and defs/ingestion/nightscout/ - defs/quality/github.py and defs/quality/nightscout.py - defs/validation/asset_checks.py and pandera_validator.py - schemas/glucose.py and schemas/github.py - plugins/examples.py and quality/examples.py - Example-specific config (glucose_freshness_hours, github_* settings) Also: - Remove duplicate src/cascade folder and src/pyproject.toml - Fix FreshnessPolicy import (dagster.preview -> dagster) - Make FreshnessValidatorResource generic (no hardcoded assets) The core phlo package is now a clean framework. Example code lives in examples/glucose-platform with its own workflows, schemas, and quality checks.
- Remove defs/ingestion/, defs/quality/, defs/jobs.py, defs/jobs/config.yaml - Remove defs/publishing/config.yaml, defs/schedules/schedules.py, defs/sensors/sensors.py - Make dbt translator generic (derive group from folder path or naming convention) - Make schema validator generic (discover tables dynamically) - Add get_quality_checks() registry to quality decorator - Update CLI scaffold to always use workflows/ directory - Update job/publishing factories to return empty list if config missing The framework is now fully generic with no hardcoded example references.
- Remove all GitHub-related dbt models (bronze, silver, gold, marts) - Update sources.yml to only include glucose_entries source - Update yml files to use data_tests (not deprecated tests) - Fix mrt_glucose_readings.yml test configuration The example now only contains Nightscout/glucose assets as intended.
Framework changes: - Add _discover_publishing_assets() to auto-detect dbt marts models - Auto-generate publish_marts_to_postgres asset from dbt manifest - Fix TrinoResource to not use invalid session property Example changes: - Remove GitHub test files from glucose-platform - Update dbt_project.yml (marts stay in Iceberg, published to Postgres) Pipeline now: Ingestion -> Bronze -> Silver -> Gold/Marts (Iceberg) -> Postgres
Nessie Branching: - Add iceberg_dev.properties catalog for dev branch - Update iceberg.properties with explicit main branch prefix - Update TrinoResource with branch parameter (main/dev) - Document Write-Audit-Publish pattern in trino.py Auto dbt Compile: - Add _run_dbt_compile() to services.py - Runs dbt deps + compile after services start - Auto-restarts Dagster to pick up new manifest Architecture: - Feature branch: ingestion -> transforms -> validation - Main branch: validated data -> publishing to Postgres
- Add NessieResource for branch management (list, create, delete, merge) - Add _init_nessie_branches() to auto-create dev branch on startup - Supports Write-Audit-Publish pattern: - Feature branch: ingestion -> transforms -> validation - Merge to main after validation passes - Publishing from main to Postgres for BI
Sensor fixes: - auto_promotion_sensor now returns properly after successful promotion - branch_cleanup_sensor uses config.branch_cleanup_enabled flag - Added type annotation to promote_branch_to_main NessieResource improvements: - Added 30s timeout to all HTTP requests - Added optimistic locking (expectedHash) to delete_branch - Use params dict instead of query string in URL Config: - Added branch_cleanup_enabled (default False for safety) Code quality: - Applied ruff format to all src/phlo files - Fixed E402 import ordering in definitions.py
Structure changes: - Remove duplicate README.md (use index.md as entry point) - Move CLI_GUIDE.md to guides/cli.md - Move POSTGREST_DEPLOYMENT.md to setup/postgrest.md - Remove audit/ directory (internal analysis docs) - Remove planning docs (PHASE_3_FEATURES.md, prd-postgrest-migration.md) Content fixes: - Replace cascade_ingestion -> phlo_ingestion across all docs - Replace yourusername -> iamgp in clone URLs - Fix broken links to moved files - Update index.md with new file locations Clean structure: - blog/ - 12-part tutorial series - getting-started/ - quickstart, beginners guide - guides/ - cli, workflow, dbt, dagster, data modeling - setup/ - openmetadata, postgrest, hasura, observability - reference/ - architecture, api, configuration, errors - operations/ - troubleshooting, best practices, testing - errors/ - error code documentation
Part 4 (Nessie): - Replace invalid session property (iceberg.nessie_reference_name) with catalog-based branching - Add Write-Audit-Publish (WAP) pattern documentation - Document automatic sensors (branch_creation, auto_promotion, branch_cleanup) - Update manual branch operations with correct Nessie v1 API calls - Show NessieResource Python API Part 6 (dbt): - Remove session_properties from profiles.yml - Show catalog-based branching (iceberg_dev vs iceberg) - Remove invalid pre_hook for branch switching Part 8 (Real-world example): - Replace manual publish_glucose_marts with auto-publishing explanation - Document how Phlo auto-discovers marts and generates publishing assets - Simplify: just create dbt models in marts schema, Phlo handles the rest
- Rewrite README.md as concise intro pointing to docs/ - Fix broken links to deleted files - Move PLATFORM_REVIEW.md to specs/platform-review.md (gitignored)
- Add MockIcebergCatalog with DuckDB backend (Task 1.1) - Add mock_dlt_source helper for API mocking (Task 1.2) - Add test_asset_execution helper (Task 1.3) - Add MockTrinoResource with DuckDB backend (Task 1.4) - Add pytest fixtures for common test scenarios (Task 1.5) - Add LocalTestMode and FixtureRecorder (Task 1.6) - Add conftest_template.py as separate module per spec - Update CLI to set PHLO_TEST_LOCAL env var for --local flag - Add comprehensive test suite (29 tests)
Task 2.1 - phlo validate-workflow: - Validate @phlo_ingestion decorator parameters - Check unique_key, cron syntax, table_name, group - Support directory scanning for batch validation - Rich-formatted error messages with suggestions Task 2.2 - phlo status: - Show asset materialization status and freshness - Monitor service health (Dagster, Trino, MinIO, Nessie) - Color-coded status indicators (green/yellow/red) - Filtering by --assets, --services, --group, --stale, --json Task 2.3 - phlo backfill: - Date range backfill with --start-date/--end-date - Explicit partitions with --partitions - Parallel execution with --parallel N - Resume capability with --resume - Dry-run mode and progress tracking Task 2.4 - phlo logs: - Filter by --asset, --job, --level, --since, --run-id - Tail mode with --follow - JSON output for scripting - Fast retrieval (< 1 second for 100 logs) Tests: 142 passing across all CLI commands
Task 3.1 - Complete @phlo_quality Decorator: - Full Dagster asset check integration - All check types: NullCheck, RangeCheck, FreshnessCheck, UniqueCheck, CountCheck, SchemaCheck, CustomSQLCheck - Threshold-based results with warn_threshold support - Rich metadata generation for Dagster UI - Auto-discovery and registration of quality checks - Support for Trino and DuckDB backends API: - Use `@phlo.quality(...)` after `import phlo` - Import checks from `phlo.quality`: NullCheck, RangeCheck, etc. Tests: 35 passing covering all check types and decorator options
…and 010 - Add phlo schema commands (list, show, diff, validate) for Pandera schema management - Add phlo catalog commands (tables, describe, history) for Iceberg catalog queries - Add phlo branch commands (list, create, delete, merge, diff) for Nessie branching - Add phlo contract commands (list, show, validate, check) for data contract management - Implement data contracts module with schema evolution tracking and SLA validation - Add consumer notification system for contract changes (Slack, Email, PagerDuty) - Include example contracts in glucose-platform examples directory - Add comprehensive tests for all new CLI commands and contract system
…c 005 - Add MetricsCollector with Prometheus, Postgres, and Iceberg integration - Add 30-second intelligent caching for metrics - Implement phlo metrics commands: summary, asset, export (JSON/CSV) - Add AlertManager with pluggable destination architecture - Implement alert destinations: Slack, PagerDuty, Email - Add alert deduplication to prevent storms - Implement phlo alerts commands: test, list, status - Add LineageGraph for asset dependency tracking - Implement graph traversal: get_upstream, get_downstream, get_impact - Add lineage export: Graphviz DOT, Mermaid, JSON - Implement phlo lineage commands: show, export, impact, status - Add alerting configuration to Settings (config.py) - Add 21 comprehensive tests for all observability features
- Add OpenMetadataClient for REST API access (authentication, CRUD, lineage) - Add NessieTableScanner for Iceberg table metadata extraction - Add DbtManifestParser for dbt manifest/catalog parsing and sync - Add LineageExtractor for building and publishing lineage graphs - Add QualityCheckMapper/Publisher for syncing quality checks to OpenMetadata - Add Dagster sensors for automatic metadata, lineage, and quality sync - Add config settings for OpenMetadata, dbt manifest paths, Nessie - Add 97 comprehensive tests for all catalog module components
- Add PostgREST view generator from dbt models (parse manifest, generate SQL, permissions) - Add phlo api postgrest generate-views command with --apply/--diff/--output options - Add HasuraClient for Metadata API v1 (track, untrack, permissions, relationships) - Add HasuraTableTracker for auto-discovery and table tracking - Add HasuraPermissionManager for permission config sync from YAML/JSON - Add HasuraMetadataSync for metadata export/import with diff support - Add RoleHierarchy for permission inheritance (admin -> analyst -> anon) - Add phlo api hasura commands: track, relationships, permissions, export, apply, status - Add 35 comprehensive tests for PostgREST and Hasura functionality
…r spec 006 - Extend PluginRegistry with metadata retrieval and validation - Implement entry point discovery with whitelist/blacklist support - Add phlo plugin commands: list, info, check, create (scaffolding) - Add plugin config settings (enabled, auto_discover, whitelist, blacklist) - Create example plugin package with JSONPlaceholder source, threshold quality check, uppercase transform - Add 21 tests for plugin registration, discovery, validation, and integration - Update AGENTS.md with plugin CLI documentation
- GitHub API ingestion (events, repos, profile) - Pandera validation schemas - Quality checks with NullCheck and RangeCheck - dbt transformations (bronze, silver, gold, marts) - PostgreSQL API views - Dagster orchestration with partitioned assets
Add comprehensive infrastructure configuration architecture that centralizes service topology in phlo.yaml. The infrastructure section is auto-generated by phlo init and phlo services init, providing a complete map of all services including container names, hostnames, and network configuration. Key features: - Pydantic schema for validation with ServiceConfig and InfrastructureConfig - Configuration loader with caching and fallback to defaults - Config-first container discovery with dynamic fallback - Three new CLI commands: config show, config validate, config upgrade - Full backward compatibility - infrastructure section is optional Configuration precedence: 1. phlo.yaml infrastructure section (explicit config) 2. Dynamic discovery via docker ps 3. Default configuration Ports remain in .env following 12-factor app methodology. The infrastructure section defines service topology while .env defines secrets and values.
Catch ValueError exceptions when casting arrow tables to target schema. This fixes issues with nullable/non-nullable field mismatches during schema evolution. Fixes casting errors like: 'Casting field 'type' with null values to non-nullable'
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 I have created a release beep boop
0.6.1 (2026-03-01)
Bug Fixes
This PR was generated with Release Please. See documentation.