zscan

Zero-scan data quality observability via Apache Iceberg table metadata.

What is zscan?

zscan is a zero-scan data quality tool that monitors data quality using only metadata from Apache Iceberg tables — without ever reading actual Parquet data files.

Traditional data quality tools scan entire datasets (GB–TB) to validate data. zscan instead reads Iceberg manifest statistics (record counts, null counts, value bounds) that are already computed during writes and stored in lightweight metadata files (KB–MB).

Key Benefits

Benefit	Traditional DQ	zscan
Speed	Minutes to hours	Milliseconds
Cost	High (full table scans)	Near-zero (metadata only)
Latency	2–24 hours	Real-time (< 20 min)
Coverage	~100% of rules	~60% exact, ~90% with extensions

How It Works

Architecture Overview

graph TB
    subgraph "Write Path"
        App[Application] -->|Append| Table[Iceberg Table]
        Table -->|Generates| Manifest[Manifest Files]
        Manifest -->|Contains| Stats[Column Statistics]
    end

    subgraph "Metadata Layer"
        Stats -->|Record Count| Metadata[Metadata Index]
        Stats -->|Null Counts| Metadata
        Stats -->|Value Bounds| Metadata
        Stats -->|File Sizes| Metadata
    end

    subgraph "zscan (Zero-Scan DQ)"
        Metadata -->|Read| Extractor[MetadataExtractor]
        Extractor -->|Evaluate| Rules[Quality Rules]
        Rules -->|Generate| Report[Quality Report]
    end

    subgraph "Traditional DQ (Avoided)"
        Data[Parquet Data Files] -.->|Full Scan| Scanner[Data Scanner]
        Scanner -.->|Expensive| Traditional[Traditional DQ]
    end

    style zscan fill:#e1f5fe
    style Metadata fill:#f3e5f5
    style Stats fill:#fff3e0
    style Traditional fill:#ffebee

Zero-Scan vs Full-Scan

sequenceDiagram
    participant User
    participant zscan
    participant Metadata as Iceberg Metadata
    participant Data as Parquet Data (NOT READ)

    User->>zscan: Run quality checks
    zscan->>Metadata: Read manifest statistics
    Note over Metadata: ~KB-MB size
    Metadata-->>zscan: Return column stats
    zscan->>zscan: Evaluate rules
    zscan-->>User: Quality report

    Note over Data: NOT accessed

Three-Tier Rule Coverage

graph LR
    subgraph "Tier 1: Exact (~60%)"
        T1[Row Counts]
        T2[Null Counts]
        T3[Value Bounds]
        T4[NaN Counts]
    end

    subgraph "Tier 2: With Sketches (~90%)"
        T5[Uniqueness - Theta]
        T6[Quantiles - KLL]
        T7[Distinctness]
    end

    subgraph "Tier 3: Full Scan (~100%)"
        T8[Cross-Column]
        T9[Cross-Row]
        T10[Full Distributions]
    end

    T1 --> T5
    T5 --> T8

    style Tier 1 fill:#c8e6c9
    style Tier 2 fill:#fff9c4
    style Tier 3 fill:#ffcdd2

Tier	Source	What It Answers	Coverage
Tier 1	Manifest stats only	Row counts, nulls, bounds, NaNs	~60%
Tier 2	Stats + Puffin sketches	Uniqueness, quantiles, distinctness	~90%
Tier 3	Full data scan	Cross-column, correlations	~100%

Installation

Prerequisites

Python 3.10+
uv (recommended) or pip

Distribution is GitHub Releases only (no PyPI). Each tag v*.*.* publishes a .whl + .tar.gz as release assets.

From GitHub Release (Recommended)

# Latest release (example v0.0.1) — via pip
pip install https://github.com/sendalcurian/zscan/releases/download/v0.0.1/zscan-0.0.1-py3-none-any.whl

# With uv
uv pip install https://github.com/sendalcurian/zscan/releases/download/v0.0.1/zscan-0.0.1-py3-none-any.whl

# Specific version
pip install https://github.com/sendalcurian/zscan/releases/download/v0.0.2/zscan-0.0.2-py3-none-any.whl

# Via git tag (pluggable — always tracks source)
pip install git+https://github.com/sendalcurian/zscan@v0.0.1
uv pip install git+https://github.com/sendalcurian/zscan@v0.0.1

Pluggable use: pin another project to zscan @ https://.../zscan-0.0.1-py3-none-any.whl in its pyproject.toml / requirements.txt without needing PyPI.

From Source (Development)

# Clone the repository
git clone https://github.com/sendalcurian/zscan.git
cd zscan

# Install with uv
uv sync

# Or install with dev dependencies
uv sync --extra dev

# Or with pip (editable)
pip install -e .
pip install -e ".[dev]"

Quick Start

1. Create a Sample Iceberg Table

import pyarrow as pa
from pyiceberg.catalog import SqlCatalog
from pathlib import Path

# Setup catalog
warehouse = Path("./warehouse")
warehouse.mkdir(exist_ok=True)

catalog = SqlCatalog(
    "default",
    **{
        "uri": f"sqlite:///{warehouse / 'catalog.db'}",
        "warehouse": str(warehouse),
    },
)

# Create table
catalog.create_namespace_if_not_exists("mydb")
schema = pa.schema([
    pa.field("id", pa.int64()),
    pa.field("name", pa.string()),
    pa.field("value", pa.float64()),
])

table = catalog.create_table_if_not_exists("mydb.data", schema=schema)

# Insert data
data = pa.table({
    "id": [1, 2, 3],
    "name": ["Alice", "Bob", None],
    "value": [10.0, 20.0, 30.0],
})
table.append(data)

2. Run Quality Checks (Python API)

from zscan import MetadataExtractor, QualityChecker
from zscan.core.rules import NullRateRule, RangeViolationRule

# Extract metadata (zero-scan!)
extractor = MetadataExtractor("./warehouse")
metadata = extractor.get_table_metadata("mydb.data")

# Configure rules
checker = QualityChecker(extractor)
checker.add_rule(NullRateRule(default_threshold=0.1))
checker.add_rule(RangeViolationRule(column_bounds={"value": (0, None)}))

# Run checks
report = checker.run_checks("mydb.data", metadata=metadata)

if report.passed:
    print("All checks passed.")
else:
    print(f"{report.total_violations} violations found")
    for result in report.failed_rules:
        for v in result.violations:
            print(f"   - {v.message}")

3. Run Quality Checks (CLI)

# Run checks against a table
zscan check mydb.data --warehouse ./warehouse

# Inspect table metadata
zscan inspect mydb.data --warehouse ./warehouse

# Compare snapshots
zscan diff mydb.data 1234567890 9876543210 --warehouse ./warehouse

# JSON output
zscan check mydb.data --warehouse ./warehouse --json

Project Structure

zscan/
├── src/
│   └── zscan/
│       ├── __init__.py          # Package exports
│       ├── cli.py               # Typer CLI application
│       ├── core/
│       │   ├── __init__.py
│       │   ├── metadata.py      # MetadataExtractor class
│       │   ├── checks.py        # QualityChecker orchestration
│       │   └── rules.py         # Rule definitions
│       ├── models/
│       │   ├── __init__.py
│       │   └── schemas.py       # Configuration schemas
│       └── utils/
│           ├── __init__.py
│           └── logging.py       # Logging configuration
├── tests/
│   ├── conftest.py              # Test fixtures
│   ├── test_metadata.py         # Metadata tests
│   └── test_checks.py           # Quality check tests
├── examples/
│   └── demo.py                  # Full working demo
├── pyproject.toml               # Project configuration
├── README.md                    # This file
├── LICENSE                      # MIT License
└── plan.md                      # Research plan

API Reference

MetadataExtractor

from zscan import MetadataExtractor

extractor = MetadataExtractor("/path/to/warehouse")

# Get complete table metadata
metadata = extractor.get_table_metadata("db.table")

# Compare snapshots
diff = extractor.get_snapshot_diff("db.table", snap_id_1, snap_id_2)

# Query with DuckDB (zero-scan)
result = extractor.query_with_duckdb(
    "db.table",
    "SELECT * FROM iceberg_metadata('{table}')"
)

QualityChecker

from zscan import QualityChecker
from zscan.core.rules import *

checker = QualityChecker(extractor)

# Add rules
checker.add_rule(RowCountDriftRule(threshold_pct=20.0))
checker.add_rule(NullRateRule(default_threshold=0.1))
checker.add_rule(RangeViolationRule(column_bounds={"age": (0, 150)}))
checker.add_rule(FileCountAnomalyRule())

# Run checks
report = checker.run_checks("db.table")

Built-in Rules

Rule	Description	Default Threshold
`RowCountDriftRule`	Detects significant row count changes	20%
`NullRateRule`	Checks null rate thresholds per column	10%
`RangeViolationRule`	Validates value bounds	Per-column
`FileCountAnomalyRule`	Detects file count spikes/drops	100%/50%

Custom Rules

from zscan.core.rules import Rule, RuleResult, CheckStatus

class MyCustomRule(Rule):
    def __init__(self):
        super().__init__(
            name="my_rule",
            description="Custom quality check",
        )

    def evaluate(self, metadata):
        # Your logic here
        return RuleResult(
            rule_name=self.name,
            status=CheckStatus.PASSED,
        )

Metadata Available (Zero-Scan)

Each Iceberg data file exposes these statistics via manifests:

Statistic	Description	Use Case
`record_count`	Number of records	Row count drift detection
`file_size_in_bytes`	File size	Storage anomaly detection
`column_sizes`	Size per column	Column growth monitoring
`value_counts`	Non-null value count	Completeness checks
`null_value_counts`	Null count	Null rate monitoring
`nan_value_counts`	NaN count	Float data quality
`lower_bounds`	Min values	Range validation
`upper_bounds`	Max values	Range validation

Development

Setup Development Environment

# Install with dev dependencies
uv sync --extra dev

# Run tests
uv run pytest

# Run tests with coverage
uv run pytest --cov=zscan --cov-report=html

# Lint code
uv run ruff check .

# Format code
uv run ruff format .

# Type checking
uv run mypy src/

Pre-commit Hooks

# Install pre-commit hooks
uv run pre-commit install

# Run all hooks
uv run pre-commit run --all-files

Demo Data Conditions

The demo (examples/demo.py) creates a controlled dataset with injected quality issues to demonstrate zero-scan detection capabilities.

Schema

Column	Type	Nullable	Description
`id`	`int64`	No	Unique identifier (1–15)
`name`	`string`	Yes	Person names
`value`	`float64`	Yes	Numeric measurements
`category`	`string`	Yes	Category labels (A/B)

Snapshot History

The demo creates 3 snapshots with progressively worse data quality:

graph LR
    subgraph "Snapshot 1 (Clean)"
        S1[5 rows]
        S1C[0% nulls]
        S1R[All values in range]
    end

    subgraph "Snapshot 2 (Nulls Injected)"
        S2[10 rows]
        S2C[~12% nulls]
        S2R[All values in range]
    end

    subgraph "Snapshot 3 (Outliers Injected)"
        S3[15 rows]
        S3C[~8% nulls]
        S3R[value=-5.0 out of range]
    end

    S1 -->|append| S2
    S2 -->|append| S3

    style S1 fill:#c8e6c9
    style S2 fill:#fff9c4
    style S3 fill:#ffcdd2

Snapshot Details

Snapshot 1: Clean Data

id	name	value	category
1	Alice	10.0	A
2	Bob	20.0	B
3	Charlie	30.0	A
4	David	40.0	B
5	Eve	50.0	A

Conditions: 5 rows, 0% nulls, all values positive.

Snapshot 2: Null Injection

id	name	value	category
6	Frank	60.0	B
7	null	null	A
8	Hank	80.0	null
9	null	null	B
10	Jack	100.0	null

Conditions: 5 new rows (10 total), 4 nulls across 3 columns (~12% null rate per column).

Injected Issues:

name: 2 nulls (rows 7, 9)
value: 2 nulls (rows 7, 9)
category: 2 nulls (rows 8, 10)

Snapshot 3: Outlier Injection

id	name	value	category
11	Kate	-5.0 [OUTLIER]	A
12	Liam	120.0	B
13	Mia	130.0	A
14	Noah	140.0	B
15	Olivia	150.0	A

Conditions: 5 new rows (15 total), 1 out-of-range value.

Injected Issues:

value: -5.0 violates expected range [0, ∞)

Quality Rule Thresholds

Rule	Threshold	Rationale
`RowCountDriftRule`	20%	Detects unexpected drops/spikes in row counts
`NullRateRule`	10%	Flags columns with excessive nulls
`RangeViolationRule`	`value >= 0`	Ensures non-negative values
`FileCountAnomalyRule`	+100% / -50%	Detects file count anomalies

Expected Detection Results

Rule	Snapshot 1→2	Snapshot 2→3	Explanation
`row_count_drift`	DETECTED (100% change)	DETECTED (50% change)	Row count doubled, then grew 50%
`null_rate_check`	DETECTED (11.76% > 10%)	DETECTED (11.76% > 10%)	Nulls persist across snapshots
`range_violation`	— No violation	SKIPPED (binary bounds)	-5.0 detected but bounds are binary-encoded
`file_count_anomaly`	PASSED	PASSED	No file count anomalies

Cumulative Statistics

After all 3 snapshots:

Metric	Value
Total rows	15
Null rate (`name`)	2/15 = 13.3%
Null rate (`value`)	2/15 = 13.3%
Null rate (`category`)	2/15 = 13.3%
Out-of-range values	1 (`value = -5.0`)
Data files	3 (1 per snapshot)

Customizing Demo Conditions

To modify the demo data, edit examples/demo.py:

# Change null rate threshold
checker.add_rule(NullRateRule(default_threshold=0.05))  # 5% instead of 10%

# Change row count drift threshold  
checker.add_rule(RowCountDriftRule(threshold_pct=10.0))  # 10% instead of 20%

# Add custom range bounds
checker.add_rule(RangeViolationRule(
    column_bounds={
        "value": (0, 200),      # 0 <= value <= 200
        "id": (1, 1000),        # 1 <= id <= 1000
    }
))

Examples

Run the Demo

# Full working example with sample data
uv run python examples/demo.py

The demo will:

Create a sample Iceberg table with quality issues
Extract metadata (zero-scan)
Run quality checks
Display a formatted report

References

Zero-Scan Data Quality (arXiv:2605.30308, SIGMOD 2026) — LinkedIn's production deployment
Apache Iceberg Specification — Manifest and metadata format
PyIceberg Documentation — Python Iceberg library
DuckDB Iceberg Extension — SQL-based metadata queries

License

MIT License - see LICENSE for details.

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

Acknowledgments

Built with:

PyIceberg - Apache Iceberg Python library
DuckDB - In-process SQL OLAP database
Typer - CLI framework
Rich - Terminal formatting
uv - Python package manager
Ruff - Python linter/formatter

FilesExpand file tree

README.md

Latest commit

History