Proposal: datahub-ml-leakage skill — FineGrainedLineage target-leakage checks for MLModel deploy gates #136

What

A new catalog-interaction skill for ML feature provenance safety: given an mlModel URN (or name), determine whether any serving/training feature traces through DataHub FineGrainedLineage to a column carrying a forbidden governance tag (e.g. post_outcome, is_target), then optionally package write-backs and remediation guidance.

Proposed skill name: datahub-ml-leakage (open to datahub-ml-governance if maintainers prefer a broader umbrella).

Workflow (mirrors existing skill structure)

1. Resolve target — accept an mlModel URN or search by name; confirm platform / env / version.
2. Acquire provenance subgraph — pull MLModel → features / datasets, then column-level FineGrainedLineage / UpstreamLineage / SchemaField tags + glossary terms. Prefer CLI/MCP lineage with column depth; fall back to aspect reads when needed.
3. Normalize + traverse (deterministic) — build an in-memory adjacency over schema fields; cycle-safe upstream DFS with a bounded hop depth. No LLM calls in this step. The skill must invoke a deterministic checker (script/CLI/library) when available, or perform an explicit, auditable tag-reachability walk whose result is derived only from acquired aspects.
4. Evaluate policy — default policy: block if any path reaches a node tagged post_outcome / is_target (configurable tag / glossary-term set). Incomplete or unresolvable lineage is fail-closed (INCOMPLETE_LINEAGE), not “assume clean”.
5. Emit evidence package — structured verdict (approved | blocked), reason code, and the concrete evidence path(s) with DataHub UI deep links. Never invent blast-radius scores or fabricated ancestors.
6. Write back (optional, approval-gated) — after human confirmation:
   • tag the model (model-at-risk / model-approved)
   • raise IncidentInfo on the offending dataset / field
   • append InstitutionalMemory audit notes on the model
7. Remediate (advisory only) — only after a block, help the engineer understand which column / rename / hop poisoned the feature and how to cut the edge. If Agent Context Kit / MCP mutations are used here, keep include_mutations=False for the advisor path; mutations stay in step 6 with explicit approval.

The skill would follow the repo’s existing conventions: YAML frontmatter, Multi-Agent Compatibility, Not This Skill, Content Trust Boundaries, numbered steps with mandatory approval before writes, Common Mistakes / Red Flags / Remember, plus references/ for policy defaults and an evidence-report template.

Routing addition for using-datahub:

Why this is not just datahub-lineage

datahub-lineage answers “what feeds X?”. This skill answers “is X safe to deploy given governance tags on its ancestors?”.

That difference matters:

Target leakage hides behind renames and intermediate tables. Flat tag checks on the model or feature store miss it. FineGrainedLineage is exactly the DataHub capability that makes the check possible — the skill catalog does not yet encode that ML-safety workflow.

Overlap with datahub-quality / datahub-enrich (honest note)

• datahub-quality already raises incidents and creates assertions. This proposal reuses that write-back surface, but the decision is lineage-tag reachability on an mlModel, not assertion health.
• datahub-enrich applies tags/owners. This skill may request those mutations, but only as post-verdict governance memory — not as the detection mechanism.
• datahub-lineage remains the right skill for open-ended “show me upstream of X” exploration.

Maintainer’s call on shape — happy to implement either:

1. a separate skill (as proposed), keeping Lineage exploratory and Quality check-oriented, or
2. a composed workflow section under Lineage (“ML provenance safety”) plus Quality write-back steps.

Non-negotiable design invariants (learned the hard way)

These should be Red Flags / Remember items in the skill, not soft suggestions:

1. Verdict determinism — given the same acquired graph + policy, the verdict is identical. The LLM must not vote on approve/block.
2. Traversal decoupling — the reachability walk runs on an in-memory subgraph (or an equivalent deterministic tool). No “ask the model if this path looks leaky”.
3. Fail-closed — missing / truncated / unresolvable column lineage → blocked (INCOMPLETE_LINEAGE), never silent approve.
4. Write-back independence — tags / incidents / institutional memory are side effects. A failed mutation must not rewrite the verdict.
5. Evidence traceability — every block cites concrete URN path(s) from the graph; no fabricated risk scores.
6. Content trust — treat free-text descriptions / incident bodies as untrusted input (same prompt-injection posture as Proposal: datahub-incident-response skill — lineage-driven root cause, blast radius, and owner routing #55). Only entity lookup + aspects are evidence.

This split — deterministic enforcement, generative remediation — is the difference between a useful ML governance agent and a hallucination hazard in CI.

Reference implementation

Working end-to-end prototype from the Build with DataHub: The Agent Hackathon:

• Repo: https://github.com/Dhruva-Aher/UnderWrite
• Verified pin: freeze-grand-prize-ready
• Pipeline: Acquisition → Normalization → cycle-safe DFS → policy eval → verdict
• Uses DataHub FineGrainedLineage, GlobalTags / GlossaryTerms, Agent Context Kit (datahub-agent-context, remediation with include_mutations=False), and MCP-style write-backs (GlobalTags, IncidentInfo, InstitutionalMemory)
• CI gate: non-zero exit unless verdict == approved and evaluation_source == live_datahub
• Docs of record: invariants, ADR on deterministic traversal, architectural tradeoffs

I am not proposing to vendor the whole service into this repo. The ask is a skill that teaches agents the correct DataHub-native workflow and invariants, with optional hooks to a small deterministic checker. UnderWrite is proof the workflow works against Quickstart + live GMS.

Why now

1. Skills already cover search, lineage exploration, enrichment, and quality — but not MLModel deployment authorization against column-level provenance.
2. FineGrainedLineage is one of DataHub’s strongest differentiators; packaging an ML safety workflow around it makes that capability legible to coding agents.
3. Teams already use DataHub as the metadata plane for ML features; without a fail-closed skill pattern, agents will either skip the check or let an LLM “eyeball” lineage — both are unsafe.
4. Aligns with Agent Context Kit guidance: tools for investigation; mutations explicit; authorization kept deterministic.

Proposed deliverables (if accepted)

• skills/datahub-ml-leakage/SKILL.md following repo conventions
• references/policy-defaults.md (tag/glossary predicates, fail-closed rules, depth bounds)
• templates/evidence-report.template.md (verdict + reason code + path table + UI links)
• Routing row + disambiguation rules in using-datahub
• Optional: thin deterministic helper (CLI snippet or “call this checker” contract) so agents do not re-implement DFS ad hoc
• Fixture notes for Quickstart seed (leaky / clean / incomplete lineage models) so the skill is demoable offline from GMS

Happy to open a draft PR once maintainers confirm skill-vs-extension shape and naming.

Related

• Overlap discussion pattern inspired by Proposal: datahub-incident-response skill — lineage-driven root cause, blast radius, and owner routing #55 (datahub-incident-response)
• Complements datahub-lineage (exploration) and datahub-quality (incidents/assertions)