feat(ingestion/hive-metastore): add upstream lineage to hive-metastore

[acrylJonny]

Add Storage Lineage to Hive/Hive Metastore + Code Refactoring

Summary

Adds storage lineage support to Hive and Hive Metastore connectors, enabling lineage tracking between Hive tables and their underlying storage locations (S3, Azure, GCS, HDFS, DBFS). Also refactors Hive sources into a clean directory structure.

Key Changes

Storage Lineage (Opt-in Feature)

New configuration options (disabled by default):

• emit_storage_lineage: Enable storage lineage extraction
• hive_storage_lineage_direction: Set direction (upstream or downstream)
• include_column_lineage: Enable column-level lineage
• storage_platform_instance: Platform instance for storage URNs

Supported platforms: S3, Azure (ADLS/ABFS), GCS, HDFS, DBFS, local files

Code Refactoring

• Moved hive.py → hive/hive_source.py
• Moved hive_metastore.py → hive/hive_metastore_source.py
• Created hive/storage_lineage.py with shared logic
• Extracted HiveStorageLineageConfigMixin to eliminate duplication
• Updated setup.py entry points to use fully qualified paths

Code Quality Improvements

• Converted HiveStorageLineageConfig to Pydantic model
• Created LineageDirection and StoragePlatform StrEnums for type safety
• Fixed unsafe exception handling in get_db_schema (now raises ValueError for invalid input)
• Improved error reporting in get_workunits_internal (specific exceptions + proper logging)
• Removed 50+ redundant comments

PR Checks

• The PR conforms to DataHub's Contributing Guideline (particularly PR Title Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[codecov]

Codecov Report

❌ Patch coverage is 87.08861% with 51 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...tahub/ingestion/source/sql/hive/storage_lineage.py	87.79%	21 Missing ⚠️
...c/datahub/ingestion/source/sql/hive/hive_source.py	88.57%	20 Missing ⚠️
...ingestion/source/sql/hive/hive_metastore_source.py	77.27%	10 Missing ⚠️

📢 Thoughts on this report? Let us know!

[sgomezvillamor]

this exception swallowing pattern, is that what we usually do in other sources when we fail emission?

[sgomezvillamor]

codecoverage report shows little coverage in failure/exception scenarios, we could iimprove a little bit there, mainly in hive_source.py

also, concerning the lack of coverage for dbapi_get_columns_patched

[acrylJonny]

codecoverage report shows little coverage in failure/exception scenarios, we could iimprove a little bit there, mainly in hive_source.py

also, concerning the lack of coverage for dbapi_get_columns_patched

I'll add these today so that we can get this signed off.