-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(ingest/unity): add row count in table profile of delta tables #12480
base: master
Are you sure you want to change the base?
Conversation
Codecov ReportAttention: Patch coverage is
❌ Your patch check has failed because the patch coverage (7.14%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.
... and 4 files with indirect coverage changes Continue to review full report in Codecov by Sentry.
|
metadata-ingestion/src/datahub/ingestion/source/ge_profiling_config.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/source/ge_profiling_config.py
Outdated
Show resolved
Hide resolved
…onfig.py Co-authored-by: Sergio Gómez Villamor <[email protected]>
…onfig.py Co-authored-by: Sergio Gómez Villamor <[email protected]>
# Presence of size_in_bytes confirms this is DELTA table and that we have | ||
# SELECT permission on this table. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if we may transform this comment in actual code. Something like:
is_delta_table = table.size_in_bytes is not None
and then use the new flag accordingly to call delta-specific code
WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In fact, we can add is_delta_table
boolean in UnityCatalogSQLGenericTable based on custom property data_source_format
in original table.
Or we can add is_delta_table_with_select_enabled: table.size_in_bytes is not None
as suggested. Maybe I can do both.
Checklist