[m-abulazm]

Changes

What does this PR do?

Fixes three problems in the SQL Server profiler extract:

• table_sizes.RowCount was multiplied by the number of indexes. A table with a clustered key and 9 nonclustered indexes reported 705,100 rows instead of 70,510.
• System objects appeared in the inventory: change data capture and change tracking tables, their columns and routines, and sys.database_firewall_rules on Azure SQL Database.
• proc_stats.OBJECT_NAME was empty for most rows.

Relevant implementation details

• table_sizes counts rows from the heap or clustered index only (index_id 0 or 1). Page and size columns were already correct.
• tables, views, columns, routines and table_sizes exclude is_ms_shipped objects and the sys and cdc schemas. The cdc schema is excluded too because the per-table functions change data capture generates aren't flagged as system objects.
• proc_stats drops rows from the hidden Resource database (database_id 32767). These are SQL Server's own procedures, whose names never resolve.
• The on-premises grants now say where each one goes: VIEW SERVER STATE at server level, then a user with VIEW DEFINITION in each profiled database. The redundant INFORMATION_SCHEMA grant is removed from both the on-premises and Azure SQL Database recipes.
• All changes apply to both the single-database and multi-database queries.

Caveats/things to watch out for when reviewing:

• Extracts get fewer tables, columns and routines rows on databases that use change data capture or change tracking. The removed rows are all system objects.
• Procedure names still need the grants above; the query can't resolve them without permission.
• Not fixed here: table_sizes groups by schema but doesn't output it, so same-named tables in different schemas look like duplicates. Fixing that changes the table definition, so it's left for a separate PR.

Linked issues

None.

Functionality

• added relevant user documentation
• modified existing command: databricks labs lakebridge execute-database-profiler

Tests

• manually tested
• added unit tests
• added integration tests

Manually tested on SQL Server 2022, including a case-sensitive database with change data capture and change tracking enabled, and on SQL Server 2017. A login with only the documented grants returns the same rows as sa for every profiler query, single and multi database, on both versions. Every row the fix removes is a system object, and the row counts match the actual tables. New static checks in test_mssql_sql_rules.py fail on main and pass here. make test passes.

This pull request and its description were written by Isaac.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 71.40%. Comparing base (b919dec) to head (2157732).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2686   +/-   ##
=======================================
  Coverage   71.40%   71.40%           
=======================================
  Files         109      109           
  Lines       10024    10024           
  Branches     1093     1093           
=======================================
  Hits         7158     7158           
  Misses       2648     2648           
  Partials      218      218           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ 179/179 passed, 8 flaky, 2 skipped, 1h31m2s total

Flaky tests:

• 🤪 test_installs_and_runs_local_bladebridge (13.207s)
• 🤪 test_installs_and_runs_pypi_bladebridge (21.044s)
• 🤪 test_transpiles_informatica_to_sparksql (22.664s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (21.695s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[True] (4.175s)
• 🤪 test_transpile_teradata_sql_non_interactive[True] (6.157s)
• 🤪 test_transpile_teradata_sql_non_interactive[False] (26.164s)
• 🤪 test_transpile_teradata_sql (26.319s)

_{Running from acceptance #5539}

[sundarshankar89]

LGTM