feat(ingestion/grafana): Add datasets and charts to dashboards with lineage and tags. Lineage back to source #12417

acrylJonny · 2025-01-21T15:20:10Z

Adding functionality to the existing Grafana connector. The existing connector supports Dashboard identification only Changed implement the following:

Charts
- Input datasets
- Input Columns
Datasets
- Dataset Schema
- Subtyping with definition where SQL
Lineage:
- between upstream sources and datasets
- between datasets and charts
- between charts and workbooks
- Supports SQL lineage reconciliation using datahub.sql_parsing.sqlglot_lineage falling back to sqlparse
Tag extraction with propagation back from dashboard through to chart and dataset
Owner extraction

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

codecov · 2025-01-21T15:58:54Z

Codecov Report

Attention: Patch coverage is 94.57831% with 27 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...on/src/datahub/ingestion/source/grafana/lineage.py	83.78%	12 Missing ⚠️
...datahub/ingestion/source/grafana/grafana_source.py	92.38%	8 Missing ⚠️
...tion/src/datahub/ingestion/source/grafana/types.py	61.53%	5 Missing ⚠️
...rc/datahub/ingestion/source/grafana/grafana_api.py	97.05%	2 Missing ⚠️

Files with missing lines	Coverage Δ
...rc/datahub/ingestion/source/grafana/field_utils.py	`100.00% <100.00%> (ø)`
...datahub/ingestion/source/grafana/grafana_config.py	`100.00% <100.00%> (ø)`
...ion/src/datahub/ingestion/source/grafana/models.py	`100.00% <100.00%> (ø)`
...ion/src/datahub/ingestion/source/grafana/report.py	`100.00% <100.00%> (ø)`
.../src/datahub/ingestion/source/grafana/snapshots.py	`100.00% <100.00%> (ø)`
...rc/datahub/ingestion/source/grafana/grafana_api.py	`97.05% <97.05%> (ø)`
...tion/src/datahub/ingestion/source/grafana/types.py	`61.53% <61.53%> (ø)`
...datahub/ingestion/source/grafana/grafana_source.py	`93.49% <92.38%> (-0.85%)`	⬇️
...on/src/datahub/ingestion/source/grafana/lineage.py	`83.78% <83.78%> (ø)`

... and 1 file with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a4f8d17...f0f88c6. Read the comment docs.

metadata-ingestion/src/datahub/ingestion/source/grafana/models.py

metadata-ingestion/src/datahub/ingestion/source/grafana/field_utils.py

sgomezvillamor · 2025-01-23T09:15:32Z

metadata-ingestion/src/datahub/ingestion/source/grafana/field_utils.py

+
+
+def _deduplicate_fields(fields: List[SchemaFieldClass]) -> List[SchemaFieldClass]:
+    """Remove duplicate fields based on fieldPath while preserving order."""


I found this in the codebase — does it fit the purpose?

datahub/metadata-ingestion/src/datahub/utilities/dedup_list.py

Line 6 in 8eda51e

def deduplicate_list(iterable: Iterable[_T]) -> List[_T]:

gave it a try and unfortunately it doesn't do the business. I gave it a go and it was causing hashing issues.

metadata-ingestion/src/datahub/ingestion/source/grafana/field_utils.py

sgomezvillamor

Overall looks really good!
Reviewing lengthy PRs is always challenging.
Since the code is very-well-structured (kudos!), this would be a great opportunity to add proper unit testing.

sgomezvillamor · 2025-01-23T09:22:22Z

metadata-ingestion/src/datahub/ingestion/source/grafana/field_utils.py

+            for col in columns
+        ]
+    except (IndexError, ValueError, StopIteration):
+        return []


should we log some warning here?

I've put this in logger but were you looking for this to be in the report?

logger should be fine
given we are catching three types of exceptions, adding the exception to the log trace may help to discriminate the issue

logger.warning(f"Failed to parse SQL {target.get('rawSql')}", ex)

metadata-ingestion/src/datahub/ingestion/source/grafana/field_utils.py

sgomezvillamor · 2025-01-23T09:30:41Z

metadata-ingestion/src/datahub/ingestion/source/grafana/grafana_api.py

+        """Fetch all folders from Grafana with pagination."""
+        folders: List[Folder] = []
+        page = 1
+        per_page = 100


Based on the experience with other sources, it's nice to have this as a config parameter with a default value (page_size).

I was thinking to just specify the page size to reduce the overhead of anyone needing to think about changing this, but happy to change.

metadata-ingestion/src/datahub/ingestion/source/grafana/grafana_api.py

sgomezvillamor · 2025-01-23T09:40:13Z

metadata-ingestion/src/datahub/ingestion/source/grafana/grafana_config.py

+    platform_instance: Optional[str] = Field(
+        default=None, description="Platform instance"
+    )
+    env: str = Field(default="PROD", description="Environment")


These two may be removed if using PlatformInstanceConfigMixin and EnvConfigMixin in GrafanaSourceConfig, are they really needed at PlatformConnectionConfig?

Updated the to use the parent class

platform_instance and env still there as well as platform_instance in GrafanaSourceConfig 🤔

what parent class do you refer to?

metadata-ingestion/src/datahub/ingestion/source/grafana/models.py

metadata-ingestion/src/datahub/ingestion/source/grafana/report.py

metadata-ingestion/src/datahub/ingestion/source/grafana/snapshots.py

mayurinehate

Looking good. I'd much appreciate a "Concept Mapping" section in docs.

mayurinehate · 2025-01-28T07:26:15Z

metadata-ingestion/src/datahub/ingestion/source/grafana/grafana_source.py

+    - Automated tag extraction
+    - Support for both HTTP and HTTPS connections with optional SSL verification
+
+    Prerequisites:


This is great. Can you upgrade Prerequisites to a H3 heading and maybe move to metadata-ingestion/docs/sources/grafana/grafana_pre.md ?

mayurinehate · 2025-01-28T07:27:11Z

metadata-ingestion/src/datahub/ingestion/source/grafana/grafana_source.py

+       - Access data source configurations
+       - View user information
+
+    A sample configuration file:


While I absolutely enjoyed reading this doc in flow, we have a place to special place to keep the recipe. Can you move it to metadata-ingestion/docs/sources/grafana/grafana_recipe.yml

mayurinehate · 2025-01-28T07:49:46Z

metadata-ingestion/src/datahub/ingestion/source/grafana/grafana_source.py

+        )
+
+        # Generate dashboard container first
+        yield from gen_containers(


If I understand correctly this is new addition over only dashboard entity that was present earlier. so one container corresponding to each dashboard - to encompass panels (chart) and datasource (dataset) entity ?

If so wondering if datasources are dashboard scoped or global in grafana.

mayurinehate · 2025-01-28T07:53:17Z

metadata-ingestion/src/datahub/ingestion/source/grafana/grafana_source.py

-                platform_instance=self.source_config.platform_instance,
+
+            yield from add_dataset_to_container(
+                container_key=folder_key,


You can set parent_container_key = folder_key in gen_containers for dashboard instead

mayurinehate · 2025-01-28T08:13:58Z

metadata-ingestion/src/datahub/ingestion/source/grafana/grafana_source.py

+                yield lineage.as_workunit()
+
+            # Create chart MCE
+            dataset_urn, chart_mce = build_chart_mce(


Better to move away from Snapshot / MCE and use Aspect / MCP instead.

Also applicable for Chart and Dataset Snapshots below.

mayurinehate · 2025-01-28T09:30:51Z

metadata-ingestion/src/datahub/ingestion/source/grafana/lineage.py

+            return None
+
+        ds_type, ds_uid = self._extract_datasource_info(panel.datasource)
+        raw_sql = self._extract_raw_sql(panel.targets)


Just curious if we ingest raw query used in panel anywhere in model. I believe the language of query depends on the source it integrates with.

mayurinehate · 2025-01-28T09:31:41Z

metadata-ingestion/src/datahub/ingestion/source/grafana/lineage.py

+            return None
+
+        ds_type, ds_uid = self._extract_datasource_info(panel.datasource)
+        raw_sql = self._extract_raw_sql(panel.targets)


Just curious if we ingest raw query used in panel anywhere in datahub model? I believe the language of query depends on the source it integrates with.

mayurinehate · 2025-01-28T09:31:51Z

metadata-ingestion/src/datahub/ingestion/source/grafana/lineage.py

+            env=self.env,
+        )
+
+    def _process_platform_lineage(


Where is this used ? Let's remove if this is not used

mayurinehate · 2025-01-28T09:34:13Z

metadata-ingestion/src/datahub/ingestion/source/grafana/types.py

+}
+
+
+class GrafanaTypeMapper:


Is this class used anywhere ?

mayurinehate · 2025-01-28T09:35:33Z

metadata-ingestion/tests/integration/grafana/test_grafana.py

    service_account = grafana_client.create_service_account(
-        name="example-service-account", role="Viewer"
+        name="example-service-account", role="Admin"


Any specific reason for changing role here ? Do we require elevated permissions for changes in this PR?

sgomezvillamor · 2025-01-29T12:27:20Z

metadata-ingestion/src/datahub/ingestion/source/grafana/field_utils.py

+        from_start = sql.index("from")
+        select_part = sql[select_start:from_start].strip()
+
+        columns = [col.strip().split()[-1].strip() for col in select_part.split(",")]


Assuming that split by , will actually split by columns may be wrong assumption, eg: SELECT CONCAT(first_name, ' ', last_name) AS full_name FROM users
Not sure if we need to address this complexity though. WDYT?

acrylJonny added 2 commits January 21, 2025 15:17

initial commit

d4be2de

Delete grafana2

660daf8

acrylJonny marked this pull request as draft January 21, 2025 15:20

Merge branch 'master' into grafana-improvements

6cfbf5f

github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label Jan 21, 2025

vercel bot deployed to Preview January 21, 2025 15:55 View deployment

acrylJonny and others added 3 commits January 21, 2025 21:44

second commit

23e4eca

updating tests and adding tags and ownership options

5468909

Merge branch 'master' into grafana-improvements

e684eed

acrylJonny marked this pull request as ready for review January 21, 2025 22:21

datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Jan 21, 2025

acrylJonny added 2 commits January 21, 2025 22:39

error updates and better formatting of docs

eda52c4

Update grafana_api.py

507c811

vercel bot deployed to Preview January 21, 2025 23:39 View deployment

hsheth2 reviewed Jan 22, 2025

View reviewed changes

metadata-ingestion/src/datahub/ingestion/source/grafana/models.py Outdated Show resolved Hide resolved

hsheth2 requested review from sgomezvillamor and mayurinehate January 22, 2025 00:04

datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Jan 22, 2025

acrylJonny and others added 3 commits January 22, 2025 11:08

updating to use basemodel. Cleaning up code

0e9f0dc

test updates

c832e91

Merge branch 'master' into grafana-improvements

0fe68d2

datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Jan 22, 2025

reorder libraries to be alphabetical

c0e63d7

vercel bot had a problem deploying to Preview January 22, 2025 11:54 Failure

sgomezvillamor reviewed Jan 22, 2025

View reviewed changes

metadata-ingestion/src/datahub/ingestion/source/grafana/field_utils.py Outdated Show resolved Hide resolved

datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Jan 22, 2025

Update grafana_source.py

64a3bae

datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Jan 22, 2025

vercel bot deployed to Preview January 22, 2025 18:19 View deployment

datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Jan 23, 2025

sgomezvillamor reviewed Jan 23, 2025

View reviewed changes

metadata-ingestion/src/datahub/ingestion/source/grafana/field_utils.py Outdated Show resolved Hide resolved

sgomezvillamor reviewed Jan 23, 2025

View reviewed changes

metadata-ingestion/src/datahub/ingestion/source/grafana/field_utils.py Show resolved Hide resolved

sgomezvillamor reviewed Jan 23, 2025

View reviewed changes

acrylJonny and others added 2 commits January 24, 2025 18:08

adding unit tests

09b1ed8

Merge branch 'master' into grafana-improvements

b06fd12

datahub-cyborg bot added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Jan 24, 2025

acrylJonny added 2 commits January 24, 2025 18:28

Adding optionals

e0533ad

removing sqlparse requirement

27a9ceb

vercel bot deployed to Preview January 24, 2025 19:00 View deployment

acrylJonny and others added 2 commits January 24, 2025 20:22

test updates

a7e9e19

Merge branch 'master' into grafana-improvements

4714b24

vercel bot deployed to Preview January 24, 2025 20:49 View deployment

Merge branch 'master' into grafana-improvements

f0f88c6

vercel bot deployed to Preview January 26, 2025 13:31 View deployment

mayurinehate reviewed Jan 28, 2025

View reviewed changes

datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Jan 28, 2025

sgomezvillamor reviewed Jan 29, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ingestion/grafana): Add datasets and charts to dashboards with lineage and tags. Lineage back to source #12417

feat(ingestion/grafana): Add datasets and charts to dashboards with lineage and tags. Lineage back to source #12417

acrylJonny commented Jan 21, 2025 •

edited

Loading

codecov bot commented Jan 21, 2025 •

edited

Loading

sgomezvillamor Jan 23, 2025

acrylJonny Jan 24, 2025

sgomezvillamor left a comment

sgomezvillamor Jan 23, 2025

acrylJonny Jan 25, 2025

sgomezvillamor Jan 29, 2025

sgomezvillamor Jan 23, 2025

acrylJonny Jan 24, 2025

sgomezvillamor Jan 23, 2025

acrylJonny Jan 25, 2025

sgomezvillamor Jan 29, 2025

mayurinehate left a comment

mayurinehate Jan 28, 2025

mayurinehate Jan 28, 2025

mayurinehate Jan 28, 2025

mayurinehate Jan 28, 2025

mayurinehate Jan 28, 2025

mayurinehate Jan 28, 2025

mayurinehate Jan 28, 2025

mayurinehate Jan 28, 2025

mayurinehate Jan 28, 2025

mayurinehate Jan 28, 2025

mayurinehate Jan 28, 2025

mayurinehate Jan 28, 2025

sgomezvillamor Jan 29, 2025



		def _deduplicate_fields(fields: List[SchemaFieldClass]) -> List[SchemaFieldClass]:
		"""Remove duplicate fields based on fieldPath while preserving order."""

feat(ingestion/grafana): Add datasets and charts to dashboards with lineage and tags. Lineage back to source #12417

Are you sure you want to change the base?

feat(ingestion/grafana): Add datasets and charts to dashboards with lineage and tags. Lineage back to source #12417

Conversation

acrylJonny commented Jan 21, 2025 • edited Loading

Checklist

codecov bot commented Jan 21, 2025 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sgomezvillamor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mayurinehate left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acrylJonny commented Jan 21, 2025 •

edited

Loading

codecov bot commented Jan 21, 2025 •

edited

Loading