diff --git a/services/libs/tinybird/datasources/projects_list_for_top_projects_widget_ds.datasource b/services/libs/tinybird/datasources/projects_list_for_top_projects_widget_ds.datasource new file mode 100644 index 0000000000..1538c69703 --- /dev/null +++ b/services/libs/tinybird/datasources/projects_list_for_top_projects_widget_ds.datasource @@ -0,0 +1,21 @@ +DESCRIPTION > + - `projects_list_for_top_projects_widget_ds` contains data specifically for the Top LF Projects widget. + - Created via the projects_list_for_top_projects_widget_copy pipe with the same filters that the frontend was applying before filtering by activity type. + - `id` is the primary key identifier for the project record. + - `name` and `slug` are basic project metadata fields. + - `logo` is the URL for the project logo. + - `contributorCount` is the computed contributor statistics from insights_projects_populated_ds. + +TAGS "Project metadata", "Analytics enrichment" + +SCHEMA > + `id` String, + `name` String, + `slug` String, + `logo` String, + `contributorCount` UInt64, + `snapshotAt` DateTime64(3) DEFAULT now() + +ENGINE ReplacingMergeTree +ENGINE_PARTITION_KEY tuple() +ENGINE_SORTING_KEY id diff --git a/services/libs/tinybird/pipes/projects_list_for_top_projects_widget.pipe b/services/libs/tinybird/pipes/projects_list_for_top_projects_widget.pipe new file mode 100644 index 0000000000..340cbabc0b --- /dev/null +++ b/services/libs/tinybird/pipes/projects_list_for_top_projects_widget.pipe @@ -0,0 +1,16 @@ +DESCRIPTION > + - `projects_list_for_top_projects_widget.pipe` serves the data for the "Top LF projects" widget. + - It uses the projects_list_for_top_projects_widget_ds data source for activity type filtering, sorting, and a + dynamic LIMIT count to serve the data for both the main page version (10 results) + and the pop up version (50 items) of the widget. + - `pageSize`: Optional integer for result limit, defaults to 10 + +TAGS "API", "Projects", "ActivityTypes" + +NODE projects_for_top_lf_projects_widget +SQL > + % + SELECT * + FROM projects_list_for_top_projects_widget_ds + ORDER BY contributorCount DESC + LIMIT {{ Int32(pageSize, 10) }} diff --git a/services/libs/tinybird/pipes/projects_list_for_top_projects_widget_copy.pipe b/services/libs/tinybird/pipes/projects_list_for_top_projects_widget_copy.pipe new file mode 100644 index 0000000000..471efac162 --- /dev/null +++ b/services/libs/tinybird/pipes/projects_list_for_top_projects_widget_copy.pipe @@ -0,0 +1,38 @@ +NODE projects_list_filtered_by_activity_types +DESCRIPTION > + This is similar to the insightsProjects_filtered pipe but includes filtering for activity types, + which we don't have on the former, and some other hard-coded filters for the Top LF projects widget. + These have to be hard-coded because this is a copy pipe and thus, we can't use dynamic filtering. + This exists just for the aforementioned widget, because adding the activity type filtering to + insightsProjects_filtered could have too much of a performance hit, and that pipe is used in + a lot of other places, so we would negatively impact the whole application. + +SQL > + SELECT DISTINCT p.id, p.name, p.slug, p.logoUrl as logo, p.contributorCount + FROM insights_projects_populated_ds p + INNER JOIN activityRelations_deduplicated_cleaned_ds a ON a.segmentId = p.segmentId + WHERE + p.enabled = 1 + -- This is the same filter as the "onboarded" filter in insightsProjects_filtered.pipe + -- but hard-coded for the filter parameters used by the Top LF projects widget. + AND NOT (p.organizationCount = 0 AND p.contributorCount = 0) + -- This is the same filter as the "isLf" filter in insightsProjects_filtered.pipe but + -- hard-coded to as used by the Top LF projects widget. + AND insights_projects_populated_ds.isLF = 1 + -- This is a hard-coded filter for counting activities that are code contributions or + -- collaborations, because this is what we want to show on the Top LF projects widget. + AND (a.type, a.platform) IN ( + SELECT activityType, platform + FROM activityTypes + WHERE isCodeContribution = 1 OR isCollaboration = 1 + ) + -- This is the sorting order used by the widget. + ORDER BY p.contributorCount DESC + -- As the Top LF projects widget pulls either 10 or 50 results, we only need to get 50 here + -- and if necessary, in the pipe that is consumed by the app, we can narrow it down to 10. + LIMIT 50 + +TYPE COPY +TARGET_DATASOURCE projects_list_for_top_projects_widget_ds +COPY_MODE replace +COPY_SCHEDULE 13 * * * *