feat(engagement): retrofit rating & feedback system onto main#250
Open
gblanc-1a wants to merge 92 commits into
Open
feat(engagement): retrofit rating & feedback system onto main#250gblanc-1a wants to merge 92 commits into
gblanc-1a wants to merge 92 commits into
Conversation
- Add telemetry types (TelemetryEvent, TelemetryFilter, TelemetryEventType) - Add rating types (Rating, RatingScore, RatingStats) - Add feedback types (Feedback, EngagementResourceType) - Add backend configuration types (FileBackendConfig, GitHubDiscussionsBackendConfig) - Add privacy settings types (EngagementPrivacySettings) - Add hub engagement configuration types (HubEngagementConfig) These types form the foundation for the engagement system, supporting telemetry tracking, user ratings, feedback collection, and multiple backend implementations.
- Add Wilson score lower bound for binary ratings (up/down votes) - Add Bayesian smoothing for positive-only ratings - Add score-to-stars and stars-to-score conversion utilities - Add aggregate score calculation for collections - Add comprehensive tests for all algorithms These algorithms provide robust ranking that handles small sample sizes well, preventing items with few votes from dominating rankings.
- Add EngagementStorage class for persistent storage of engagement data - Support telemetry events, ratings, and feedback storage - Implement JSON file-based persistence with atomic writes - Add query methods with filtering support - Add comprehensive tests for all storage operations This provides the persistence layer for the engagement system, storing data in the extension's global storage directory.
- Add IEngagementBackend interface defining backend contract - Add BaseEngagementBackend abstract class with common functionality - Implement FileBackend for local file-based storage - Support telemetry, ratings, and feedback operations - Add comprehensive tests for FileBackend The backend abstraction allows multiple implementations (file, GitHub Discussions, API) while maintaining a consistent interface.
- Implement GitHubDiscussionsBackend for voting via GitHub reactions - Support collection-level voting on discussions - Support resource-level voting on comments - Load collection mappings from collections.yaml URL - Post feedback as discussion comments - Use FileBackend for local telemetry/feedback storage - Add comprehensive tests with nock HTTP mocking This backend enables community engagement through GitHub Discussions, using reactions for voting and comments for feedback.
- Implement singleton EngagementService as unified facade - Support hub-specific backend routing - Initialize default file backend on startup - Register hub backends dynamically based on configuration - Emit events for rating and feedback submissions - Support privacy settings enforcement - Add comprehensive tests for service operations The EngagementService provides a single entry point for all engagement operations, routing requests to the appropriate backend based on hub.
- Add VoteService for voting via GitHub Discussion reactions - Support upvote (+1) and downvote (-1) on collections and resources - Implement vote toggle and removal - Add VoteCommands for VS Code command registration - Use VS Code GitHub authentication provider - Add comprehensive tests for voting operations Users can now vote on collections directly from VS Code, with votes synced to GitHub Discussions as reactions.
- Add RatingService for fetching ratings from hub's ratings.json - Add RatingCache for in-memory caching with TTL - Support rating lookup by bundle ID with source prefix matching - Parse and validate ratings.json format - Add comprehensive tests for rating operations Ratings are pre-computed by GitHub Actions and served as static JSON, enabling fast rating display without runtime computation.
- Add FeedbackService for feedback operations - Add FeedbackCache for in-memory feedback caching - Add FeedbackCommands with unified feedback dialog - Implement star rating (1-5) with optional comment - Support issue redirect to source repository - Extract GitHub org/repo for proper issue URL construction - Add comprehensive tests for feedback operations Users can submit feedback with star ratings and optional comments, with the option to open a GitHub issue for detailed bug reports.
- Add engagement configuration schema to hub-config.schema.json - Add registerHubEngagement() to HubManager for backend initialization - Add initializeEngagementBackends() for activation-time initialization - Support GitHub Discussions backend configuration - Add validation tests for engagement configuration Hubs can now configure engagement backends, enabling automatic registration of the appropriate backend when a hub is imported.
- Add rating display to marketplace bundle tiles - Add rating display to tree view items - Integrate RatingCache for fetching cached ratings - Show star rating, vote count, and confidence level - Update registry types for rating display support Users can now see ratings directly in the marketplace and tree view, helping them discover high-quality bundles.
- Add marketplace webview assets (HTML, CSS, JS) as separate files - Add bundleDetails webview assets (HTML, CSS, JS) as separate files - Use IIFE pattern for JavaScript to comply with CSP - Use data-action attributes instead of inline event handlers - Add CopyWebpackPlugin to webpack config for asset copying Externalizing webview content improves maintainability, enables CSP compliance, and simplifies debugging of webview code.
- Initialize EngagementService on extension activation - Initialize engagement backends for all existing hubs - Register FeedbackCommands and VoteCommands - Add copy-webpack-plugin dependency for webview assets - Add feedback and vote commands to package.json - Update context menu entries for engagement commands The extension now fully integrates the engagement system, initializing all services and registering commands on activation.
- Add compute-ratings.ts for rating computation from GitHub Discussions - Add setup-discussions.ts for creating discussions from collections - Implement Wilson score and Bayesian smoothing algorithms - Support star rating parsing from feedback comments - Add user deduplication (last rating wins) - Add CLI wrappers for direct script execution - Add comprehensive tests for rating computation - Configure as standalone npm package (@prompt-registry/lib) This package runs in GitHub Actions to compute ratings from Discussion reactions and feedback comments, outputting ratings.json for static hosting.
- Update compute-ratings description to mention star rating parsing from comments - Update setup-discussions description to mention star ratings instead of reactions - Add npx --package syntax to all engagement tool examples - Note backward compatibility with legacy thumbs up/down reactions
- Add comprehensive user guide for engagement features - Document feedback submission flow with star ratings - Add hub maintainer setup guide for GitHub Discussions - Add compute-ratings.yml workflow as downloadable asset - Update hub schema documentation for engagement config - Update commands reference for feedback and vote commands - Document webview architecture and CSP compliance - Fix ESLint errors in FeedbackCommands.ts regex patterns
- Update license from MIT to Apache-2.0 to match main repository - Bump version to 1.0.3
- Remove VoteCommands.ts and VoteService.ts (not used in UI) - Remove all voting command registrations from package.json - Remove voting command imports from extension.ts - Update documentation to focus on feedback-only engagement - Update .gitignore to exclude working/analysis files The voting functionality was implemented but never integrated into the UI or used anywhere in the codebase. This cleanup removes the unused code while keeping the core feedback and rating features.
- Remove legacy reaction-based fallback when no star ratings exist - Collections with no ratings now show star_rating: 0 instead of 3.5 - Convert bin/compute-ratings.js to wrapper script calling dist/ code - Fix wilson_score calculation to properly normalize 5-star ratings Previously, when no star ratings were found, the system fell back to the legacy reaction-based system which used bayesianSmoothing() with a prior mean of 3.5. This caused collections with 0 ratings to show a star_rating of 3.5, which is incorrect. Now, collections with no ratings show 0, and only actual star ratings from feedback comments are used for the rating computation.
- Add rating_count field to CollectionRating interface - Track actual number of star ratings separately from reactions - Update RatingService to use rating_count instead of up+down - Fix issue where ratings with 0 reactions but >0 star ratings were hidden Previously, totalVotes was calculated as up+down (reactions), which caused bundles with star ratings but no reactions to show voteCount=0 and be hidden from the UI. Now we properly track the actual number of star ratings in the rating_count field.
The RatingCache was storing ratings using source_id from ratings.json (e.g., 'awesome-copilot-official'), but the UI was looking them up using the full extension source ID (e.g., 'hub-awesome-copilot-hub-099578-awesome-copilot-official'). This caused all cache lookups to return undefined, resulting in 'No ratings yet' being displayed even when ratings existed. Fix by: - Adding sourceIdMap parameter to RatingCache.refreshFromHub() - Building source ID mapping in extension.ts when loading hub ratings - Using mapped source IDs when populating the cache This ensures ratings are stored with the correct source IDs that match what the UI uses for lookups.
Covers network resilience, optimistic rating updates, interactive stars, sort fix, and other UX improvements from issue AmadeusITGroup#98. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
12-task TDD implementation plan covering network resilience, optimistic rating updates, interactive stars, sort fix, confidence removal, zero-rating consistency, report/request links, and retry mechanism. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use starRating as primary sort key with voteCount as tiebreaker. Previously used wilsonScore (0-1 range) which produced unintuitive ordering. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Keep vote count but remove confidence text (low/medium/high/very_high) from both marketplace badges and TreeView tooltips. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hide rating badge entirely when voteCount is 0 or starRating is 0. Previously could show empty stars or '0.0' for unrated bundles. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add /^installed_bundle/ to VALID_CONTEXT_PATTERNS_FOR_MENUS to match the package.json when clause for feedback menu items. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add PendingFeedback type and CRUD operations to EngagementStorage for tracking feedback that may not yet be synced to GitHub. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Save feedback locally regardless of GitHub API success. Show warning on network failure with instructions to retry. Feedback is marked as synced/unsynced for future retry support. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
After user submits a rating, immediately update the cache with a recalculated average. Silently overwritten on next ratings.json fetch. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Address type errors, unused variables, and stricter type checking issues introduced by dependency upgrades. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ncy upgrades Auto-fixed formatting and import issues detected by updated ESLint after merging the dependency upgrade branch. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Private/internal repos return 404 on raw.githubusercontent.com without auth. Thread the user's GitHub session token through: - RatingService.fetchRatings() → Authorization header - FeedbackService.fetchFeedbacks() → Authorization header - GitHubDiscussionsBackend.loadCollectionsMappings() → uses getAccessToken() - HubManager.registerHubEngagement() → obtains token, passes to cache warm-up
handleBundleDetailRateBundle was only updating the panel webview, not the marketplace view behind it. Add postRatingUpdate() calls in the optimistic and rollback callbacks so the marketplace tile reflects the new rating immediately. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
GitHub Discussions reactions are only available via GraphQL, not REST.
The REST endpoint /repos/{owner}/{repo}/discussions/{num}/reactions
returns 404. Switch submitRating and deleteRating to use the
addReaction/removeReaction GraphQL mutations.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move ratingSubmitted/openFeedbackModal to the optimistic update callback so the feedback form appears instantly after clicking a star, rather than waiting for the GraphQL API calls to complete. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
raw.githubusercontent.com returns 404 for internal/private repos even with an auth token. When the primary fetch fails, convert the raw URL to the GitHub API contents endpoint with Accept: application/vnd.github.v3.raw. Applies to both ratingsUrl (rating-service) and collectionsUrl (github-discussions-backend). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The marketplace was loading bundles before the rating cache finished warming from the hub's ratingsUrl. Since it never re-rendered after the cache populated, ratings never appeared on tiles. Subscribe to RatingCache.onCacheUpdated to trigger loadBundles() when new ratings arrive. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds utility to extract star ratings from GitHub Discussion comment bodies. Supports the format "Rating: ⭐⭐⭐" and validates that ratings are 1-5 stars. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add optimisticKeys Set to track in-session optimistic ratings
- Modify hydrateUserRatings to accept options: { overwrite?: boolean }
- When overwrite=true, replace existing entries except optimistic ones
- Clear optimisticKeys in clear() and dispose()
- Add tests for overwrite behavior
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ings Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When users submit feedback text after rating, the feedback is now merged into their existing rating comment by updating it, rather than creating a separate comment. This provides a cleaner comment thread and groups all user feedback in one place. The submitFeedback flow now: 1. Checks for existing rating comment via findViewerComment 2. If found: updates it with the full feedback (rating + comment) 3. If not found: creates a new comment (same as before) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix member ordering in GitHubDiscussionsBackend (move private methods before public) - Fix Array<T> to T[] syntax in type definitions - Add JSDoc description to parseRatingFromComment function Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previously used .find() which returns the first (oldest) match. If duplicate comments existed, the edit would target the wrong one while compute-ratings picked the latest by createdAt. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When ratings.json is fetched on startup, it may not yet include the user's latest vote (compute hasn't re-run). Previously this overwrote the optimistic aggregate, reverting the displayed rating. Now doRefresh re-applies in-session user votes via the extracted injectUserVote method (shared with applyOptimisticRating, no duplicated logic). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…tart On extension restart, optimisticKeys is empty so doRefresh can't re-apply user votes. After local hydration sets userRatings, reapplyHydratedVotes() injects those votes into the aggregate cache so the displayed star_rating reflects the user's vote even when ratings.json is stale (compute hasn't re-run yet). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
28 tasks
…k flow Extract hydration logic from HubManager into dedicated EngagementHydrator class for better separation of concerns. Add github-url-utils for URL parsing, share EngagementStorage instance via FileBackend.setSharedStorage(), and add tests for feedback-service and rating-cache edge cases. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
# Conflicts: # src/ui/webview/marketplace/marketplace.js
…hub-release-analyzer.test.ts - Fixed @typescript-eslint/member-ordering in 4 engagement service files by moving private methods before public ones - Fixed jsdoc/require-description by adding missing JSDoc block descriptions - Restored lib/test/hub-release-analyzer.test.ts to main version (lint-clean copy) - Added linting guidance to src/services/AGENTS.md: method ordering rules and JSDoc requirements - All tests passing: 2563 unit tests, 7 integration tests Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…ules Root AGENTS.md: add missing src/ directories (config, integrations, migrations, notifications) and new key files to the reference table. src/services/AGENTS.md: correct member-ordering rule to match actual ESLint config, fix JSDoc rule attribution to eslint-plugin-jsdoc, add 19 missing services to the key services table. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Restore .gitignore and hub-release-analyzer.test.ts to main's version, and delete cross-machine-rating-recovery plan docs (unimplemented future work). Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Extract fetchDiscussionComments helper to deduplicate GraphQL queries - Inject dependencies into EngagementHydrator instead of calling singletons - Add comment explaining benign timeout race in collections loading - Add "Where Ratings Live" reconciliation docs with file references Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Re-integrates the engagement (rating + feedback) subsystem preserved on
feat/feedback-cleanon top of main's merged telemetry system. Main'sTelemetryServiceremains the sole telemetry layer; the branch's duplicate telemetry types/methods were removed. Data plane, tree view, marketplace view, bundle details view, and per-hub engagement backend registration are all wired up.Provenance: merge commit
6e5a8fbtook main's side for every conflict; this PR's 21 commits retrofit the engagement code onto that post-merge base (fa07fdb).What's inside
Data plane
refactor(engagement): rip duplicate telemetry types/methods; drop dead privacy-settings surfacefeat(engagement):HubManager.registerHubEngagement+ activation-timeinitializeEngagementBackendswith cache warmup from hubratingsUrl/feedbackUrlfeat(engagement):RatingCache.userRatingsmap +rollbackOptimisticRatingso re-rating updates (not double-counts) and failed submits undo cleanlyfeat(engagement): drain unsynced pending feedback on activationfix(hub): route engagement config throughvalidateHubEngagementConfigso invalid URLs are caught at hub loadUI
feat(types): optionalbundleRating?: CachedRatingon the bundle view-modelfeat(ui): rating suffix★ X.X (N)on tree-view installed bundles (with fix forsetVersionDisplayoverwrite)feat(ui): static stars on marketplace tiles → click-to-rate → optimistic update → feedback modal → pending-feedback retryfeat(ui): bundle-details webview rating flow + Report Issue / Request Feature / Retry Feedback buttonsfix(ui): resolve hubId from source so ratings route to the right backend (not the fallback local file)Commands, build & linting
refactor(feedback): drop unusedsubmitFeedbackalias commandchore(lint): clean up linting errors (member ordering, jsdoc); added linting guidance to src/services/AGENTS.mdtest(ui): register/^installed_bundle/pattern used by engagement context menu entriesMerge-policy notes
TelemetryServiceatsrc/services/telemetry-service.tsis the sole telemetry layer. All engagement-layerrecordTelemetry/getTelemetry/_onTelemetryRecorded/TelemetryEventtypes were removed.HubConfig.engagement?: HubEngagementConfigwas missing on the TS interface post-merge; added.copy-webviewscript inpackage.json:752already handles both marketplace and bundle-details webview directories — no webpack changes needed.Test plan
npm run lintis green (0 errors)npm run test:unitis green (2563 passing, 33 pending, 0 failing)npm run test:integrationis green (7 passing)engagement.enabled: true★ X.X (N)suffix on installed bundles with cached ratingscollections.yamlmapping exists) or stores locallypending-feedback.json; reconnect + reload → entry drainsFollow-up work (not in this PR)
The hub's GitHub Discussions backend requires
collections.yamlto be seeded in the engagement repo mapping bundle IDs → discussion numbers. A proposed follow-up: auto-create the discussion on first-rate when the mapping is missing, and havecompute-ratingsdetect new discussions and updatecollections.yaml. Design note to be added to the plan.🤖 Generated with Claude Code