feat(save-user-data): extend user data abstract class for api backend COMPASS-9558 #7114

myang1220 · 2025-07-15T19:46:36Z

Description

Extend the IUserData abstract class to create an AtlasUserData class that uses the endpoints created in CCS. Contains unit testing.

Checklist

New tests and/or benchmarks are included
Documentation is changed or added
If this change updates the UI, screenshots/videos are added and a design review is requested
I have signed the MongoDB Contributor License Agreement (https://www.mongodb.com/legal/contributor-agreement)

Motivation and Context

Bugfix
New feature
Dependency update
Misc

Open Questions

See comments

Dependents

Types of changes

Backport Needed
Patch (non-breaking change which fixes an issue)
Minor (non-breaking change which adds functionality)
Major (fix or feature that would cause existing functionality to change)

Merge branch 'user-data-interface' of https://github.com/mongodb-js/compass into user-data-interface

Merge branch 'extend-user-data' of https://github.com/mongodb-js/compass into extend-user-data

…end-user-data

packages/compass-user-data/src/user-data.ts

packages/compass-user-data/package.json

packages/compass-user-data/src/user-data.ts

myang1220 · 2025-07-18T21:09:39Z

packages/compass-user-data/src/user-data.ts

+      }
+
+      return true;
+    } catch (error) {


How does the logging look? Are there any changes that I should make to make this more consistent with Compass behavior?

packages/compass-user-data/src/user-data.ts

…e queries

Merge branch 'extend-user-data' of https://github.com/mongodb-js/compass into extend-user-data

myang1220 · 2025-07-29T20:48:01Z

packages/bson-transpilers/lib/symbol-table/shelltocsharp.js

@gribnoysup @Anemy
Do you guys have any clue where these may have come from? I haven't personally modified any of them, so I wonder whether it's due to command line commands?

Yeah, those are not really related. Probably accidentally run a compile command across all packages? We haven't updated this in a while, so I guess there might be some slight formatting variations or something, probably better to just revert those from your patch

gribnoysup · 2025-07-30T08:56:04Z

packages/compass-shell/src/modules/history-storage.ts

@@ -6,9 +6,8 @@ export class HistoryStorage {
  userData;

  constructor(basePath?: string) {
-    this.userData = new FileUserData(z.string().array(), {
+    this.userData = new FileUserData(z.string().array(), getAppName() ?? '', {
      // Todo: https://jira.mongodb.org/browse/COMPASS-7080


Nit: this todo should probably still point at the getAppName() part

Is this just referring to the TODO comment?

Remove unwanted bson-transpilers changes that were accidentally included in the rebase. This resets the bson-transpilers package back to its master state while preserving the user-data related changes that are needed for this PR.

Rebuild the generated symbol table files to ensure they are properly formatted and up-to-date with the current source templates.

nvs119 · 2025-08-12T22:24:51Z

Trying to figure out if there are more outstanding comments here that I'll need to resolve as I'm finishing up this PR for Moses! This PR can be merged without fear (I believe) since the actual storage providers don't get added until the next PR. Let me know if that understanding is incorrect!

Anemy

Nice, left a couple comments. Mostly thinking about how we can make the error handling surface easily to the consumer.

Anemy · 2025-08-13T16:02:35Z

packages/compass-user-data/src/user-data.ts

+    }
+  }
+
+  // TODO: change this depending on whether or not updateAttributes can provide all current data


Is this something we need to do in this pr?

I believe Moses had a follow up PR for this 1a0fcc3

Anemy · 2025-08-13T16:07:26Z

packages/my-queries-storage/src/compass-pipeline-storage.ts

+      });
+      return true;
+    } catch {
+      return false;


Here and in other places, would it make sense to instead be throwing these errors? That way the places that use these storage methods can surface errors to users when they happen. Otherwise folks wouldn't know when something failed to write/load/update without checking the logs.

I may be wrong, but when I follow this up to the UI calls, the UI ignores return values. if we let the errors bubble up, the UI would crash (to my understanding) with unhandled promise rejections, right?

do you want me to add proper error handling in the UI layer as well? Just not clear if that change is within the scope of this PR or not, happy to do so that if that's what you're suggesting

Anemy · 2025-08-13T16:10:52Z

packages/compass-user-data/src/user-data.ts

+    orgId: string,
+    projectId: string,
+    getResourceUrl: GetResourceUrl,
+    authenticatedFetch: AuthenticatedFetch,


When we start getting this many arguments it reads nicer if we include them in an object type. How does it sound if we have all of these except the validator in AtlasUserDataOptions?

gribnoysup · 2025-08-14T09:32:15Z

packages/compass-user-data/src/user-data.ts

+      if (!response.ok) {
+        throw new Error(
+          `Failed to post data: ${response.status} ${response.statusText}`
+        );
+      }


This is already handled by authenticatedFetch method, so no need to check ok here and it all the other places where authenticatedFetch is used

gribnoysup · 2025-08-14T09:33:03Z

packages/compass-user-data/src/user-data.ts

+          url: await this.getResourceUrl(
+            `${this.dataType}/${this.orgId}/${this.projectId}`
+          ),


Super nit, maybe let's just get it once at the top of the function?

Changing this for write(), delete(), and readOne()

Left updateAttributes unchanged due to the dependency on the readOne() call order

Copilot

Pull Request Overview

This PR extends the user data storage abstraction to support Atlas backend APIs by creating a new AtlasUserData class alongside the existing FileUserData class. The primary purpose is to enable saving user data to Atlas cloud services through HTTP endpoints while maintaining the same interface as the file-based storage.

Key changes include:

Refactoring the FileUserData constructor to accept dataType as a separate parameter instead of within options
Creating a new AtlasUserData class that implements HTTP-based storage operations
Modifying interface return types from domain objects to boolean success indicators

Reviewed Changes

Copilot reviewed 18 out of 19 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
packages/compass-user-data/src/user-data.ts	Adds new `AtlasUserData` class and refactors constructor parameter structure
packages/compass-user-data/src/user-data.spec.ts	Comprehensive test suite for the new `AtlasUserData` functionality
packages/compass-user-data/src/index.ts	Exports the new `AtlasUserData` class
packages/my-queries-storage/src/*.ts	Updates constructors and return types to align with new interface
packages/connection-storage/src/compass-main-connection-storage.ts	Updates constructor calls for refactored parameter structure
packages/compass-shell/src/modules/history-storage.ts	Updates constructor calls for refactored parameter structure
packages/compass-preferences-model/src/*.ts	Updates constructor calls for refactored parameter structure
packages/compass-data-modeling/src/services/data-model-storage-electron.tsx	Updates constructor calls for refactored parameter structure
packages/atlas-service/src/*.ts	Renames `wsBaseUrl` to `ccsBaseUrl` for consistency

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-08-14T21:07:21Z

packages/compass-user-data/src/user-data.ts

+  private readonly authenticatedFetch;
+  private readonly getResourceUrl;
+  private orgId: string = '';
+  private projectId: string = '';


The orgId and projectId are initialized as empty strings but are required constructor parameters. This initialization is redundant and potentially confusing since these values are immediately overwritten in the constructor.

Suggested change

private projectId: string = '';

private orgId: string;

private projectId: string;

Copilot · 2025-08-14T21:07:21Z

packages/compass-user-data/src/user-data.ts

+  private readonly authenticatedFetch;
+  private readonly getResourceUrl;
+  private orgId: string = '';
+  private projectId: string = '';


The projectId is initialized as an empty string but is a required constructor parameter. This initialization is redundant and potentially confusing since these values are immediately overwritten in the constructor.

Suggested change

private projectId: string = '';

private projectId: string;

Copilot · 2025-08-14T21:07:21Z

packages/compass-user-data/src/user-data.ts

+          error: (error as Error).message,
+        }
+      );
+      return null;


The readOne method returns null on error, but the return type is Promise<z.output>. This will cause type errors since null is not assignable to z.output. Consider throwing the error or returning a default value that matches the schema.

Suggested change

return null;

throw error;

Copilot · 2025-08-14T21:07:22Z

packages/compass-user-data/src/user-data.ts

+    data: Partial<z.input<T>>
+  ): Promise<boolean> {
+    try {
+      const prevData = await this.readOne(id);


The updateAttributes method calls readOne which can return null on error, but the code assumes it will always return valid data. This could cause runtime errors when spreading null into the newData object.

Suggested change

const prevData = await this.readOne(id);

const prevData = await this.readOne(id);

if (prevData == null) {

log.error(

mongoLogId(1_001_000_370),

'Atlas Backend',

'Error updating data: could not read previous data',

{

url: await this.getResourceUrl(

`${this.dataType}/${this.orgId}/${this.projectId}/${id}`

),

id,

}

);

return false;

}

Copilot · 2025-08-14T21:07:22Z

packages/compass-user-data/src/user-data.spec.ts

+
+      const [getUrl, getOptions] = authenticatedFetchStub.firstCall.args;
+      expect(getUrl).to.equal(
+        'cluster-connection.cloud.mongodb.com/FavoriteQueries/test-org/test-proj'


The test expects the first call to be a GET request, but the updateAttributes implementation calls readOne first which makes a GET request to a different URL pattern than expected. The URL construction should match the actual implementation.

Suggested change

'cluster-connection.cloud.mongodb.com/FavoriteQueries/test-org/test-proj'

'cluster-connection.cloud.mongodb.com/FavoriteQueries/test-org/test-proj/test-id'

Copilot · 2025-08-14T21:07:22Z

packages/compass-data-modeling/src/services/data-model-storage-electron.tsx

-    });
+    this.userData = new FileUserData(
+      MongoDBDataModelDescriptionSchema,
+      'DataModelDescription',


The dataType parameter was changed from 'DataModelDescriptions' (plural) to 'DataModelDescription' (singular), which could break existing file storage paths and cause data migration issues.

Suggested change

'DataModelDescription',

'DataModelDescriptions',

myang1220 and others added 13 commits July 10, 2025 11:55

feat(save-user-data): create a user data abstract class

3f336ec

Merge branch 'main' into user-data-interface

ad5e416

fix: rm defaults and add semaphore to FileUserData

78026a3

merge remote branch back into local

b24d748

Merge branch 'user-data-interface' of https://github.com/mongodb-js/compass into user-data-interface

Merge branch 'main' into user-data-interface

8413d43

feat(save-user-data): push for help

dc38640

feat(save-user-data): comment out nulls

41abc96

feat(save-user-data): create a user data abstract class

e90412e

fix: rm defaults and add semaphore to FileUserData

701adaf

feat(save-user-data): push for help

daee65a

feat(save-user-data): comment out nulls

90573c7

rebase main into branch

0e997ff

Merge branch 'extend-user-data' of https://github.com/mongodb-js/compass into extend-user-data

Merge branch 'main' of https://github.com/mongodb-js/compass into ext…

5d39eb3

…end-user-data

github-actions bot added the feat label Jul 15, 2025

myang1220 added the no release notes Fix or feature not for release notes label Jul 15, 2025

gribnoysup reviewed Jul 16, 2025

View reviewed changes

packages/compass-user-data/src/user-data.ts Outdated Show resolved Hide resolved

gribnoysup reviewed Jul 16, 2025

View reviewed changes

packages/compass-user-data/package.json Outdated Show resolved Hide resolved

gribnoysup reviewed Jul 16, 2025

View reviewed changes

packages/compass-user-data/src/user-data.ts Outdated Show resolved Hide resolved

myang1220 added 2 commits July 17, 2025 10:34

potentital org and group id retrieval

5edf5e2

minor fixes

729c6db

Anemy reviewed Jul 17, 2025

View reviewed changes

packages/compass-user-data/src/user-data.ts Outdated Show resolved Hide resolved

myang1220 added 2 commits July 18, 2025 16:59

unit testing

055b898

fix package-lock.json merge conflict

fed4e96

myang1220 commented Jul 18, 2025

View reviewed changes

packages/compass-user-data/src/user-data.ts Outdated Show resolved Hide resolved

myang1220 commented Jul 18, 2025

View reviewed changes

packages/compass-user-data/src/user-data.ts Outdated Show resolved Hide resolved

myang1220 commented Jul 18, 2025

View reviewed changes

packages/compass-user-data/src/user-data.ts Outdated Show resolved Hide resolved

myang1220 requested review from Anemy and gribnoysup July 18, 2025 21:21

myang1220 marked this pull request as ready for review July 18, 2025 21:21

myang1220 added 7 commits July 23, 2025 16:36

put data type in IUserData class

04ba804

Merge branch 'main' into extend-user-data

cf510ce

fix(user-data): add datatype to IUserData abstract class & tests

73b2bd1

fix: merge stats change to branch

51ab01e

fix(user-data): make create and update return type to be boolean, lik…

df135f3

…e queries

pull remote branch back into local

0bb77b8

Merge branch 'extend-user-data' of https://github.com/mongodb-js/compass into extend-user-data

fix(compass-aggregations): fix method typing

f6efcd6

myang1220 commented Jul 29, 2025

View reviewed changes

gribnoysup reviewed Jul 30, 2025

View reviewed changes

myang1220 force-pushed the extend-user-data branch from a7d1e33 to f6efcd6 Compare August 1, 2025 15:19

nvs119 added 6 commits August 12, 2025 16:20

revert: reset bson-transpilers files to master state

4037adf

Remove unwanted bson-transpilers changes that were accidentally included in the rebase. This resets the bson-transpilers package back to its master state while preserving the user-data related changes that are needed for this PR.

fix: rebuild bson-transpilers symbol tables

2b5cc80

Rebuild the generated symbol table files to ensure they are properly formatted and up-to-date with the current source templates.

restore big files

6ef0617

restore again

20625c9

package lock

cdb80d6

comments

53375ce

nvs119 requested review from Anemy and gribnoysup August 12, 2025 22:23

merge conflicts

75ccb8b

Anemy reviewed Aug 13, 2025

View reviewed changes

log ids

4063a50

gribnoysup reviewed Aug 14, 2025

View reviewed changes

changes

ae44d23

nvs119 requested review from Anemy, gribnoysup and Copilot August 14, 2025 20:38

Copilot AI reviewed Aug 14, 2025

View reviewed changes

copilot review changes

cc1ecd2

	private projectId: string = '';
	private orgId: string;
	private projectId: string;

-      const prevData = await this.readOne(id);
+      const prevData = await this.readOne(id);
+      if (prevData == null) {
+        log.error(
+          mongoLogId(1_001_000_370),
+          'Atlas Backend',
+          'Error updating data: could not read previous data',
+          {
+            url: await this.getResourceUrl(
+              `${this.dataType}/${this.orgId}/${this.projectId}/${id}`
+            ),
+            id,
+          }
+        );
+        return false;
+      }

	'cluster-connection.cloud.mongodb.com/FavoriteQueries/test-org/test-proj'
	'cluster-connection.cloud.mongodb.com/FavoriteQueries/test-org/test-proj/test-id'

feat(save-user-data): extend user data abstract class for api backend COMPASS-9558 #7114

Are you sure you want to change the base?

feat(save-user-data): extend user data abstract class for api backend COMPASS-9558 #7114

Conversation

myang1220 commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Motivation and Context

Open Questions

Dependents

Types of changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nvs119 commented Aug 12, 2025

Uh oh!

Anemy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gribnoysup Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

myang1220 commented Jul 15, 2025 •

edited

Loading

gribnoysup Aug 14, 2025 •

edited

Loading