[DAPS-1515] Integrate repository type system into core endpoints #1543

AronPerez · 2025-07-29T15:35:23Z

Ticket

Description

The updates include integrating the new repository type system into existing API endpoints, adding type-aware operations for data uploads and allocations, and making production code changes that enable repository type features.

What's included:

data_router.js

Add check for repository data operation support before uploads
Prevent data uploads to metadata-only repositories
Return appropriate error messages with repository type info

repo_router.js

Update allocation create/delete endpoints to use RepositoryOps
Add repository type awareness to allocation operations
Support different response formats based on repository type

support.js

Add new error codes for repository type operations
Add helper functions for repository type validation

tasks.js

Update task creation to handle different repository types
Add type-specific task handling logic

How Has This Been Tested?

See next PR

Artifacts

See CI

JoshuaSBrown · 2025-08-04T18:48:06Z

core/database/foxx/api/data_router.js

+                    {
+                        repo_type: repository.type,
+                        repo_id: repository.data._id,
+                    },


I like that this information is being included in the exception.

core/database/foxx/api/tasks.js

core/database/foxx/api/repo_router.js

AronPerez · 2025-08-17T22:20:11Z

@JoshuaSBrown Changes complete:

Dynamic error messages using repository.type
Added subject existence check at L776-779
Extracted validation helper to reduce duplication

PTAL

AronPerez · 2025-08-19T17:30:31Z

@JoshuaSBrown Addressed all feedback:

Dynamic error messages now use repository.type for context
Added subject existence validation at L776-779
Extracted common operations to RepositoryOps to reduce duplication

Changes committed and CI passing (ignoring shfmt per team decision).

Ready for re-review.

JoshuaSBrown · 2025-08-25T16:13:54Z

core/database/foxx/api/data_router.js


+        // Check if repository supports data operations
+        var findResult = RepositoryOps.find(repo_alloc._to);
+        if (findResult.ok) {


What happens when findResult is not ok, shouldn't we be throwing?

JoshuaSBrown · 2025-08-25T19:04:34Z

core/database/foxx/api/repo_router.js

+    if (obj.path && !obj.path.endsWith("/")) {
+        obj.path += "/";
+    }


You are making sure that the .path ends with "/", in the original code, it is also checking to make sure that it begins with "/". Is that check being done somewhere else?

I see it is in the validation.js file.

JoshuaSBrown · 2025-08-25T19:12:53Z

core/database/foxx/api/repo_router.js

-                    if (obj.path.substr(idx + 1, obj.path.length - idx - 2) != obj._key)
-                        throw [
-                            g_lib.ERR_INVALID_PARAM,
-                            "Last part of repository path must be repository ID suffix (" +
-                                obj._key +
-                                ")",
-                        ];


This is contained in the validation.js file, in the validateRepositoryPath function.

JoshuaSBrown · 2025-08-25T19:21:25Z

core/database/foxx/api/repo_router.js

+                    if (!permResult.ok || !permResult.value) {
+                        throw g_lib.ERR_PERM_DENIED;
+                    }
+
+                    // Check if subject exists
+                    if (!g_db._exists(subject_id)) {
+                        throw [g_lib.ERR_NOT_FOUND, "Subject not found: " + subject_id];


It would be clearer if we include in the exeception, that the allocation delete request failed, for audit reasons, in cases around authorization it is really useful to know who attempted to do what.

JoshuaSBrown · 2025-08-25T19:30:32Z

core/database/foxx/api/repository/metadata.js

+        // For metadata-only repos, allocations are just database records
+        // No actual storage allocation happens
+        const allocation = {
+            _key: `alloc_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`,


Why was this chosen for the key?

Arangodb will automatically create the key and id values, deviating from how the rest of the ArangoDB key and id values will introduce some level of technical debt that we might want to have justification for.

For instance here if you save allocation you will end up with an id like this

_id: 'alloc/alloc_data...blah..blah`

_id: 'alloc/403756027'

Just a placeholder, need to update that

JoshuaSBrown · 2025-08-25T19:57:35Z

core/database/foxx/api/repository/metadata.js

+const deleteAllocation = (repoData, subjectId) => {
+    if (!subjectId || typeof subjectId !== "string") {
+        return Result.err({
+            code: g_lib.ERR_INVALID_PARAM,
+            message: "Subject ID is required for allocation deletion",
+        });
+    }
+
+    try {
+        // For metadata-only repos, just remove the database record
+        // No actual storage deallocation needed
+        const result = {
+            repo_id: repoData._id,
+            subject: subjectId,
+            status: "completed",
+            message: "Metadata allocation removed",
+        };
+
+        return Result.ok(createAllocationResult(ExecutionMethod.DIRECT, result));
+    } catch (e) {
+        return Result.err({
+            code: g_lib.ERR_INTERNAL_FAULT,
+            message: `Failed to delete metadata allocation: ${e.message}`,
+        });
+    }
+};


Please take a look in taskInitAllocDelete, in the support.js file. Even though we are dealing with metadata only allocations, you will still need to remove all of the records and collections that are in there, otherwise you will end up with dangling documents.

Maybe you are implementing that in a later PR? It doesn't look like you are removing the allocation document here.

JoshuaSBrown · 2025-08-25T20:01:34Z

core/database/foxx/api/repository/metadata.js

+            repo_id: repoData._id,
+            subject: params.subject,
+            size: params.size,
+            path: params.path || `/${params.subject}`,


What is the reason for having a path here, that is not some default size and also for not just setting the size to 0?

JoshuaSBrown · 2025-08-25T20:13:14Z

core/database/foxx/api/support.js

+                // Check if repository supports data operations
+                var findResult = RepositoryOps.find(alloc._to);
+                if (findResult.ok) {
+                    var repository = findResult.value;
+                    var dataOpsResult = RepositoryOps.supportsDataOperations(repository);
+
+                    // Skip metadata-only repositories
+                    if (dataOpsResult.ok && !dataOpsResult.value) {
+                        continue;
+                    }
+                }
+


It should be ok to assign a metadata only repository when a recordCreate request comes in. We would probably want to check if there is raw data attached to the recordCreate request before picking a metadata only repository.

Note, if we only support metadata repositories and you don't enable this you will have to explicitly specify what allocation the record should be created in even if you only have the one. So we want to turn this on.

Here, you can specify the default allocation.

This should still be possible with a metadata only repository.

JoshuaSBrown · 2025-08-25T20:44:27Z

core/database/foxx/api/repository/globus.js

+        return Result.ok({
+            total_capacity: repoData.capacity,
+            used_capacity: 0, // Would be populated from actual usage
+            available_capacity: repoData.capacity,


Shouldn't this be capacity - used_capacity?

JoshuaSBrown · 2025-08-25T20:45:28Z

core/database/foxx/api/repository/globus.js

+            total_capacity: repoData.capacity,
+            used_capacity: 0, // Would be populated from actual usage
+            available_capacity: repoData.capacity,
+            supports_quotas: true,


How will this be used, and what for?

JoshuaSBrown · 2025-08-26T11:31:27Z

core/database/foxx/api/repository/metadata.js

+            subject: allocation.subject,
+            size: allocation.size,
+            path: allocation.path,
+            status: "completed",


It would be better to have status entries defined in a centralized file that can be referenced by other parts of the code base, so that it is not ambigous how these items should be returned.

This is also not consistent with what is in the tasks.js and globus files. If you look in the support.js file you will see that there are these entried:

obj.TS_BLOCKED = 0; obj.TS_READY = 1; obj.TS_RUNNING = 2; obj.TS_SUCCEEDED = 3; obj.TS_FAILED = 4; obj.TS_COUNT = 5;

They are not returned as strings but as codes. Either we need to be consistent with what is in here or also change what status is being returned by the globus status entry.

…tterns - Implement RepositoryOps.find() for repository lookup - Add getRepository() to expose the new repository object - Adjust pathType() to handle metadata-only repositories - Preserve backward compatibility with existing API

- Implement repository type checks for data operations. - Replace allocation create/delete logic with RepositoryOps. - Add support for metadata-only repositories and relevant error handling. - Ensure API response compatibility is preserved.

This was referenced Jul 29, 2025

[DAPS-1515] Add comprehensive repository tests #1544

Merged

[DAPS-1515] update endpoints #1538

Closed

AronPerez force-pushed the feat-DAPS-1515-legacy-repo-updates branch from b872681 to 2050419 Compare July 29, 2025 16:19

AronPerez force-pushed the feat-DAPS-1515-endpoint-integration branch from adbf105 to 6d667b8 Compare July 29, 2025 16:20

AronPerez force-pushed the feat-DAPS-1515-legacy-repo-updates branch from 2050419 to 8607e8b Compare July 29, 2025 16:25

AronPerez force-pushed the feat-DAPS-1515-endpoint-integration branch from 119eaa1 to a4daf27 Compare July 29, 2025 16:26

AronPerez force-pushed the feat-DAPS-1515-legacy-repo-updates branch from 8607e8b to 94f9d61 Compare July 29, 2025 16:42

AronPerez force-pushed the feat-DAPS-1515-endpoint-integration branch from a4daf27 to cc0a958 Compare July 29, 2025 16:42

JoshuaSBrown reviewed Aug 4, 2025

View reviewed changes

AronPerez force-pushed the feat-DAPS-1515-legacy-repo-updates branch from 871a716 to 6bb4228 Compare August 7, 2025 16:55

AronPerez force-pushed the feat-DAPS-1515-endpoint-integration branch from cc0a958 to b04e32d Compare August 7, 2025 16:55

JoshuaSBrown reviewed Aug 12, 2025

View reviewed changes

core/database/foxx/api/tasks.js Outdated Show resolved Hide resolved

JoshuaSBrown reviewed Aug 12, 2025

View reviewed changes

core/database/foxx/api/tasks.js Show resolved Hide resolved

JoshuaSBrown reviewed Aug 12, 2025

View reviewed changes

core/database/foxx/api/repo_router.js Show resolved Hide resolved

JoshuaSBrown reviewed Aug 12, 2025

View reviewed changes

core/database/foxx/api/repo_router.js Show resolved Hide resolved

AronPerez force-pushed the feat-DAPS-1515-endpoint-integration branch from b04e32d to f12d22b Compare August 13, 2025 14:57

AronPerez force-pushed the feat-DAPS-1515-legacy-repo-updates branch 2 times, most recently from e726e65 to 8e5ad6c Compare August 14, 2025 17:07

AronPerez force-pushed the feat-DAPS-1515-endpoint-integration branch from dd396d4 to 8b67bd3 Compare August 14, 2025 17:08

AronPerez force-pushed the feat-DAPS-1515-legacy-repo-updates branch from 8e5ad6c to eb93c6d Compare August 18, 2025 16:57

AronPerez force-pushed the feat-DAPS-1515-endpoint-integration branch from 4f6f23a to cde5516 Compare August 18, 2025 16:57

AronPerez self-assigned this Aug 25, 2025

JoshuaSBrown reviewed Aug 25, 2025

View reviewed changes

JoshuaSBrown reviewed Aug 26, 2025

View reviewed changes

AronPerez force-pushed the feat-DAPS-1515-legacy-repo-updates branch from eb93c6d to 69728ad Compare August 26, 2025 17:13

AronPerez force-pushed the feat-DAPS-1515-endpoint-integration branch from cde5516 to db5f941 Compare August 26, 2025 17:13

AronPerez changed the base branch from feat-DAPS-1515-legacy-repo-updates to feat-DAPS-1513-rust-js-design August 26, 2025 17:39

AronPerez force-pushed the feat-DAPS-1513-rust-js-design branch from 4d1a056 to 33c787f Compare August 27, 2025 15:49

AronPerez force-pushed the feat-DAPS-1515-endpoint-integration branch from 7007e52 to 5e9fe8f Compare August 27, 2025 15:50

AronPerez force-pushed the feat-DAPS-1513-rust-js-design branch from 33c787f to 7c755f9 Compare August 28, 2025 18:11

AronPerez added 7 commits August 28, 2025 14:11

[DAPS-1515] Address comments

4d37418

[DAPS-1515] Run prettier

d117b5a

[DAPS-1515] Readd globus and metadata

66bdb31

[DAPS-1515] Make usage of globus and metadata files

90b0a2a

[DAPS-1515] Address pr comments

50be6a3

AronPerez force-pushed the feat-DAPS-1515-endpoint-integration branch from f4f6690 to 50be6a3 Compare August 28, 2025 18:11

JoshuaSBrown assigned JoshuaSBrown and unassigned AronPerez Sep 25, 2025

[DAPS-1515] Integrate repository type system into core endpoints #1543

Are you sure you want to change the base?

[DAPS-1515] Integrate repository type system into core endpoints #1543

Uh oh!

Conversation

AronPerez commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Ticket

Description

How Has This Been Tested?

Artifacts

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AronPerez commented Aug 17, 2025

Uh oh!

AronPerez commented Aug 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AronPerez commented Jul 29, 2025 •

edited

Loading