Skip to content

Add object store #299

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Jul 23, 2025
Merged

Conversation

balvisio
Copy link
Contributor

Description

This PR addresses https://jirasw.nvidia.com/browse/AIQ-846

By Submitting this PR I confirm:

  • I am familiar with the Contributing Guidelines.
  • We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
    • Any contribution which contains commits that are not Signed-Off will not be accepted.
  • When the PR is ready for review, new or existing tests cover these changes.
  • When the PR is ready for review, the documentation is up to date with these changes.

Copy link

copy-pr-bot bot commented May 19, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@mdemoret-nv mdemoret-nv added feature request New feature or request non-breaking Non-breaking change labels Jun 10, 2025
Copy link
Collaborator

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a really good start. I have a few general suggestions:

  1. One goal of this was to allow accessing these objects from the fastapi server. I think we need to update the server configuration options to support something like this:
        # Upload static files to the object store
        async def static_files(file: UploadFile):

            object_store_client = builder.get_object_store_client(self.front_end_config.object_store)

            file_data = await file.read()

            await object_store_client.put_object(file.filename, ObjectStoreItem(data=file_data, content_type=file.content_type))

            return {"filename": file.filename}

        app.add_api_route(
            path="/static",
            endpoint=static_files,
            methods=["POST"],
        )

        # Get static files from the object store
        async def get_static_files(file_name: str):

            object_store_client = builder.get_object_store_client(self.front_end_config.object_store)

            file_data = await object_store_client.get_object(file_name)

            return FileResponse(file_data.data, media_type=file_data.content_type)

        app.add_api_route(
            path="/static/{file_name}",
            endpoint=get_static_files,
            methods=["GET"],
        )
  1. The functions in the user_report example should be generalizable and could be included in the main library since they will work with any data store.
  2. Are there other more complex data store types that we should consider to ensure the interface works. Redis? MySQL?
  3. Will need to update documentation

@balvisio balvisio force-pushed the ba/AIQ-846-add-object-store branch from 927b14f to 3a434d6 Compare July 2, 2025 22:33
@mdemoret-nv
Copy link
Collaborator

@balvisio We will need the DCO checks to pass before we can merge. Can you force push according to this: https://github.com/NVIDIA/NeMo-Agent-Toolkit/pull/299/checks?check_run_id=45251496229

Copy link
Collaborator

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking really, really good! A few more changes but they are mostly naming and documentation related.

One thing we this is missing is some unit tests on the object store interface (can test with the in-memory object store).

@willkill07
Copy link
Member

@balvisio I noticed you responded with "Done" on some of Michael's comments but I don't see a commit reflecting those changes yet. Are you still working on addressing the other comments?

To reduce your effort, I'd be happy to take over on the remaining feedback.

@balvisio
Copy link
Contributor Author

Hello @willkill07 , it is ok, I will finish addressing the last set of comments. Thank you!. I will let you know when I push the last changes.

@balvisio balvisio force-pushed the ba/AIQ-846-add-object-store branch from 3a434d6 to 2dfa308 Compare July 16, 2025 00:15
@balvisio
Copy link
Contributor Author

@mdemoret-nv @willkill07 I have addressed the review comments. Thanks!

Signed-off-by: Bruno Alvisio <[email protected]>
@balvisio balvisio force-pushed the ba/AIQ-846-add-object-store branch from 2dfa308 to 723cb6d Compare July 16, 2025 14:06
Copy link
Contributor

@lvojtku lvojtku left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please change all mentions of AIQ toolkit or AgentIQ to NeMo Agent toolkit.

mdemoret-nv and others added 9 commits July 22, 2025 16:41
Signed-off-by: Michael Demoret <[email protected]>
Signed-off-by: Michael Demoret <[email protected]>
Signed-off-by: Michael Demoret <[email protected]>
Co-authored-by: lvojtku <[email protected]>
Signed-off-by: Michael Demoret <[email protected]>
Co-authored-by: lvojtku <[email protected]>
Signed-off-by: Michael Demoret <[email protected]>
Signed-off-by: Michael Demoret <[email protected]>
Signed-off-by: Michael Demoret <[email protected]>
@mdemoret-nv
Copy link
Collaborator

/ok to test 1ca5cd4

@mdemoret-nv
Copy link
Collaborator

/ok to test 69bd707

Copy link
Collaborator

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Directly pushed changes

- Updated index.md to include links for Object Store and Object Store Provider.
- Enhanced tool_test_runner.py with mock methods for managing Object Store functionality.

Signed-off-by: Michael Demoret <[email protected]>
Signed-off-by: Michael Demoret <[email protected]>
@mdemoret-nv
Copy link
Collaborator

/ok to test e8d2317

@mdemoret-nv
Copy link
Collaborator

/ok to test c9b261d

@mdemoret-nv
Copy link
Collaborator

/ok to test b775fff

@mdemoret-nv
Copy link
Collaborator

/ok to test a9704cd

@mdemoret-nv
Copy link
Collaborator

/merge

@rapids-bot rapids-bot bot merged commit 7350945 into NVIDIA:develop Jul 23, 2025
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request non-breaking Non-breaking change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants