Skip to content

Commit 1ff6eaf

Browse files
feat: Support upload private file (#674)
* init private support for python BE * feat: Add private file handling and upload support in FastAPI - Introduced `main.py` to set up the FastAPI application with file upload capabilities. - Created `workflow.py` to manage file reading and tool creation for uploaded files. - Updated `server.py` to include upload API configuration. - Modified chat router to handle file uploads and return server file metadata. - Refactored chat models to support new file handling structure. - Enhanced file service to manage private file storage and retrieval. * add process base64 and update examples * add readme example * fix test * feat: Add file upload support to LlamaIndexServer TS * add get_file to fileservice * refactor: Simplify file storage logic in helpers.ts * update example * attach file to user message * fix example, improve model * feat: Add file upload support and enhance chat workflow in LlamaIndexServer * remove redundant change * support agent workflow for ts * Enhance README and add file upload examples for LlamaIndex Server. Updated instructions for running examples and added new workflows for handling uploaded files. Included detailed notes on using file attachments in workflows. * update doc * update example * Enhance README with detailed instructions for file upload in chat UI. Update custom workflow to handle file attachments and modify chat router to remove unused attachment handling. Refactor create_workflow to pass attachments from chat request. * Refactor file handling in workflows by updating the create_file_tool function to accept file attachments directly. Introduce a new ServerFileResponse model for better file response handling. Update chat router to utilize the new FileUpload model for file uploads. Clean up imports and ensure consistent file attachment processing across workflows. * Enhance file handling in workflows by updating README and example files. Introduce a new `workflowFactory` structure to support file attachments, and improve the `extractFileAttachments` function for better clarity and usability. Update descriptions in tools to reflect changes in file ID handling. * fix unstoppable * chore: fix issues * add changeset * bump chat-ui * bump chat-ui for eject project --------- Co-authored-by: Marcus Schiesser <[email protected]>
1 parent a543a27 commit 1ff6eaf

File tree

35 files changed

+2857
-371
lines changed

35 files changed

+2857
-371
lines changed

.changeset/hot-moments-slide.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"@create-llama/llama-index-server": patch
3+
---
4+
5+
Add support for upload file

.changeset/wide-queens-drop.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"@llamaindex/server": patch
3+
---
4+
5+
Add support for chat upload file

packages/server/README.md

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@ The `LlamaIndexServer` accepts the following configuration options:
6060
- `workflow`: A callable function that creates a workflow instance for each request. See [Workflow factory contract](#workflow-factory-contract) for more details.
6161
- `uiConfig`: An object to configure the chat UI containing the following properties:
6262
- `starterQuestions`: List of starter questions for the chat UI (default: `[]`)
63+
- `enableFileUpload`: Whether to enable file upload in the chat UI (default: `false`). See [Upload file example](./examples/private-file/README.md) for more details.
6364
- `componentsDir`: The directory for custom UI components rendering events emitted by the workflow. The default is undefined, which does not render custom UI components.
6465
- `layoutDir`: The directory for custom layout sections. The default value is `layout`. See [Custom Layout](#custom-layout) for more details.
6566
- `llamaCloudIndexSelector`: Whether to show the LlamaCloud index selector in the chat UI (requires `LLAMA_CLOUD_API_KEY` to be set in the environment variables) (default: `false`)
@@ -71,9 +72,18 @@ See all Nextjs Custom Server options [here](https://nextjs.org/docs/app/building
7172

7273
## Workflow factory contract
7374

74-
The `workflow` provided will be called for each chat request to initialize a new workflow instance. The contract of the generated workflow must be the same as for the [Agent Workflow](https://ts.llamaindex.ai/docs/llamaindex/modules/agents/agent_workflow).
75+
The `workflow` provided will be called for each chat request to initialize a new workflow instance. For advanced use cases, you can define workflowFactory with a chatBody which include list of UI messages in the request body.
7576

76-
This means that the workflow must handle a `startAgentEvent` event, which is the entry point of the workflow and contains the following information in it's `data` property:
77+
```typescript
78+
import { type Message } from "ai";
79+
import { agent } from "@llamaindex/workflow";
80+
81+
const workflowFactory = (chatBody: { messages: Message[] }) => {
82+
...
83+
};
84+
```
85+
86+
The contract of the generated workflow must be the same as for the [Agent Workflow](https://ts.llamaindex.ai/docs/llamaindex/modules/agents/agent_workflow). This means that the workflow must handle a `startAgentEvent` event, which is the entry point of the workflow and contains the following information in it's `data` property:
7787

7888
```typescript
7989
{

packages/server/examples/README.md

Lines changed: 33 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,38 @@
11
# LlamaIndex Server Examples
22

3-
This directory contains examples of how to use the LlamaIndex Server.
3+
This directory provides example projects demonstrating how to use the LlamaIndex Server.
44

5-
## Running the examples
5+
## How to Run the Examples
66

7-
```bash
8-
export OPENAI_API_KEY=your_openai_api_key
9-
pnpm run dev
10-
```
7+
1. **Install dependencies**
118

12-
## Open browser at http://localhost:3000
9+
In the root of this directory, run:
10+
11+
```bash
12+
pnpm install
13+
```
14+
15+
2. **Set your OpenAI API key**
16+
17+
Export your OpenAI API key as an environment variable:
18+
19+
```bash
20+
export OPENAI_API_KEY=your_openai_api_key
21+
```
22+
23+
3. **Start an example**
24+
25+
Replace `<example>` with the name of the example you want to run (e.g., `private-file`):
26+
27+
```bash
28+
pnpm nodemon --exec tsx <example>/index.ts
29+
```
30+
31+
4. **Open the application in your browser**
32+
33+
Visit [http://localhost:3000](http://localhost:3000) to interact with the running example.
34+
35+
## Notes
36+
37+
- Make sure you have [pnpm](https://pnpm.io/) installed.
38+
- Each example may have its own specific instructions or requirements; check the individual example's index.ts for details.
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# Upload File Example
2+
3+
This example shows how to use the uploaded file (private file) from the user in the workflow.
4+
5+
## Prerequisites
6+
7+
Please follow the setup instructions in the [examples README](../README.md).
8+
9+
You will also need:
10+
11+
- An OpenAI API key
12+
- The `enableFileUpload` option in the `uiConfig` is set to `true`.
13+
14+
```typescript
15+
new LlamaIndexServer({
16+
// ... other options
17+
uiConfig: { enableFileUpload: true },
18+
}).start();
19+
```
20+
21+
## How to get the uploaded files in your workflow:
22+
23+
In LlamaIndexServer, the uploaded file is included in chat message annotations. You can easily get the uploaded files from chat messages using the [extractFileAttachments](https://github.com/llamaindex/llamaindex/blob/main/packages/server/src/utils/events.ts) function.
24+
25+
```typescript
26+
import { type Message } from "ai";
27+
import { extractFileAttachments } from "@llamaindex/server";
28+
29+
async function workflowFactory(reqBody: { messages: Message[] }) {
30+
const attachments = extractFileAttachments(reqBody.messages);
31+
// ...
32+
}
33+
```
34+
35+
### AgentWorkflow
36+
37+
If you are using AgentWorkflow, to provide file access to the agent, you can create a tool to read the file content. We recommend to use the `fileId` as the parameter of the tool instead of the `filePath` to avoid showing internal file path to the user. You can use the `getStoredFilePath` helper function to get the file path from the file id.
38+
39+
```typescript
40+
import { getStoredFilePath, extractFileAttachments } from "@llamaindex/server";
41+
42+
const readFileTool = tool(
43+
({ fileId }) => {
44+
// Get the file path from the file id
45+
const filePath = getStoredFilePath({ id: fileId });
46+
return fsPromises.readFile(filePath, "utf8");
47+
},
48+
{
49+
name: "read_file",
50+
description: `Use this tool with the file id to read the file content. The available file are: [${attachments.map((file) => file.id).join(", ")}]`,
51+
parameters: z.object({
52+
fileId: z.string(),
53+
}),
54+
},
55+
);
56+
```
57+
58+
**Tip:** You can either put the attachments file information to the tool description or agent's system prompt.
59+
60+
Check: [agent-workflow.ts](./agent-workflow.ts) for the full example.
61+
62+
### Custom Workflow
63+
64+
In custom workflow, instead of defining a tool, you can use the helper functions (`extractFileAttachments` and `getStoredFilePath`) to work with file attachments in your workflow.
65+
66+
Check: [custom-workflow.ts](./custom-workflow.ts) for the full example.
67+
68+
> To run custom workflow example, update the `index.ts` file to use the `workflowFactory` from `custom-workflow.ts` instead of `agent-workflow.ts`.
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
import { extractFileAttachments, getStoredFilePath } from "@llamaindex/server";
2+
import { agent } from "@llamaindex/workflow";
3+
import { type Message } from "ai";
4+
import { tool } from "llamaindex";
5+
import { promises as fsPromises } from "node:fs";
6+
import { z } from "zod";
7+
8+
export const workflowFactory = async (reqBody: { messages: Message[] }) => {
9+
const { messages } = reqBody;
10+
// Extract the files from the messages
11+
const files = extractFileAttachments(messages);
12+
const fileIds = files.map((file) => file.id);
13+
14+
// Define a tool to read the file content using the id
15+
const readFileTool = tool(
16+
({ fileId }) => {
17+
if (!fileIds.includes(fileId)) {
18+
throw new Error(`File with id ${fileId} not found`);
19+
}
20+
21+
const filePath = getStoredFilePath({ id: fileId });
22+
return fsPromises.readFile(filePath, "utf8");
23+
},
24+
{
25+
name: "read_file",
26+
description: `Use this tool with the id of the file to read the file content. Here are the available file ids: [${fileIds.join(", ")}]`,
27+
parameters: z.object({
28+
fileId: z.string(),
29+
}),
30+
},
31+
);
32+
return agent({
33+
tools: [readFileTool],
34+
systemPrompt: `
35+
You are a helpful assistant that can help the user with their file.
36+
You can use the read_file tool to read the file content.
37+
`,
38+
});
39+
};
Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
import { extractFileAttachments } from "@llamaindex/server";
2+
import { ChatMemoryBuffer, MessageContent, Settings } from "llamaindex";
3+
4+
import {
5+
agentStreamEvent,
6+
createStatefulMiddleware,
7+
createWorkflow,
8+
startAgentEvent,
9+
stopAgentEvent,
10+
workflowEvent,
11+
} from "@llamaindex/workflow";
12+
import { Message } from "ai";
13+
import { promises as fsPromises } from "node:fs";
14+
15+
const fileHelperEvent = workflowEvent<{
16+
userInput: MessageContent;
17+
fileContent: string;
18+
}>();
19+
20+
/**
21+
* This is an simple workflow to demonstrate how to use uploaded files in the workflow.
22+
*/
23+
export function workflowFactory(reqBody: { messages: Message[] }) {
24+
const llm = Settings.llm;
25+
26+
// First, extract the uploaded file from the messages
27+
const attachments = extractFileAttachments(reqBody.messages);
28+
29+
if (attachments.length === 0) {
30+
throw new Error("Please upload a file to start");
31+
}
32+
33+
// Then, add the uploaded file info to the workflow state
34+
const { withState, getContext } = createStatefulMiddleware(() => {
35+
return {
36+
memory: new ChatMemoryBuffer({ llm }),
37+
uploadedFile: attachments[attachments.length - 1],
38+
};
39+
});
40+
const workflow = withState(createWorkflow());
41+
42+
// Handle the start of the workflow: read the file content
43+
workflow.handle([startAgentEvent], async ({ data }) => {
44+
const { userInput } = data;
45+
// Prepare chat history
46+
const { state } = getContext();
47+
if (!userInput) {
48+
throw new Error("Missing user input to start the workflow");
49+
}
50+
state.memory.put({ role: "user", content: userInput });
51+
52+
// Read file content
53+
const fileContent = await fsPromises.readFile(
54+
state.uploadedFile.path,
55+
"utf8",
56+
);
57+
58+
return fileHelperEvent.with({
59+
userInput,
60+
fileContent,
61+
});
62+
});
63+
64+
// Use LLM to help the user with the file content
65+
workflow.handle([fileHelperEvent], async ({ data }) => {
66+
const { sendEvent } = getContext();
67+
68+
const prompt = `
69+
You are a helpful assistant that can help the user with their file.
70+
71+
Here is the provided file content:
72+
${data.fileContent}
73+
74+
Now, let help the user with this request:
75+
${data.userInput}
76+
`;
77+
78+
const response = await llm.complete({
79+
prompt,
80+
stream: true,
81+
});
82+
83+
// Stream the response
84+
for await (const chunk of response) {
85+
sendEvent(
86+
agentStreamEvent.with({
87+
delta: chunk.text,
88+
response: chunk.text,
89+
currentAgentName: "agent",
90+
raw: chunk.raw,
91+
}),
92+
);
93+
}
94+
sendEvent(stopAgentEvent.with({ result: "" }));
95+
});
96+
97+
return workflow;
98+
}
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
import { OpenAI, OpenAIEmbedding } from "@llamaindex/openai";
2+
import { LlamaIndexServer } from "@llamaindex/server";
3+
import { Settings } from "llamaindex";
4+
import { workflowFactory } from "./agent-workflow";
5+
// Uncomment this to use a custom workflow
6+
// import { workflowFactory } from "./custom-workflow";
7+
8+
Settings.llm = new OpenAI({
9+
model: "gpt-4o-mini",
10+
});
11+
12+
Settings.embedModel = new OpenAIEmbedding({
13+
model: "text-embedding-3-small",
14+
});
15+
16+
new LlamaIndexServer({
17+
workflow: workflowFactory,
18+
suggestNextQuestions: false,
19+
uiConfig: {
20+
enableFileUpload: true,
21+
},
22+
port: 3000,
23+
}).start();
Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
import crypto from "node:crypto";
2+
import fs from "node:fs";
3+
import path from "node:path";
4+
5+
import { type ServerFile } from "@llamaindex/server";
6+
7+
export const UPLOADED_FOLDER = "output/uploaded";
8+
9+
export async function storeFile(
10+
name: string,
11+
fileBuffer: Buffer,
12+
): Promise<ServerFile> {
13+
const parts = name.split(".");
14+
const fileName = parts[0];
15+
const fileExt = parts[1];
16+
if (!fileName) {
17+
throw new Error("File name is required");
18+
}
19+
if (!fileExt) {
20+
throw new Error("File extension is required");
21+
}
22+
23+
const id = crypto.randomUUID();
24+
const fileId = `${sanitizeFileName(fileName)}_${id}.${fileExt}`;
25+
const filepath = path.join(UPLOADED_FOLDER, fileId);
26+
const fileUrl = await saveFile(filepath, fileBuffer);
27+
return {
28+
id: fileId,
29+
size: fileBuffer.length,
30+
type: fileExt,
31+
url: fileUrl,
32+
path: filepath,
33+
};
34+
}
35+
36+
// Save document to file server and return the file url
37+
async function saveFile(filepath: string, content: string | Buffer) {
38+
if (path.isAbsolute(filepath)) {
39+
throw new Error("Absolute file paths are not allowed.");
40+
}
41+
42+
const dirPath = path.dirname(filepath);
43+
await fs.promises.mkdir(dirPath, { recursive: true });
44+
45+
if (typeof content === "string") {
46+
await fs.promises.writeFile(filepath, content, "utf-8");
47+
} else {
48+
await fs.promises.writeFile(filepath, content);
49+
}
50+
51+
const fileurl = `/api/files/${filepath}`;
52+
return fileurl;
53+
}
54+
55+
function sanitizeFileName(fileName: string) {
56+
return fileName.replace(/[^a-zA-Z0-9_-]/g, "_");
57+
}

0 commit comments

Comments
 (0)