Implement "uploads" endpoint by e-marchand · Pull Request #9 · 4d/4D-AIKit

e-marchand · 2025-10-08T06:28:17Z

OpenAI Uploads API - Complete Guide

Feature: Multipart File Uploads
Status: ✅ Production Ready
Last Updated: October 8, 2025

OpenAI Uploads API Documentation
Class documentation in /Documentation/Classes/
- OpenAIUploadsAPI.md
- OpenAIUpload.md
- OpenAIUploadPart.md
- OpenAIUploadParameters.md
- OpenAIUploadCompleteParameters.md
- OpenAIUploadResult.md
- OpenAIUploadPartResult.mdonal parameters for complete() function.

Property	Type	Required	Description
`md5`	Text	No	MD5 checksum for verification

Note: Mandatory parameter (part_ids) is passed as an explicit function parameter.rence)
5. Testing
6. Complete Examples

Overview

The Uploads API enables uploading large files (up to 8 GB) in multiple parts to OpenAI. This is useful for:

Fine-tuning datasets
Batch processing files
Assistant file uploads
Vision model training data
Large user data uploads

Key Features

Large file support: Up to 8 GB total
Chunked uploads: Parts up to 64 MB each
Parallel uploads: Parts can be uploaded concurrently
Flexible data types: Supports 4D.File, 4D.Blob, or BLOB
Async support: Full callback-based asynchronous operations
Error handling: Comprehensive error validation

Upload Workflow

Create Upload - Define file metadata (bytes, filename, purpose)
Add Parts - Upload file chunks (up to 64 MB each)
Complete Upload - Finalize with ordered part IDs
Use File - File ID available for other OpenAI APIs

Quick Start

Basic Example

var $client : cs.AIKit.OpenAI
var $uploadId : Text
var $partIds : Collection
var $fileId : Text

// Initialize client
$client:=cs.AIKit.OpenAI.new()
$partIds:=[]

// 1. Create upload
var $params : cs.AIKit.OpenAIUploadParameters
$params:=cs.AIKit.OpenAIUploadParameters.new()

var $result : cs.AIKit.OpenAIUploadResult
$result:=$client.uploads.create("training_data.jsonl"; 200000000; "fine-tune"; "text/jsonl"; $params)

If ($result.success)
    $uploadId:=$result.upload.id
    
    // 2. Upload parts (4 chunks of 50 MB each)
    For ($i; 1; 4)
        var $file : 4D.File
        var $partResult : cs.AIKit.OpenAIUploadPartResult
        
        $file:=Folder(fk desktop folder).file("chunk_"+String($i)+".dat")
        $partResult:=$client.uploads.addPart($uploadId; $file; cs.AIKit.OpenAIParameters.new())
        
        If ($partResult.success)
            $partIds.push($partResult.part.id)
        End if 
    End for 
    
    // 3. Complete upload
    var $completeParams : cs.AIKit.OpenAIUploadCompleteParameters
    $completeParams:=cs.AIKit.OpenAIUploadCompleteParameters.new()
    
    var $completeResult : cs.AIKit.OpenAIUploadResult
    $completeResult:=$client.uploads.complete($uploadId; $partIds; $completeParams)
    
    If ($completeResult.success)
        $fileId:=$completeResult.upload.file.id
        ALERT("Upload complete! File ID: "+$fileId)
    End if 
End if

Asynchronous Example

var $client : cs.AIKit.OpenAI
var $signal : 4D.Signal

$client:=cs.AIKit.OpenAI.new()
$signal:=New signal

// Create upload asynchronously
var $params : cs.AIKit.OpenAIUploadParameters
$params:=cs.AIKit.OpenAIUploadParameters.new()

var $options : cs.AIKit._OpenAIAsyncOptions
$options:=cs.AIKit._OpenAIAsyncOptions.new()
$options.onResponse:=Formula($signal.trigger($1.result.value))
$options.onError:=Formula($signal.trigger(Null))

CALL WORKER("upload"; Formula($client.uploads.create("dataset.jsonl"; 100000000; "batch"; "text/jsonl"; $params; $options)))

// Wait for result
$signal.wait(15)

If ($signal.signaled)
    var $result : cs.AIKit.OpenAIUploadResult
    $result:=$signal.value
    
    If ($result.success)
        // Continue with adding parts...
    End if 
End if

Implementation Details

Classes Implemented

API Layer

OpenAIUploadsAPI - Main API resource class
- Extends OpenAIAPIResource
- Provides CRUD operations for uploads

Data Models

OpenAIUpload - Upload object representation
OpenAIUploadPart - Single part representation

Parameters

OpenAIUploadParameters - Create upload parameters
OpenAIUploadCompleteParameters - Complete upload parameters

Results

OpenAIUploadResult - Wrapper for upload operations
OpenAIUploadPartResult - Wrapper for part operations

Class Organization

Located in folders.json under the "Uploads" folder:

"Uploads": {
    "classes": [
        "OpenAIUpload",
        "OpenAIUploadCompleteParameters",
        "OpenAIUploadParameters",
        "OpenAIUploadPart",
        "OpenAIUploadPartResult",
        "OpenAIUploadResult",
        "OpenAIUploadsAPI"
    ]
}

API Reference

OpenAIUploadsAPI

Main API class for managing uploads.

Methods

`create(filename : Text; bytes : Integer; purpose : Text; mime_type : Text; params : OpenAIUploadParameters [; options : _OpenAIAsyncOptions]) : OpenAIUploadResult`

Creates a new upload object.

Parameters:

filename - The name of the file to upload (required)
bytes - The number of bytes in the file (required)
purpose - The intended purpose of the file (required)
mime_type - The MIME type of the file (required)
params - Optional parameters (expires_after)
options - Optional async options (onResponse, onError)

Returns: OpenAIUploadResult with upload object

Example:

$params:=cs.AIKit.OpenAIUploadParameters.new()
// Optional: Set expiration policy
// $params.expires_after:={anchor: "created_at"; seconds: 86400}

$result:=$client.uploads.create("data.jsonl"; 50000000; "fine-tune"; "text/jsonl"; $params)

`addPart(uploadId : Text; data : Variant; params : OpenAIParameters [; options : _OpenAIAsyncOptions]) : OpenAIUploadPartResult`

Adds a part to an upload.

Parameters:

uploadId - Upload ID returned from create()
data - File data (4D.File, 4D.Blob, or BLOB)
params - Optional parameters (for async options)
options - Optional async options

Returns: OpenAIUploadPartResult with part object

Example:

$file:=File("/path/to/chunk.dat")
$params:=cs.AIKit.OpenAIParameters.new()
$partResult:=$client.uploads.addPart($uploadId; $file; $params)

`complete(uploadId : Text; part_ids : Collection; params : OpenAIUploadCompleteParameters [; options : _OpenAIAsyncOptions]) : OpenAIUploadResult`

Completes an upload.

Parameters:

uploadId - Upload ID (required)
part_ids - Ordered collection of part IDs (required)
params - Optional parameters (md5 checksum)
options - Optional async options

Returns: OpenAIUploadResult with completed upload

Example:

$params:=cs.AIKit.OpenAIUploadCompleteParameters.new()
// Optional: Set MD5 checksum for verification
// $params.md5:="d41d8cd98f00b204e9800998ecf8427e"

$result:=$client.uploads.complete($uploadId; $partIds; $params)

`cancel(uploadId : Text; params : OpenAIParameters [; options : _OpenAIAsyncOptions]) : OpenAIUploadResult`

Cancels an in-progress upload.

Parameters:

uploadId - Upload ID
params - Base parameters
options - Optional async options

Returns: OpenAIUploadResult with cancelled upload

Example:

$params:=cs.AIKit.OpenAIParameters.new()
$result:=$client.uploads.cancel($uploadId; $params)

Upload Object

Properties of an OpenAIUpload object:

Property	Type	Description
`id`	Text	Unique upload identifier
`object`	Text	Always "upload"
`status`	Text	"pending", "completed", "cancelled", or "expired"
`bytes`	Integer	Total file size in bytes
`filename`	Text	Original filename
`purpose`	Text	Upload purpose (fine-tune, assistants, etc.)
`mime_type`	Text	MIME type of the file
`created_at`	Integer	Unix timestamp of creation
`expires_at`	Integer	Unix timestamp of expiration
`file`	Object	File object (only when completed)

Upload Part Object

Properties of an OpenAIUploadPart object:

Property	Type	Description
`id`	Text	Unique part identifier
`object`	Text	Always "upload.part"
`upload_id`	Text	Parent upload ID
`created_at`	Integer	Unix timestamp of creation

Parameters

OpenAIUploadParameters

Property	Type	Required	Description
`filename`	Text	Yes	Filename to use
`purpose`	Text	Yes	fine-tune, assistants, batch, vision, user_data
`bytes`	Integer	Yes	Total file size in bytes
`mime_type`	Text	Yes	MIME type (e.g., "text/jsonl")

OpenAIUploadCompleteParameters

Property	Type	Required	Description
`part_ids`	Collection	Yes	Ordered list of part IDs
`md5`	Text	No	MD5 checksum for verification

Testing

Test Files

Two comprehensive test suites are provided:

test_openai_uploads.4dm - Synchronous tests
test_openai_uploads_async.4dm - Asynchronous tests

Test Coverage

Synchronous Tests (7 scenarios)

✅ Create Upload - Creates upload with test parameters
✅ Add Parts - Uploads 3 test chunks (File objects)
✅ Complete Upload - Completes upload with part IDs
✅ Cancel Upload - Creates and cancels an upload
✅ Upload with Blob - Tests Blob data instead of File
✅ Error: Invalid Upload ID - Tests error handling
✅ Error: Invalid Part IDs - Tests complete with wrong IDs

Asynchronous Tests (5 scenarios)

✅ Create Upload (Async) - Creates upload with formula callback
✅ Add Parts (Async) - Uploads parts sequentially (async)
✅ Complete Upload (Async) - Completes upload asynchronously
✅ Cancel Upload (Async) - Cancels upload asynchronously
✅ Callbacks - Tests onResponse/onError callbacks

Running Tests

// Run synchronous tests
test_openai_uploads

// Run asynchronous tests
test_openai_uploads_async

Test Data

Tests create temporary test data:

Folder: OpenAI_Upload_Test
3 chunk files with 5 lines each
Automatic cleanup after tests

Complete Examples

Example 1: Upload Training Data for Fine-Tuning

var $client : cs.AIKit.OpenAI
var $uploadId; $fileId : Text
var $partIds : Collection
var $totalBytes : Integer

$client:=cs.AIKit.OpenAI.new()
$partIds:=[]
$totalBytes:=0

// Prepare files
var $folder : 4D.Folder
$folder:=Folder("/path/to/training/data")

var $chunks : Collection
$chunks:=$folder.files(fk ignore invisible)

// Calculate total size
For each ($chunk; $chunks)
    $totalBytes:=$totalBytes+$chunk.size
End for each 

// 1. Create upload
var $params : cs.AIKit.OpenAIUploadParameters
$params:=cs.AIKit.OpenAIUploadParameters.new()

var $result : cs.AIKit.OpenAIUploadResult
$result:=$client.uploads.create("training_dataset.jsonl"; $totalBytes; "fine-tune"; "text/jsonl"; $params)

If ($result.success)
    $uploadId:=$result.upload.id
    
    // 2. Upload all chunks
    For each ($chunk; $chunks)
        var $partParams : cs.AIKit.OpenAIParameters
        $partParams:=cs.AIKit.OpenAIParameters.new()
        
        var $partResult : cs.AIKit.OpenAIUploadPartResult
        $partResult:=$client.uploads.addPart($uploadId; $chunk; $partParams)
        
        If ($partResult.success)
            $partIds.push($partResult.part.id)
            ALERT("Uploaded chunk "+String($partIds.length)+"/"+String($chunks.length))
        Else 
            ALERT("Error uploading chunk: "+$partResult.error.message)
            // Cancel upload on error
            $client.uploads.cancel($uploadId; cs.AIKit.OpenAIParameters.new())
            break
        End if 
    End for each 
    
    // 3. Complete upload if all parts successful
    If ($partIds.length=$chunks.length)
        var $completeParams : cs.AIKit.OpenAIUploadCompleteParameters
        $completeParams:=cs.AIKit.OpenAIUploadCompleteParameters.new()
        
        var $completeResult : cs.AIKit.OpenAIUploadResult
        $completeResult:=$client.uploads.complete($uploadId; $partIds; $completeParams)
        
        If ($completeResult.success)
            $fileId:=$completeResult.upload.file.id
            ALERT("Upload complete! File ID: "+$fileId)
            
            // Now use file ID for fine-tuning
            var $fineTuneParams : cs.AIKit.OpenAIFineTuneParameters
            $fineTuneParams:=cs.AIKit.OpenAIFineTuneParameters.new()
            $fineTuneParams.training_file:=$fileId
            $fineTuneParams.model:="gpt-4o-mini-2024-07-18"
            // ... continue with fine-tuning
        End if 
    End if 
End if

Example 2: Parallel Part Uploads

var $client : cs.AIKit.OpenAI
var $uploadId : Text
var $signals : Collection
var $partIds : Collection

$client:=cs.AIKit.OpenAI.new()
$signals:=[]
$partIds:=[]

// Create upload
var $params : cs.AIKit.OpenAIUploadParameters
$params:=cs.AIKit.OpenAIUploadParameters.new()

var $result : cs.AIKit.OpenAIUploadResult
$result:=$client.uploads.create("large_file.bin"; 500000000; "assistants"; "application/octet-stream"; $params)

If ($result.success)
    $uploadId:=$result.upload.id
    
    // Upload 10 parts in parallel
    For ($i; 1; 10)
        var $signal : 4D.Signal
        $signal:=New signal
        $signals.push($signal)
        
        var $file : 4D.File
        $file:=Folder("/path/to/chunks").file("chunk_"+String($i; "00")+".dat")
        
        var $partParams : cs.AIKit.OpenAIParameters
        $partParams:=cs.AIKit.OpenAIParameters.new()
        
        var $options : cs.AIKit._OpenAIAsyncOptions
        $options:=cs.AIKit._OpenAIAsyncOptions.new()
        $options.onResponse:=Formula($signal.trigger($1.result.value))
        $options.onError:=Formula($signal.trigger(Null))
        
        // Upload in separate worker
        CALL WORKER("upload_part_"+String($i); Formula(\
            $client.uploads.addPart($uploadId; $file; $partParams; $options)\
        ))
    End for 
    
    // Wait for all parts (30 second timeout)
    var $allComplete : Boolean
    $allComplete:=True
    
    For each ($signal; $signals)
        $signal.wait(30)
        
        If ($signal.signaled)
            var $partResult : cs.AIKit.OpenAIUploadPartResult
            $partResult:=$signal.value
            
            If ($partResult#Null) && ($partResult.success)
                $partIds.push($partResult.part.id)
            Else 
                $allComplete:=False
            End if 
        Else 
            $allComplete:=False
        End if 
    End for each 
    
    // Complete if all parts uploaded
    If ($allComplete) && ($partIds.length=10)
        var $completeParams : cs.AIKit.OpenAIUploadCompleteParameters
        $completeParams:=cs.AIKit.OpenAIUploadCompleteParameters.new()
        
        var $completeResult : cs.AIKit.OpenAIUploadResult
        $completeResult:=$client.uploads.complete($uploadId; $partIds; $completeParams)
        
        If ($completeResult.success)
            ALERT("All parts uploaded successfully!")
        End if 
    Else 
        // Cancel on failure
        $client.uploads.cancel($uploadId; cs.AIKit.OpenAIParameters.new())
        ALERT("Upload failed, cancelled.")
    End if 
End if

Example 3: Upload with Progress Tracking

var $client : cs.AIKit.OpenAI
var $uploadId : Text
var $partIds : Collection
var $totalParts; $uploadedParts : Integer

$client:=cs.AIKit.OpenAI.new()
$partIds:=[]

var $chunks : Collection
$chunks:=Folder("/path/to/chunks").files()
$totalParts:=$chunks.length
$uploadedParts:=0

// Create upload
var $params : cs.AIKit.OpenAIUploadParameters
$params:=cs.AIKit.OpenAIUploadParameters.new()

var $result : cs.AIKit.OpenAIUploadResult
$result:=$client.uploads.create("dataset.jsonl"; $totalBytes; "batch"; "text/jsonl"; $params)

If ($result.success)
    $uploadId:=$result.upload.id
    
    // Progress window
    Open form window("UploadProgress"; Plain form window)
    DIALOG("UploadProgress")
    
    For each ($chunk; $chunks)
        var $partParams : cs.AIKit.OpenAIParameters
        $partParams:=cs.AIKit.OpenAIParameters.new()
        
        var $partResult : cs.AIKit.OpenAIUploadPartResult
        $partResult:=$client.uploads.addPart($uploadId; $chunk; $partParams)
        
        If ($partResult.success)
            $partIds.push($partResult.part.id)
            $uploadedParts:=$uploadedParts+1
            
            // Update progress
            var $percentage : Real
            $percentage:=($uploadedParts/$totalParts)*100
            OBJECT SET TITLE(*; "Progress"; String($percentage; "##0")+"% ("+String($uploadedParts)+"/"+String($totalParts)+")")
        Else 
            ALERT("Error: "+$partResult.error.message)
            CANCEL
            $client.uploads.cancel($uploadId; cs.AIKit.OpenAIParameters.new())
            break
        End if 
    End for each 
    
    CLOSE WINDOW
    
    // Complete upload
    If ($partIds.length=$totalParts)
        var $completeParams : cs.AIKit.OpenAIUploadCompleteParameters
        $completeParams:=cs.AIKit.OpenAIUploadCompleteParameters.new()
        
        var $completeResult : cs.AIKit.OpenAIUploadResult
        $completeResult:=$client.uploads.complete($uploadId; $partIds; $completeParams)
        
        If ($completeResult.success)
            ALERT("Upload complete!")
        End if 
    End if 
End if

Example 4: Error Handling and Retry

var $client : cs.AIKit.OpenAI
var $uploadId : Text
var $partIds : Collection
var $maxRetries : Integer

$client:=cs.AIKit.OpenAI.new()
$partIds:=[]
$maxRetries:=3

// Create upload with error handling
var $params : cs.AIKit.OpenAIUploadParameters
$params:=cs.AIKit.OpenAIUploadParameters.new()

var $result : cs.AIKit.OpenAIUploadResult
$result:=$client.uploads.create("data.jsonl"; 100000000; "fine-tune"; "text/jsonl"; $params)

If ($result.success)
    $uploadId:=$result.upload.id
    
    var $chunks : Collection
    $chunks:=Folder("/path/to/chunks").files()
    
    // Upload with retry logic
    For each ($chunk; $chunks)
        var $uploaded : Boolean
        var $retries : Integer
        
        $uploaded:=False
        $retries:=0
        
        While (Not($uploaded)) && ($retries<$maxRetries)
            var $partParams : cs.AIKit.OpenAIParameters
            $partParams:=cs.AIKit.OpenAIParameters.new()
            
            var $partResult : cs.AIKit.OpenAIUploadPartResult
            $partResult:=$client.uploads.addPart($uploadId; $chunk; $partParams)
            
            If ($partResult.success)
                $partIds.push($partResult.part.id)
                $uploaded:=True
            Else 
                $retries:=$retries+1
                
                If ($retries<$maxRetries)
                    ALERT("Retry "+String($retries)+" for "+$chunk.name)
                    DELAY PROCESS(Current process; 60*2)  // Wait 2 seconds
                Else 
                    ALERT("Failed after "+String($maxRetries)+" retries: "+$chunk.name)
                    // Cancel entire upload
                    $client.uploads.cancel($uploadId; cs.AIKit.OpenAIParameters.new())
                    break
                End if 
            End if 
        End while 
        
        If (Not($uploaded))
            break
        End if 
    End for each 
    
    // Complete if all parts uploaded
    If ($partIds.length=$chunks.length)
        var $completeParams : cs.AIKit.OpenAIUploadCompleteParameters
        $completeParams:=cs.AIKit.OpenAIUploadCompleteParameters.new()
        
        var $completeResult : cs.AIKit.OpenAIUploadResult
        $completeResult:=$client.uploads.complete($uploadId; $partIds; $completeParams)
        
        If ($completeResult.success)
            ALERT("Upload complete!")
        End if 
    End if 
End if

Additional Resources

OpenAI Uploads API Documentation
Class documentation in /Documentation/Classes/
- OpenAIUploadsAPI.md
- OpenAIUpload.md
- OpenAIUploadPart.md
- OpenAIUploadParameters.md
- OpenAIParameters.md
- OpenAIUploadCompleteParameters.md
- OpenAIUploadResult.md
- OpenAIUploadPartResult.md

Notes

Maximum file size: 8 GB
Maximum part size: 64 MB
Parts can be uploaded in any order but must be specified in correct order when completing
Uploads expire after 1 hour if not completed
Use cancel() to clean up abandoned uploads
MD5 checksum validation is optional but recommended for large files

Implement "uploads" endpoint

f1105ae

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement "uploads" endpoint#9

Implement "uploads" endpoint#9
e-marchand wants to merge 1 commit intomainfrom
feature/uploads

e-marchand commented Oct 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

e-marchand commented Oct 8, 2025

OpenAI Uploads API - Complete Guide

Table of Contents

OpenAIUploadCompletePara## Additional Resources

Overview

Key Features

Upload Workflow

Quick Start

Basic Example

Asynchronous Example

Implementation Details

Classes Implemented

API Layer

Data Models

Parameters

Results

Class Organization

API Reference

OpenAIUploadsAPI

Methods

create(filename : Text; bytes : Integer; purpose : Text; mime_type : Text; params : OpenAIUploadParameters [; options : _OpenAIAsyncOptions]) : OpenAIUploadResult

addPart(uploadId : Text; data : Variant; params : OpenAIParameters [; options : _OpenAIAsyncOptions]) : OpenAIUploadPartResult

complete(uploadId : Text; part_ids : Collection; params : OpenAIUploadCompleteParameters [; options : _OpenAIAsyncOptions]) : OpenAIUploadResult

cancel(uploadId : Text; params : OpenAIParameters [; options : _OpenAIAsyncOptions]) : OpenAIUploadResult

Upload Object

Upload Part Object

Parameters

OpenAIUploadParameters

OpenAIUploadCompleteParameters

Testing

Test Files

Test Coverage

Synchronous Tests (7 scenarios)

Asynchronous Tests (5 scenarios)

Running Tests

Test Data

Complete Examples

Example 1: Upload Training Data for Fine-Tuning

Example 2: Parallel Part Uploads

Example 3: Upload with Progress Tracking

Example 4: Error Handling and Retry

Additional Resources

Notes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

`create(filename : Text; bytes : Integer; purpose : Text; mime_type : Text; params : OpenAIUploadParameters [; options : _OpenAIAsyncOptions]) : OpenAIUploadResult`

`addPart(uploadId : Text; data : Variant; params : OpenAIParameters [; options : _OpenAIAsyncOptions]) : OpenAIUploadPartResult`

`complete(uploadId : Text; part_ids : Collection; params : OpenAIUploadCompleteParameters [; options : _OpenAIAsyncOptions]) : OpenAIUploadResult`

`cancel(uploadId : Text; params : OpenAIParameters [; options : _OpenAIAsyncOptions]) : OpenAIUploadResult`