Skip to content

Conversation

@alexcos20
Copy link
Member

@alexcos20 alexcos20 commented Sep 9, 2025

Fixes #1026

Changes proposed in this PR:

1. Allow c2d algorithm image to be build on the fly, by adding dockerfile parameter to algorithm.meta.container

You can also add additional files, using additionalDockerFiles, which has file name as a key and file content

TBD: Potential Memory Issue
If additionalDockerFiles contains large files, all are loaded into memory at once. For large files, consider streaming them into the tar archive.

Example:

{
    "command": "freeStartCompute",
    "datasets":[],
    "algorithm": {
        "meta": {
            "container": {
                "dockerfile":"FROM rocm/dev-ubuntu-24.04 as base\r\nRUN apt-get update && apt-get -y install bash curl git wget python3 build-essential\r\n\r\nCOPY . /benchmark\r\nWORKDIR /benchmark\r\nRUN pip install -r requirments.txt --break-system-packages",
                "additionalDockerFiles":{
                    "requirments.txt":"pyopencl>=2023.1\r\nnumpy>=1.24"
                }
                
            }
        },
        "envs": {
            "SEED":"test"
        }

    },
    "consumerAddress": "0x00",
    "signature": "123",
    "nonce": 1,
    "environment": "0x956cb02fee19ac8466aea8cf1901e3b771b906ed7bc1fb868c3854de6fa338e5-0x14c61b597679187ca041a8eee95d0d5167e550d313a90c29c3867997f3e4ee1c",
    "resources": [
        {
            "id": "cpu",
            "amount": 1
        },
        {
            "id":"disk",
            "amount":10000
        }
    ]
}

2. Add a new type of result: imageLog , when executing getComputeStatus:

 {
        "owner": "0x00",
        "did": null,
        "jobId": "2a66f0bf4587bbb7f2824f6feedc0137240ff2fbd8166e813dcac51fa772791d",
        "dateCreated": "1757421862.13",
        "dateFinished": null,
        "status": 40,
        "statusText": "Running algorithm ",
        "results": [
            {
                "filename": "image.log",
                "filesize": 199,
                "type": "imageLog",
                "index": 0
            }
        ]
}

Sweet: you can always call getResults and pass image.log index, to get a realtime log from image build or pull process, which is especially handy when building large images using dockerfile option

3. changes in processLoop

Removed image pulling from there, as it's time consuming and blocks entire loop.
Image pull/build is now handled in startCompute, before job is handled by main loop

@alexcos20 alexcos20 linked an issue Sep 9, 2025 that may be closed by this pull request
@alexcos20 alexcos20 marked this pull request as ready for review September 9, 2025 14:02
@alexcos20 alexcos20 self-assigned this Sep 9, 2025
@alexcos20
Copy link
Member Author

alexcos20 commented Sep 10, 2025

Now, we can add files to the building process.
Query example:

{
    "command": "freeStartCompute",
    "datasets":[],
    "algorithm": {
        "meta": {
            "container": {
                "image": "oceanprotocol/node-benchmark",
                "tag":"latest",
                "dockerfile":"FROM rocm/dev-ubuntu-24.04 as base\r\nRUN apt-get update && apt-get -y install bash curl git wget python3 build-essential\r\n\r\nCOPY . /benchmark\r\nWORKDIR /benchmark\r\nRUN pip install -r requirments.txt --break-system-packages",
                "additionalDockerFiles":{
                    "requirments.txt":"pyopencl>=2023.1\r\nnumpy>=1.24"
                }
                
            }
        },
        "envs": {
            "SEED":"test"
        }

    },
    "consumerAddress": "0x123",
    "signature": "123",
    "nonce": 1,
    "environment": "0x956cb02fee19ac8466aea8cf1901e3b771b906ed7bc1fb868c3854de6fa338e5-0x5d45d99c3964a3141fa4463f6381697f452edbd6fb1dcce866ce602e0bb43477",
    "resources": [
        {
            "id": "cpu",
            "amount": 1
        },
        {
            "id":"disk",
            "amount":10000
        }
    ]
}

Result:

image

@alexcos20
Copy link
Member Author

Ocean.js update: oceanprotocol/ocean.js#1993

Copy link
Contributor

@giurgiur99 giurgiur99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@alexcos20 alexcos20 merged commit e5626e8 into main Sep 11, 2025
13 checks passed
@alexcos20 alexcos20 deleted the feature/c2d_image_building branch September 11, 2025 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dockerfile for jobs

3 participants