Automated Image Detection

[ieaves]

• Detect best base image from a compatibility matrix.
• Automatically select and execute commands with the correct image.

[AadityaAnand]

Hi, Is this issue still up for grabs, and if yes, could you please give me more context?

[ieaves]

Absolutely! RamaLama currently combines two basic inputs to create the running AI model:

1. A base_image containing a model runtime (like llama.cpp, or vLLM) compiled for a specific hardware configuration (cpu, cuda, x86 vs. arm, etc...)
2. A model file/files

We already compile base_images for llama.cpp on a variety of hardware configurations including cpu, cuda, rocm, vulkan, etc... and perform basic inspection of the users running hardware to identify supportable hardware configurations like the presence of an nvidia gpu. However, in anything other than the most typical configurations we lean on the user to specify a model runtime image to actually use.

The goal for this would be to have some mechanism for identifying which images support which capabilities and choosing an appropriate image for the user based on their full hardware configuration.

image1 -> (x86, cuda>=11 & cuda <=13, llama.cpp)
image2 -> (arm, cuda==11, vLLM)
etc...

[AadityaAnand]

Sure, will start with this.