Add other LLM runtimes/backends than ollama

[PandorasMicrowave]

As per the title, do you have any plans on implementing the llm tagging for other runtimes/backends like vLLM, LM studio or others?

[liixini]

I had no plans up until you mentioned these softwares.

Is there any one of them you're looking at more specifically, and what feature set would you like? a 1:1 with the existing Ollama integration, e.g. model selection, prompt modification and state tracking?

[PandorasMicrowave]

I actually only setup a local llm because of your project so I'm currently using ollama.
The one I would be most interested in having it added would be llama.cpp as it's the base for ollama and I've seen some posts saying that the lack of a wrapper in llama.cpp makes it considerably more efficient than ollama.

If you consider trying to implement this, I would recommend looking into OpenAI-compatible HTTP endpoints if that's not what you're already using.

As for feature set, I was, indeed, thinking of 1:1 with the existing integration but completely understand if you don't have time or motivation for this (I might take a crack at it in a couple of months when my time frees up)

I do want to mention that I'm already very happy with how this runs currently and just thought that people who already set up a local llm without ollama might not want to duplicate

[liixini]

Haha alright, I will see what I can do :)

One of the original development targets of Skwd-wall was Ollama integration for automatic tagging due to me having a very large wallpaper collection - so this goes very well with that.

And I'm all for exploring faster tagging, as smaller models can crunch my ~4000 wallpaper collection in 2 hours on my GPU, but larger models want days for the same.