Vector Search & LLM Integration

[ssddanbrown]

This is a proof of concept for some level of somewhat native LLM integration.
This manages vector indexing, searching and LLM querying, with vectors stored & queried in the local BookStack database, and an external LLM service (OpenAI, Ollama etc...) queried for embeddings & query execution using locally found vector-based results for context.

The result is a LLM generated response to user query, with a list of relevant BookStack content used for the query as reference.

Issues & Questionables

• MySQL lacks proper support. MySQL 9 supports vector columns and indexes, but you can't query distance in the community versions. MariaDB is good from 11.7.
  • Not sure how I feel about MariaDB specific features/functionality. Current implementation uses MariaDB specific functions.
• No idea how non-en/mixed language content works for these systems yet.
• Do we set official targets of support? How common is OpenAI API compatibility across other providers?
• Does embedding size vary much across difference services/models?
  • Seems to: all-minilm=384, openai:text-embedding-3-small=1536, openai:text-embedding-3-large=3072
  • Inserting 384 length embeddings into vector(1536) mariadb column seemed to result in 0 hex values. Seems like the column size has to be correct, can't insert under?
    • Response from Sergei at MariaDB on this:
      • "all vectors in a column are strictly the same size, because the distance between two vectors of different sizes is not defined. if you store 384-dim vectors in vector(1536) you can pad them with zeros and the result will be exactly the same as storing them in vector(384) except that it'll use about 5x more storage and memory"
    • Will need to provide controls/mechanism to change this.
• How is those accessed/shown in the UI? Option in the search bar maybe to toggle search type? Action/reference from the normal search page?
• What needs to be configurable? Looks like a lot of things that could be tweaked, but don't want to be maintaining loads of specific options. Need to assess what's most important.

Considerations

• This will really require a queue worker so indexing can be done in the background.
  • This does mean that search index may be behind content.
• Would need migration handling due to limited DB support, so maybe have the migration have widespread support by default but add vector parts via a command?
• For a nice UX, would probably need to have results queried via JS rather than results on page load. Not a big deal, but I generally try to have read/view operations be non-JS friendly.
• Lots of context-window/vector-size considerations to be made that have not been implemented yet.
• We'd probably want to cap the context documents (at least those shown to a user) at a certain score to avoid showing completely irrelevant content.

Todo

• Resolve above issues/considerations where needed.
• Need to ensure vectors handled on content delete/update.
• Resolve code TODOs
• Cover functionality with testing