Skip to content

New storage system (with database support) - #1391

Draft
partim wants to merge 7 commits into
mainfrom
database-storage-backend
Draft

New storage system (with database support)#1391
partim wants to merge 7 commits into
mainfrom
database-storage-backend

Conversation

@partim

@partim partim commented Aug 20, 2026

Copy link
Copy Markdown
Member

This PR introduces a revised version of the storage system for Krill.

It splits up the current key-value store into two parts: a lower layer that provides means to issue certain types of queries towards multiple types of backends, and a higher layer that implements the API of the current key-value store on top of it.

The system currently supports PostgreSQL and SQLite as SQL backends and a file system based backend compatible with the current disk backend. The intention is to drop the current dedicated memory backend in favour of using SQLite with an in-memory database.

The lower layer is implemented in terms of multiple traits that represent different types of database queries. When used, you define a marker type and implement the relevant traits for this type. For the database backends you essentially only have to provide the SQL string for the prepared statement for each backend (because they differ subtly) plus how to convert returned values into your desired output while for the file system backend you have to write a method that does the actual work.

For the statement traits, see src/common/newstore/statements.rs. For how the lower level implementation works for the key-value store, see src/common/newstore/kv.rs. (Note that src/common/newstore will become src/common/storage once it can just be dropped in for that.)

The reason for the split is that some of our use cases, most notably the task queue, require working around the limitations of a simple key-value store and would benefit from being able to use more complex database schemes.

So we decided to already consider this ability when adding database support so as to not having to re-write everything again later.


This PR is currently still a draft as it is woefully incomplete. I’ve opened it so that anyone interested can have a look already.

@partim
partim marked this pull request as draft August 20, 2026 15:21
@partim

partim commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

A couple of things to do before this can be merged (this serves as a reminder and will be extended as necessary):

  • Add locks to KeyValueStore.
  • Have the key value store check the database schema when created.
  • Refactor error types. (There are currently a lot of error types that wrap a Box<dyn error::Error>. This needs to be unified so we don’t have a box in a box in a box.)
  • Figure out how to have multiple independent instances of the in-memory store (or maybe some other store variant) for testing.
  • Add tests. (Well, d’oh.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants