New storage system (with database support) - #1391
Draft
partim wants to merge 7 commits into
Draft
Conversation
partim
marked this pull request as draft
August 20, 2026 15:21
Member
Author
|
A couple of things to do before this can be merged (this serves as a reminder and will be extended as necessary):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces a revised version of the storage system for Krill.
It splits up the current key-value store into two parts: a lower layer that provides means to issue certain types of queries towards multiple types of backends, and a higher layer that implements the API of the current key-value store on top of it.
The system currently supports PostgreSQL and SQLite as SQL backends and a file system based backend compatible with the current disk backend. The intention is to drop the current dedicated memory backend in favour of using SQLite with an in-memory database.
The lower layer is implemented in terms of multiple traits that represent different types of database queries. When used, you define a marker type and implement the relevant traits for this type. For the database backends you essentially only have to provide the SQL string for the prepared statement for each backend (because they differ subtly) plus how to convert returned values into your desired output while for the file system backend you have to write a method that does the actual work.
For the statement traits, see
src/common/newstore/statements.rs. For how the lower level implementation works for the key-value store, seesrc/common/newstore/kv.rs. (Note thatsrc/common/newstorewill becomesrc/common/storageonce it can just be dropped in for that.)The reason for the split is that some of our use cases, most notably the task queue, require working around the limitations of a simple key-value store and would benefit from being able to use more complex database schemes.
So we decided to already consider this ability when adding database support so as to not having to re-write everything again later.
This PR is currently still a draft as it is woefully incomplete. I’ve opened it so that anyone interested can have a look already.