Skip to content

Conversation

@elcritch
Copy link
Contributor

Refactor sqliteds into a synchronous backend. The ThreadDatastore was then refactored to use the synchronous sqlite backend. The ThreadDatastore is based on #48 but with simplifications due to not needing to handle Futures on the backend.

The FsDs should now be pretty easy to refactor by re-using the design for SqliteDs.

Other changes:

  • SqliteDs should be fully thread-safe now barring errors
  • SqliteDs is now fully generic and can use keys of string | KeyId and data as seq[byte] | DataBuffer
  • ThreadDatastore is about ~20% shorter than previously since there's no futures on the backend
  • ThreadDatastore seems to be about ~10% faster, but just based off timing tests
  • Queries run on a single thread simplifying key conflicts and preventing queries from swamping the system

Some todos:

  • basic stress testing
  • port FsDs over as well
  • re-add tests for canceling ThreadDatastore futures
  • there's still an async SQLiteDatastore but it's query iterator would need to be wrapped into a QueryIter


type

ThreadBackendKinds* = enum
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This breaks the open close principle - https://en.wikipedia.org/wiki/Open%E2%80%93closed_principle

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's true, but I don't believe it impedes any of the goals of using TieredDs or CacheDs, etc. Those all use composition-over-inheritance which meets the needs and goals here. Plus there's plenty of (IMHO fair) criticism of the open-close principle at least when using inheritance. The linked article points out "Identify points of predicted variation and create a stable interface around them" which we have at the Datastore object level and it works well.

That said I don't necessarily like the case-type in this instance. So I made a generic version here https://github.com/codex-storage/nim-datastore/pull/54/files . If generic methods disappeared, it'd be easy create generic async procs to do the work and call them from the methods. You'd need a specific thread implementation for each ThreadedSqliteDs* = ref object of Datastore: ds: SqliteDs[...], etc, but at that point we could probably remove non-threaded SQLiteDatastore.

In general it makes sense to be careful about what actions are run on the threadpools. libuv for example only has a set enumerate list of actions it supports for file operations. Though it does add a uv_queue_work but we have nim-taskpools for those needs. We could do similar and have an enumerate set of io-thread-pool commands, which is sort of what ThreadBackendKinds is doing here.

@elcritch
Copy link
Contributor Author

Closing in favor of #55 with FsDs

@elcritch elcritch closed this Sep 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants