Minor SQL Indexer access tweaks.

[saul-jb]

This PR is to start a conversation on the best way to expose some of the underlying SQLite functionality to the user, as a quick and dirty start I have:

• Changed the SQLiteIndices class methods to protected instead of private to allow access when extending the class.
• Exposed the database type to allow typing the overridden constructor externally.
• Exposed sqlite3 creation.

This allows me to override the SQLiteIndices and SQLLiteIndex to inject hooks and add a way to query the underlying SQL database.

Also what is up with the extra 'L' in SQLLiteIndex?

[saul-jb]

This is my current (very hacky) code for injecting hooks for context:

import type { Indices, IndexEngineInitProperties, Index, DeleteOptions } from '@peerbit/indexer-interface'
import type { AbstractType } from "@dao-xyz/borsh"
import { SQLiteIndices, SQLLiteIndex, create as createSQLiteIndexer } from '@peerbit/indexer-sqlite3'

type Binadable = string | Uint8Array | number

interface OverrideConfig<C extends new (...args: any[]) => any = new (...args: any[]) => any, T extends InstanceType<C> = InstanceType<C>> {
  class: C
  setupQuery?: (tableName: string, query: (sql: string, bindables: Binadable[]) => Promise<unknown[]>, index: SQLLiteIndex<T>) => void
  start?: (table: string, query: (sql: string, bindables: Binadable[]) => Promise<unknown[]>, index: SQLLiteIndex<T>) => Promise<void> | void
  stop?: (table: string, query: (sql: string, bindables: Binadable[]) => Promise<unknown[]>, index: SQLLiteIndex<T>) => Promise<void> | void
  put?: (table: string, item: T, query: (sql: string, bindables: Binadable[]) => Promise<unknown[]>, index: SQLLiteIndex<T>) => Promise<void> | void
  del?: (table: string, ids: (string | number | bigint | Uint8Array)[], query: (sql: string, bindables: Binadable[]) => Promise<unknown[]>, index: SQLLiteIndex<T>) => Promise<void> | void
}

class SQLLiteIndexOverride<C extends new (...args: any[]) => any, T extends InstanceType<C>> extends SQLLiteIndex<T> {
  constructor (
    private overrides: OverrideConfig[],
    properties: {
      scope: string[]
      db: any
      schema: AbstractType<any>
      start?: () => Promise<void> | void
      stop?: () => Promise<void> | void
    },
    options?: { iteratorTimeout?: number }
  ) {
    super(properties, options)

    this.getOverride()?.setupQuery?.(this.rootTables[0].name, this.runQuery.bind(this), this)
  }

  private getOverride(): OverrideConfig<C> | null {
    for (const override of this.overrides) {
      const schema = this.properties.schema as { __copiedFrom?: (new (...args: any[]) => any)[] }

      if (schema.__copiedFrom?.includes(override.class)) {
        return override as OverrideConfig<C>
      }
    }

    return null
  }

  private get rootName () {
    return this.rootTables[0].name
  }

  private async runQuery (sql: string, bindables: Binadable[]): Promise<unknown[]> {
    const stmt = await this.properties.db.prepare(sql)

    return await stmt.all(bindables)
  }

  async start(): Promise<void> {
    await super.start()
    await this.getOverride()?.start?.(this.rootName, this.runQuery.bind(this), this)
  }

  async stop(): Promise<void> {
    await super.stop()
    await this.getOverride()?.stop?.(this.rootName, this.runQuery.bind(this), this)
  }

  async put(value: T): Promise<void> {
    const data = value as unknown as T

    await super.put(value)
    await this.getOverride()?.put?.(this.rootName, data, this.runQuery.bind(this), this)
  }

  async del (query: DeleteOptions) {
    const ids = await super.del(query)

    await this.getOverride()?.del?.(this.rootName, ids.map(i => i.key), this.runQuery.bind(this), this)

    return ids
  }
}

export class SQLiteIndicesOverride extends SQLiteIndices implements Indices {
  constructor (
    private overrides: OverrideConfig[],
    properties: {
      scope?: string[]
      db: any
      parent?: SQLiteIndices
    }
  ) {
    super(properties)
  }

  async init<T extends Record<string, any>, NestedType>(
    properties: IndexEngineInitProperties<T, NestedType>,
  ): Promise<Index<T, NestedType>> {
    // @ts-expect-error
    const existing = this.indices.find((x) => x.schema === properties.schema)

    if (existing) {
      return existing.index
    }

    const index: Index<T, any> = new SQLLiteIndexOverride(this.overrides, {
      db: this.properties.db,
      schema: properties.schema,
      // @ts-expect-error
      scope: this._scope,
    })
    await index.init(properties)
    // @ts-expect-error
    this.indices.push({ schema: properties.schema, index })

    // @ts-expect-error
    if (!this.closed) {
      await index.start()
    }

    return index
  }

  async scope(name: string): Promise<Indices> {
    // @ts-expect-error
    if (!this.scopes.has(name)) {
      const scope = new SQLiteIndicesOverride(this.overrides, {
        // @ts-expect-error
        scope: [...this._scope, name],
        db: this.properties.db,
        parent: this,
      })

      // @ts-expect-error
      if (!this.closed) {
        await scope.start()
      }
      // @ts-expect-error
      this.scopes.set(name, scope)
      return scope
    }

    // @ts-expect-error
    const scope = this.scopes.get(name)!
    // @ts-expect-error
    if (!this.closed) {
      await scope.start()
    }
    return scope
  }
}

export const createModifiedSQLiteIndexer = (overrides: OverrideConfig[]) => async (directory?: string) => {
  const def = await createSQLiteIndexer(directory)
  const mod = new SQLiteIndicesOverride(overrides, def.properties)

  return mod
}

[marcus-pousette]

Hey!

Sorry for responding a bit slowly here.
I will come back next week and look into this.

But from a question side right now, I am curious what kind of functionality in general you want to get access to, and if so it is possible to just develop the appropiate method directly so you don't have to write your code knowing that SQL is running in the background (I say this because it would be cool if you want to swap out the underlying indexer at some point in the future)

[saul-jb]

No worries, take your time.

Yeah, it is definitely desirable to be indexer independent, but firstly I think the SQL functionality will always be a superset of what you expose through the API and it will make sense to give users a way to tap into that (without re-writing an entire SQL indexer) if they absolutely require it for niche use-cases.

I need to run JOIN queries to piece my data together; my peerbit schema looks like this:

{
  id: <ID String>
  parent: <Pointer to another ID>
  type: <Data Type>
  key: <String>
  value?: <Binary Data>
}

So I create an empty item and then point a bunch of other items with values to it; I can then piece that together on the front-end and that allows users to update different fields of an object simultaneously without conflict. I have found that I want to search for objects by a schema, this results in a SQL query that looks like this:

SELECT DISTINCT k1.parent
FROM ${rootTables[0].name} k1
JOIN ${rootTables[0].name} k2 ON k1.parent = k2.parent
JOIN ${rootTables[0].name} k3 ON k1.parent = k3.parent
WHERE
  k1.key = 'name' AND k1.type = 'string' AND
  k2.key = 'age' AND k2.type = 'number' AND
  k3.key = 'active' AND k3.type = 'boolean';

The code above was my hacky way of creating the possibility of injecting a way to access this but it's liable to break with updates.

I know this PR is a pretty poor way of exposing this but I figured you would a much better idea of the best way to provide the functionality I am looking for.

[threshold-862543]

Could this be used to for example, use other types of databases?

[saul-jb]

Could this be used to for example, use other types of databases?

This PR is about tweaking the existing SQLite database for custom hooks, you can already leverage the indexer interface to use whatever kind of database you want.

I just wanted to have some extra functionality in the existing one without having to re-write a copy of the whole thing with some minor tweaks.