Open Source the UI code and/or document the UI/server protocol

[nicad]

As mentioned in another issue please document the protocol for the UI to talk to the server and, ideally, also the UI client code to make it easy to build custom interfaces.

[lmangani]

FYI the http_server community extension supports API + custom UIs without dependencies on commercial products and its community maintained.

[jraymakers]

We are considering what to open source, and also how to best share the details of the protocol.

The main reason we haven't done so yet is the effort involved, not the desire to keep anything hidden or secret.

[tobilg]

Cursor outputs the following OpenAPI v3 spec when it's parsing the HttpServer code:

openapi: 3.0.0
info:
  title: DuckDB UI HTTP Server API
  version: 1.0.0
servers:
  - url: http://localhost:{port}
    description: Local server
    variables:
      port:
        default: "8080"
paths:
  /info:
    get:
      summary: Get server information
      responses:
        '200':
          description: Server information retrieved successfully
          headers:
            Access-Control-Allow-Origin:
              schema:
                type: string
              description: CORS header
            X-DuckDB-Version:
              schema:
                type: string
              description: DuckDB version
            X-DuckDB-Platform:
              schema:
                type: string
              description: DuckDB platform
            X-DuckDB-UI-Extension-Version:
              schema:
                type: string
              description: UI extension version
          content:
            text/plain:
              schema:
                type: string
  /localEvents:
    get:
      summary: Get local events
      responses:
        '200':
          description: Local events stream
          content:
            text/event-stream:
              schema:
                type: string
  /localToken:
    get:
      summary: Get local token
      responses:
        '200':
          description: Local token retrieved successfully
          content:
            text/plain:
              schema:
                type: string
        '401':
          description: Unauthorized access
        '500':
          description: Server error
          content:
            text/plain:
              schema:
                type: string
  /ddb/interrupt:
    post:
      summary: Interrupt a connection
      responses:
        '200':
          description: Connection interrupted successfully
        '401':
          description: Unauthorized access
        '404':
          description: Connection not found
  /ddb/run:
    post:
      summary: Run a query
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                content:
                  type: string
                  description: SQL query to execute
                parameter_values:
                  type: array
                  items:
                    type: string
                  description: Parameter values for the query
      responses:
        '200':
          description: Query executed successfully
          content:
            application/octet-stream:
              schema:
                type: string
        '401':
          description: Unauthorized access
        '500':
          description: Server error
          content:
            text/plain:
              schema:
                type: string
  /ddb/tokenize:
    post:
      summary: Tokenize SQL content
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                content:
                  type: string
                  description: SQL content to tokenize
      responses:
        '200':
          description: Tokenization successful
          content:
            application/octet-stream:
              schema:
                type: string
        '401':
          description: Unauthorized access

[jraymakers]

Please note that the above AI-generated "spec" in inaccurate in several important ways. Crucially, it hallucinated a JSON format for request bodies, but this API does not use JSON.

The responses are returned in a binary format leveraging DuckDB's BinarySerializer. The above does not explain this format.

We are actively working on how best to enable others to use this protocol. Stay tuned.

[tobilg]

That’s why I hinted that it’s AI generated. Better than nothing though. I will update the respective parts. If you want let me know the other error, and I‘ll update it as well.

This was a Community effort in the mind of giving a starting point, not providing a complete solution.

[danielblignaut]

This would be a great addition! The UI forms a key piece of investigating data and discovering queries to then move into other parts of our application to visualize, etc.

Being able to have greater flexibility on the UI would be really useful to build more custom workflows and UX for our users.

[jraymakers]

We've taken the first steps in open-sourcing the UI.

In particular, we've opened the source of the duckdb-ui-client package, which is a TypeScript library implementing the client side of the HTTP protocol used to talk to the UI extension.

Also opened are the packages it depends on, duckdb-data-reader, duckdb-data-types, and duckdb-data-values, which model DuckDB data chunks, types, and values (respectively) in TypeScript, and could be useful in other contexts.

See the ts directory in this repo. Also see the brief intro to these packages and architectural overview in the README.

This is obviously just one small (but crucial) part of the UI. We'll be opening the code of more packages over time.