Skip to content

feat: user-defined LLM committees - #341

Open
duncanmcqueen wants to merge 2 commits into
open-legal-products:mainfrom
duncanmcqueen:feat/model-committees
Open

feat: user-defined LLM committees#341
duncanmcqueen wants to merge 2 commits into
open-legal-products:mainfrom
duncanmcqueen:feat/model-committees

Conversation

@duncanmcqueen

@duncanmcqueen duncanmcqueen commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds user-defined model committees: a committee answers one prompt with several
models at once and has a chair model synthesize their replies into the single
answer the user sees. Committees are built under Settings > Models, stored
on the profile (user_profiles.model_committees), and selected from a
Committee group in the model picker like any other model.

Stacked on #339 — the first commit here is that PR's foundation. Retarget or
rebase once it merges.

What changed

Backend

  • lib/llm/committee.ts — members run concurrently against the ordinary
    provider layer, then the chair synthesizes. Guards against self-chairing,
    self-membership and cycles.
  • lib/userCommittees.ts — normalization for reads (malformed rows are
    dropped, never thrown, so a bad row cannot lock a user out of Settings) and
    strict validation for writes (max 8 committees, 2–8 members).
  • lib/llm/models.tsmissingCommitteeApiKeyModels / modelHasApiKey /
    resolveUsableModel: availability resolves recursively through a
    committee's members and chair.
  • lib/routerModels.tsresolveRequestedModel recognises committee ids and
    gates them on availability, taking committees the caller already loaded so
    no request reads the profile twice.
  • lib/userSettings.ts, lib/chat/streaming.ts, routes/chat.ts,
    routes/tabular.ts — committees are loaded once with the rest of the
    profile and threaded to dispatch.
  • routes/user.tsmodelCommittees on the profile (select / serialize /
    PATCH) plus GET /user/models. Profile reads get their own retry tier, so a
    database without the column keeps every other live column.
  • migrations/20260823_02_user_model_committees.sql + schema.sql.

Frontend

  • components/settings/CommitteeSettingsSection.tsx — create, edit and delete
    committees. Only models the user can actually run are offered as members.
  • ModelToggle — a Committee group, shown only when every member and the
    chair is usable.
  • mikeApi / UserProfileContext — committee types, profile state, save.

Design notes

Availability is all-or-nothing. A committee is hidden from the picker and
refused at request time unless every member and its chair is usable. Half a
committee is not a usable answer.

A deleted committee is an explicit error, not a silent fallback to another
model — quietly answering with a different model misrepresents which model
wrote the response.

No tool-calling loop. When the caller supplies tools (the main chat path
always does) they are dropped and the members are told so, rather than being
left to claim document or case-law work they could not do.

Cost. Members are separate model calls, so a committee costs roughly the
sum of its members plus the chair. Running them concurrently keeps the latency
close to the slowest member instead of the sum.

Why

Different models disagree in useful ways on legal questions. A committee makes
that disagreement visible to a synthesizing model rather than making the user
run the same prompt several times by hand.

Testing

  • backend: tsc --noEmit clean; vitest run789 passed, 25 skipped
    (+25 over feat(llm): add model registry and OpenAI-compatible provider foundation #339's foundation, covering committee dispatch, concurrency,
    cycle/self-reference guards, and committee validation).
  • frontend: tsc --noEmit clean; npm run lint 0 errors; vitest run
    564 passed, 14 failed. All 14 failures are pre-existing on main
    (mikeApi.test.ts Blob handling, useSelectedModel,
    ChatInput.modelSelection) and were confirmed against the baseline before
    these changes.
  • Migration follows the dated convention with schema.sql updated to match.

28 files changed in this commit (1681 insertions); 37 files and 3498 insertions
including #339's foundation.

@CLAassistant

CLAassistant commented Aug 15, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

duncanmcqueen and others added 2 commits August 23, 2026 11:12
Adds MIKE_MODEL_CONFIG_JSON so a deployment can declare a self-hosted or
third-party OpenAI-compatible endpoint without a code change. Declared
models resolve like any other id and are served by the existing AI SDK
provider layer via @ai-sdk/openai-compatible, so they inherit its
transport, retries and streaming rather than a second HTTP client.

Endpoints that describe tool calls in prose instead of emitting them
structured — routine in self-hosted Qwen, DeepSeek and GLM builds — are
wrapped in a language-model middleware that routes <think> prose to the
reasoning channel, suppresses tool markup in the visible text, and turns
the described calls into real tool calls. A tolerant model answering a
tool-declaring turn is served through the non-streaming endpoint, since
these models interleave markup with prose in a way a partial stream
cannot be reassembled from.

Only OpenAI-compatible endpoints are declarable: the hosted providers are
already covered by the static catalog and the router prefixes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A committee answers one prompt with several models and has a chair model
synthesize their replies into the single answer the user sees. Committees
are built under Settings > Models, stored on the profile, and selected
from a Committee group in the model picker like any other model.

Members run concurrently against the ordinary provider layer, so a
committee can mix hosted and self-hosted models freely. Committee mode has
no tool-calling loop: when the caller supplies tools the members are told
so, rather than being left to claim document work they could not do.

Availability is all-or-nothing. A committee is hidden from the picker and
refused at request time unless every member and its chair is usable, and
selecting a committee that has since been deleted is an explicit error —
quietly answering with a different model misrepresents which model wrote
the response. Cycles, self-chairing and nested committees are rejected on
save and again at request time.

Profile reads tolerate a database that has not applied the migration yet,
reporting no committees rather than failing every chat and tabular request.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@duncanmcqueen
duncanmcqueen force-pushed the feat/model-committees branch from e0219db to 1789323 Compare August 23, 2026 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants