architecture.md

description	Understand the OpenPets desktop app, companion pets, plugin host, catalogs, SDK packages, and local agent integrations as one system.

Architecture

OpenPets is a pnpm + TypeScript monorepo for an Electron desktop companion app and a set of npm packages that let coding agents drive animated desktop pets. This doc is the one-page mental model: what runs where, how a request travels end to end, and the vocabulary used throughout the rest of the docs.

The product in one sentence

A small animated pet lives on your desktop and reacts to what your coding agent is doing - thinking, editing, waiting for permission, succeeding, failing - and can be extended with companion plugins, while pets themselves are downloadable from a public catalog.

Runtime topology

There are three runtime worlds. Keep them distinct in your head.

1. The desktop app (apps/desktop/) - an Electron process tree. The main process owns state, windows, the tray, the pet windows, the plugin runtime, and a local IPC server, plus a separate opt-in remote-control listener. This is the only long-lived process; remote control is disabled by default.
2. Agent-side integrations (packages/*) - short-lived code that runs inside or alongside a coding agent (Claude Code hooks, the MCP server, OpenCode plugin, Cursor config, Pi extension, the native OpenClaw plugin, the DSH Cordis bundle, the CLI). They translate agent activity into pet commands and send them over local IPC unless an explicit remote endpoint/token configuration selects the separate remote protocol. OpenClaw is intentionally local-only and never selects that remote path. @open-pets/dsh is the strict local-only v1 exception: it always uses local IPC and the default pet and ignores all remote configuration.
3. The public web origin (openpets.dev, source in web/) - static catalogs and asset hosting. The app fetches pet/plugin catalogs and downloads ZIPs from here. Only the data side of web/ (catalogs, ZIP hosting, pet metadata) is in scope for these docs; the marketing site/frontend is not.

coding agent  ──(hook/MCP/plugin event)──▶  @open-pets/client
                                                  │  local IPC (socket/pipe/TCP)
                                                  ▼
                                         desktop app (main process)
                                          ├─ lease manager → pet windows
                                          ├─ app state (JSON)
                                          ├─ plugin runtime + SDK bridge
                                          └─ catalog/install
                                                  │  HTTPS
                                                  ▼
                                         openpets.dev (catalogs, ZIPs on R2)

An explicitly configured remote agent uses a separate path: private IPv4 endpoint plus a paired token → @open-pets/client → the remote-control service → the default pet only. It never reads local discovery, exposes the local IPC router, or participates in LAN pet presence or leases. The v1 transport is raw unencrypted TCP and is intended only for a trusted private network or an encrypted overlay with its own ACLs; CGNAT addressing alone is not encryption.

The host provider service owns exactly three independent selections: one text profile, one STT profile, and one TTS profile. Secret credential values are resolved only from PluginSecretsStore; optional static provider header values are persisted in the local provider-profile settings, while Control Center snapshots expose header names only. Generic OpenAI-compatible text covers cloud gateways and Ollama/LM Studio/vLLM, while native Anthropic, MiniMax speech, ElevenLabs speech, system TTS, and explicit Whisper-compatible transcription retain their distinct wire contracts.

The packages, and what each is for

Package	Role	Doc
@open-pets/client	The IPC client every integration uses to talk to the app	IPC and remote control
@open-pets/cli	User-facing CLI: configure agents, manage pets, run MCP, scaffold/validate plugins	Agent integrations, Development
@open-pets/mcp	Stdio MCP server exposing openpets_status / react / say to MCP agents	Agent integrations
@open-pets/claude	Claude Code hooks + MCP/settings/memory management	Agent integrations
@open-pets/opencode	OpenCode plugin runtime + config management	Agent integrations
@open-pets/cursor	Cursor MCP config + project rules management	Agent integrations
@open-pets/pi	Pi coding-agent extension + /openpets commands	Agent integrations
@open-pets/openclaw	Native OpenClaw plugin and OpenClaw plugin lifecycle management	Agent integrations
@open-pets/agent-events	Shared, validated speech pools for agent feedback	Agent integrations
@open-pets/dsh	Strict local-only v1 Cordis bundle for DSH lifecycle reactions	Agent integrations
@open-pets/plugin-sdk	Public SDK v3 type contract + deterministic test harness	Plugin SDK v3
install-pet	Standalone pet installer (works with or without the running app)	Pets
pet-format	Tiny marker/identity type for pet packages	-

The dependency spine: every integration, including @open-pets/dsh and @open-pets/openclaw, depends on @open-pets/client; openclaw also uses @open-pets/agent-events and the OpenClaw plugin SDK as an optional peer dependency. The cli composes claude, opencode, cursor, mcp, and openclaw management. claude/opencode/pi/dsh/openclaw use curated speech for safe automatic feedback. OpenClaw management is a native OpenClaw plugin install, not an OpenPets SDK v3 catalog-plugin install. @open-pets/dsh is strict local-only v1: it always uses local IPC and the default pet and ignores remote configuration.

End-to-end flows

Host Pet Assistant loop (#138)

After PluginService.start() resolves, the desktop constructs one canonical in-memory Pet Assistant service. Each turn reads the current host text-provider provider operation and the current host-owned personality profile, discovers enabled generation-pinned capabilities, and routes validated tool calls back through PluginService. Provider codecs and the lifecycle enforce bounded context, payloads, results, timeouts, cancellation, and final output. Assistant requests do not use the plugin ctx.ai gateway.

The host composes each request in a fixed order: immutable host rules, optional curated context, a serialized owner-authored personality data block, recent bounded conversation messages, and the current provider-neutral capability definitions/results. Personality values are communication preferences only and cannot change host rules, permissions, available capabilities, or authoritative capability outcomes. The profile is persisted with app state and captured at turn start, so a Settings edit applies to the next turn without changing an already-running turn. If any structured capability outcome is rejected, unavailable, or indeterminate, the terminal user-visible response is a deterministic host-generated status summary instead of untrusted model prose; turns whose outcomes all complete retain the model response. The Control Center Conversation route consumes a host-owned, in-memory current-session projection of those canonical events. It is presentation state only and remains distinct from the host-owned local archive delivered by #149. The archive is atomic, local-only, and stores only terminal user/assistant text from the canonical shared voice/chat conversation. It retains at most 200 messages for 30 days and 512 KiB total, with a 64 KiB per-entry cap and newest entries preserved. Corrupt or malformed data is quarantined when possible, replaced with an empty archive, and never partially trusted. If the archive cannot be opened or repaired, it is disabled for that session without blocking the Pet Assistant. A most-recent archive window of at most 24 entries and 128 KiB may be added to the next assistant prompt; tool definitions/results, provider payloads, and personality data never enter that archive window. Owner delete-one/delete-all operations are exposed only through a narrow main-process bridge to the Control Center's separate local-history list/open/delete panel; the panel refreshes after a terminal turn or deletion and never clears active context. There is no semantic retrieval, summary, preferences, network synchronization, or provider call for archive reads/erasure. Provider-profile management is implemented through the host-owned Control Center bridge.

Generic host voice session and Talk controls (#147, #150)

The desktop owns a host controller/factory for one bounded generic session at a time. Activation creates a fresh VoiceAssistantSession; ending it releases its microphone reservation, and a later activation creates a new session rather than reusing the terminal object. The session composes final-only bounded STT input, the canonical Pet Assistant capability loop, and authoritative response TTS. The text, STT, and TTS profiles are snapshotted independently; the STT snapshot is captured before microphone acquisition and remains fixed through transcription. #150 adds pet-owned Talk, tray/Control Center controls, and a conservative global shortcut without changing the provider-neutral voice contract. The host supplies one Talk snapshot/event contract over IPC: host-observed session state is paired with shortcut status and its failure reason directly from the shortcut manager; the contract does not fabricate microphone device metadata. Voice turns pass a host-owned turn id into the canonical assistant service, so canonical events and voice transcript projection use explicit correlation rather than matching text. Ending voice releases voice-only resources without clearing the shared assistant conversation. The shared conversation can contribute terminal text to the host-owned local archive; Realtime protocol behavior remains separate work. Typed chat and Talk acquire the same host-owned modality lease before model or microphone work for this conversation; competing starts fail with an actionable busy error and cleanup releases the lease on terminal, end, or shutdown. Pet terminal feedback is driven only by canonical terminal events, remains visible above resumed listening for a bounded lifetime, and recognizes missing information only from an explicit structured capability outcome discriminator; the host input validator marks missing required capability fields with that discriminator before the result reaches canonical feedback.

Pet-window TTS is request-scoped. Audio and system speech retain a request id and kind in the renderer, settle replacement/stop/error/close/renderer-loss/navigation paths exactly once, and the main process rejects a lost or non-completing playback request at a bounded, duration-aware deadline. System speech is split into bounded utterance chunks while preserving the authoritative assistant text and emits one completion after the final chunk. Voice activity is rendered through a dedicated composable slot, leaving unrelated plugin display and status slots intact when voice activity clears.

Provider profile management for issue #145 is a host-owned Control Center flow: the renderer consumes redacted snapshots and explicit actions over preload while the main process owns validation, persistence, and credentials. These are the flows worth holding in memory. Each links to the doc that details it.

• Agent reaction → visible pet. Agent activity is classified into a reaction category, sent via the client over IPC, the lease manager routes it to a pet window, and the window plays the mapped animation with localized speech. See IPC and remote control and Pets.
• Remote agent reaction → default pet. A paired remote client uses the separate versioned protocol. Scope checks, bounded payloads, timeouts, and address rate limiting happen before the default-pet adapter; no lease or arbitrary target is involved. See IPC and remote control.
• Installing a pet. The app fetches catalog v3 (paginated, with a v2/fixture fallback), downloads the pet ZIP from zip.openpets.dev, validates and extracts it, and records it in app state. See Catalogs and Pets.
• Running a plugin. The plugin service loads an approved manifest (catalog or local), the runtime starts a sandboxed JS host, and the SDK bridge applies permission-checked calls to pet/schedule/storage/UI/etc. See Plugin platform and Plugin SDK v3.
• Listening through a plugin. voice.listen() performs one bounded capture in a host-owned temporary session, shows the privacy indicator only after microphone acquisition succeeds, transcribes through the configured provider, and cleans up on success, cancellation, timeout, teardown, or shutdown. It is never ambient.
• Realtime voice adapter. The host contains an optional optimized OpenAI Realtime adapter over the same Pet Assistant conversation. A hidden sandboxed renderer validates and normalizes provider events; the main process validates them again and routes bounded tool calls through the generation-pinned PetAssistantService seam. Canonical capability outcomes are returned as structured function_call_output items followed by response.create. Provider response IDs and input item IDs are carried through normalization and bound to the active canonical turn; retired response/item identities are dropped deterministically. Normalized transcripts and canonical activity/action events project into the shared Conversation surface. Provider wire events stop at this adapter; Realtime is not exposed to plugins and does not add memory or local-machine authority.
• Configuring an agent. The CLI or Control Center detects the agent, writes MCP config + hooks/rules atomically, and installs a memory/instructions file. OpenClaw is the separate native-plugin path: its status is read from the OpenClaw CLI's cold inventory and its managed install/update/enable/remove actions are issued through OpenClaw's plugin commands. See Agent integrations.
• Publishing content. Pets and plugins are packaged into versioned catalogs and ZIPs, validated, and uploaded to R2 behind openpets.dev. See Catalogs and Testing and validation.

Cross-cutting invariants

These hold everywhere; the rest of the docs assume them.

• Forward-only product direction. Move the current app forward. Do not keep legacy compat code in current runtime paths. Old released apps must not break catastrophically on versioned data, but the current app carries no legacy bloat. (From AGENTS.md.)
• Catalog v3 is the source of truth for pets; catalog v2 is legacy/fallback only. Plugin catalog v2 is active; v1 is an empty compatibility shim.
• Validate at every boundary. Catalog entries, ZIP contents, pet metadata, IPC params, and plugin manifests are all strictly validated before use.
• Atomic, safe I/O. All persisted state uses temp-write + rename; all path handling rejects traversal and symlink escapes.
• Least privilege. Renderers are sandboxed with narrow preload bridges and a strict CSP; plugins run in a permission-gated sandbox; local IPC over TCP is restricted to private/loopback addresses; remote control is separate, disabled-by-default, explicitly bound, authenticated, and scope-limited.
• Voice is bounded and visible. Listening is one-shot, one-at-a-time, explicitly cancellable, visibly indicated while a media track is live, and bounded by separate microphone-acquisition and transcription timeouts.
• Voice resource ownership is centralized. Assistant, plugin one-shot, and native Realtime lanes release their own leases/tracks; only the shared voice resource owner destroys the privacy indicator after every lane has stopped.
• Pet Assistant lifecycle is bounded. The host loop is stopped and active turns are cancelled before plugin teardown; capability handles remain pinned to the plugin generation that registered them.

Glossary

• Default pet - the always-on pet shown when enabled; persistent, not lease-bound.
• Agent pet - a pet shown on explicit agent request, routed by a lease and closed when the last lease for it is released.
• Lease - a short-lived (15s TTL) claim with heartbeat renewal that routes agent commands to a specific pet and governs agent-pet visibility. See IPC and remote control.
• Reaction - a categorical pet state (e.g. thinking, editing, waiting, success, error) that maps to a sprite animation and a speech pool. See Pets.
• Reaction → animation mapping - user-configurable table from reaction types to sprite animation states.
• Spritesheet - the spritesheet.webp grid of frames a pet animates from.
• Codex pet - a locally-developed pet imported from ~/.codex/pets/.
• Control Center - the React/Tailwind renderer UI (Dashboard, Pets, Integrations, Plugins, Settings) opened from the tray.
• SDK v3 / manifestVersion 3 - the current plugin contract. See Plugin SDK v3 and Plugin platform.
• Official plugins - the reviewed companion plugin lineup and bundling rules. See Official plugins.
• Catalog - a versioned JSON index of installable pets or plugins served from openpets.dev. See Catalogs.