testing-and-validation.md

description	Follow the OpenPets quality ladder for desktop tests, package contracts, plugin harnesses, production release gates, and catalog verification.

Testing and validation

OpenPets ships an Electron app, npm packages, third-party-runnable plugin code, and remotely-hosted catalogs - so "does it pass tests" is necessary but not sufficient. This doc lays out the full quality ladder: unit/behavior tests, contract tests at public boundaries, runtime checks, the plugin release validators, and catalog verification - i.e. what "production-valid" means before you ship pets, plugins, packages, or the app.

The quality ladder

From fastest/narrowest to broadest:

1. Behavior tests - unit tests of pure logic.
2. Contract tests - validate public boundaries (IPC, catalog, manifest) against fixtures so producers and consumers can't drift apart.
3. Runtime checks (check-*.ts) - assertions about packaging, CSP, SDK conformance, and integration previews that run as part of check/test.
4. Release validators - the gates that catch production-breaking mistakes the test suite alone misses (catalog/package drift, missing ZIPs, SHA mismatches, unresolved $t:).
5. Live validation - post-deploy checks against the real origin.

Run the suite with pnpm test (builds first, then each package's tests) and pnpm check (per-package typecheck + build + contract checks). See Development for the command surface.

Desktop tests

The desktop runner (apps/desktop/scripts/run-tests.mjs) orchestrates: preload syntax checks → test compilation → behavior tests → contract tests → dist checks. Three buckets:

• Behavior (apps/desktop/tests/*.test.ts): lease manager, app state, version checking, ZIP safety, Codex pets, Claude memory, reaction-animation mapping, plugin bridge/gateway guards, and voice-lifecycle.test.ts for the privacy indicator, capture cancellation/cleanup races, separate timeouts, empty transcripts, and shutdown behavior. remote-control.test.ts covers secure opt-in configuration, verifier-only persistence, authentication, scopes, malformed/oversized requests, rate limiting, rotation, revocation, canonical IPv4/CGNAT boundaries, peer normalization, socket caps/deadlines, away-pet side-effect suppression, and listener shutdown. Compiled to .test-dist/.
• Provider-profile behavior is covered by provider-profiles.test.ts, provider-service.test.ts, text-model-client.test.ts, and plugin-ai-gateway.test.ts: independent persistence/selection, URL/header boundaries, fake-endpoint routing, native versus compatible codecs, operation snapshots, redacted status, and no-fetch unsupported realtime. Tests use fake fetches and no credentials.
• Contract (apps/desktop/contracts/*.contract.ts): the public boundaries - - catalog-fixture.contract.ts - catalog validation against fixture data.
  • local-ipc-protocol.contract.ts - IPC request/response parsing (IPC and remote control).
  • remote-control-protocol.contract.ts - remote allowlist and secure configuration boundary.
  • plugin-manifest.contract.ts - manifest v1 schema, config refs, permissions, deferred features, action validation (Plugin platform).
• Runtime checks (apps/desktop/src/check-*.ts): notably
  • check-packaging-contract.ts - asserts the packaged app includes bundled official plugins as extra resources, every bundled plugin's manifest + entry exist, the pet-window CSP allows the bundled emoji font, etc. This is the guard that a packaged build is actually shippable.
  • check-opencode-desktop-setup.ts - verifies the bundled OpenCode setup preview matches expectations.

Package tests & contracts

Each package runs its own check/test. Notable contract/boundary coverage:

• packages/client/contracts/client-protocol.contract.ts - the client side of the local IPC and explicit remote-client protocols, paired with the desktop's server-side contract so both ends validate their separate shapes. Its remote fixture asserts that remote mode works without consulting local discovery.
• packages/sdk/src/check-plugin-sdk.ts - SDK conformance: compiles/runs a representative plugin against the test harness to detect drift between the published types (index.ts), the harness (testing.ts), and the desktop bridge. Changing the SDK without updating all three fails here. See Plugin SDK v3.
• packages/openclaw/src/check-openclaw.ts - native OpenClaw package contract: exact management command shapes, supported status/conflict classification, post-install planning, and payload-free non-blocking lifecycle dispatch.
• @open-pets/dsh - package artifact/load smoke: confirm the built or published artifact loads as the DSH Cordis bundle and its automatic dispatch wiring is available without model tools or MCP setup. See Agent integrations.
• packages/cursor/src/check-cursor.ts, packages/opencode checks, etc. - validate the safe config-write behavior (status classification, redaction, symlink/oversize rejection, atomic writes, uninstall preserving user entries). See Agent integrations.

Plugin testing

• Unit: each official plugin has a test.js using @open-pets/plugin-sdk/testing - fake time/events, descriptor-level assertions, no Electron. Run via pnpm plugins:test, which first runs pnpm plugins:locales (scripts/check-plugin-locales.mjs) to verify every $t:/ctx.t() key resolves. See Plugin SDK v3.

• Manifest validation: openpets plugin validate <dir> checks manifest, permissions, SDK compatibility, config field types, network hosts, asset formats/size caps, entry files, and panels - run it before packaging.

• Calendar Airmail: its deterministic harness coverage should exercise the primary-calendar reconciliation, state-appropriate connection commands, ten-minute and start deliveries, duplicate suppression, selected/default couriers, and reconnect-required cleanup. Run its plugin test alongside pnpm plugins:locales, pnpm plugins:test, and pnpm --filter @open-pets/plugin-sdk check when changing its SDK-facing behavior.

• Delivery/picker boundary: desktop bridge tests cover ui:delivery permission and lifecycle semantics; manifest validation covers declared sprite-grid options and asset references. For an Electron end-to-end smoke run, verify that the Airmail settings grid loads each bundled courier, keyboard and pointer selection persist, reduced motion is static, and a test delivery uses the selected courier without requiring any installed pet.

Plugin release validation (production gate)

plugins:check alone is not release-readiness. The dedicated validators are the production gate (scripts/validate-plugin-release.mjs):

Command	When	Catches
pnpm plugins:package	build artifacts	(produces catalog + ZIP staging)
pnpm plugins:validate-release	before deploy	unresolved $t: names/descriptions in catalog cards, missing plugin ZIPs, SHA mismatches, missing locales/en.json, missing declared assets/entry files (including courier sprites), catalog/package drift, and community plugin sidecar validation (provenance.json, submissions.json)
pnpm plugins:validate-live	after deploy/R2 upload	the same, against the live catalog + live ZIPs & live sidecars

Plugin sidecar validation

The release validator automatically loads web/public/plugins/provenance.json and web/public/plugins/submissions.json and asserts:

1. Every community plugin mapped in the catalog has a matching provenance entry.
2. All provenance entries contain valid URLs, hex SHAs (40 characters), and formatted dates.
3. Update policy is strictly limited to either safe-auto or manual-review.
4. Pending submissions are well-formed and are not also present in the installable catalog.

The full pre-ship sequence (from AGENTS.md): pnpm plugins:package → pnpm plugins:validate-release → deploy/upload → pnpm plugins:validate-live. Treat a failing validator as a hard stop - these are exactly the mistakes that 404 a plugin or render a raw $t:... to users. For the full plugin catalog release path in one command, run pnpm plugins:release; it packages, validates, publishes ZIPs, deploys the web catalog, then validates the live catalog.

Catalog verification (production gate for pets)

Pet catalogs have a parallel "doctor" run from web/ (read-only; safe anytime). The gate in brief:

Command	Adds
bun run verify:catalog	manifest integrity, artifact freshness vs manifest, on-disk assets, orphan dirs
bun run verify:catalog:remote	range-validates every ZIP referenced by the local v3 catalog against the desktop install contract
bun run verify:catalog:prod	fetches deployed v3 pages, validates every deployed ZIP, and diffs local vs prod (pending/removed)
bun run verify:catalog:all	all local, local-catalog remote, deployed-prod ZIP, and drift checks

The non-negotiable rule: never ship a catalog entry whose ZIP is missing or fails the desktop install contract. Run verify:catalog:remote before deploying and verify:catalog:prod after to confirm both the deployed catalog and its ZIPs are valid. The ZIP checks use bounded HTTP range reads of the central directory; they do not download spritesheet payloads. See Catalogs.

What "production-valid" means

Before shipping, the relevant gate must be green:

• A package change → pnpm check + pnpm test (incl. contract + conformance checks) pass; for @open-pets/dsh, the package artifact/load smoke also passes.
• An app change → desktop behavior + contract + runtime checks pass; if it touches packaging/CSP/bundled plugins, check-packaging-contract.ts passes. A delivery, trusted-asset protocol, or sprite-picker change additionally needs the desktop bridge/static checks and the targeted Electron smoke above.
• A plugin release → validate-release before deploy, validate-live after.
• A pet catalog change → verify:catalog:remote before, verify:catalog:prod after; both validate the relevant v3 ZIP archive contracts.
• Linux-specific behavior → validated on the Ubuntu VM (Development).

If a gate is skipped, say so explicitly rather than implying coverage. Contract and validator failures are signal, not noise - they encode the ways this product has broken in production before.