From 31a8a7657ad4c049525cf68f1039c9cead76f7d9 Mon Sep 17 00:00:00 2001 From: Florian Loitsch Date: Fri, 20 Mar 2026 00:28:36 +0100 Subject: [PATCH 1/7] Remove separate Artemis server, consolidate operations through broker Introduce AdminBrokerCli interface for admin operations (org/profile/member management) previously handled by the separate ArtemisServerCli. The Supabase broker implements this interface directly. - Remove Artemis dependency from all CLI commands - Remove Artemis service image embedding from envelope customization - Make --broker flag on auth commands a hidden no-op for backward compat - Deprecate --artemis flag in config (warn but continue processing) - Gut sdk list command (data source removed with Artemis server) - Validate organization at fleet init time via broker when supported - Clean up dead CONFIG-ARTEMIS-DEFAULT-KEY reference in server-config docs: map existing codebase --- .planning/codebase/ARCHITECTURE.md | 146 ++++++++++++ .planning/codebase/CONCERNS.md | 238 +++++++++++++++++++ .planning/codebase/CONVENTIONS.md | 186 +++++++++++++++ .planning/codebase/INTEGRATIONS.md | 170 ++++++++++++++ .planning/codebase/STACK.md | 108 +++++++++ .planning/codebase/STRUCTURE.md | 219 +++++++++++++++++ .planning/codebase/TESTING.md | 311 +++++++++++++++++++++++++ src/cli/broker.toit | 106 ++++----- src/cli/brokers/broker.toit | 79 +++++++ src/cli/brokers/supabase/supabase.toit | 93 +++++++- src/cli/cmds/auth.toit | 60 ++--- src/cli/cmds/config.toit | 5 +- src/cli/cmds/device-container.toit | 1 - src/cli/cmds/device.toit | 7 +- src/cli/cmds/fleet.toit | 53 +++-- src/cli/cmds/org.toit | 74 +++--- src/cli/cmds/pod.toit | 13 +- src/cli/cmds/profile.toit | 25 +- src/cli/cmds/sdk.toit | 29 +-- src/cli/cmds/serial.toit | 14 +- src/cli/cmds/utils_.toit | 25 +- src/cli/fleet.toit | 55 ++--- src/cli/pod.toit | 18 +- src/cli/server-config.toit | 2 +- tests/utils.toit | 1 - 25 files changed, 1755 insertions(+), 283 deletions(-) create mode 100644 .planning/codebase/ARCHITECTURE.md create mode 100644 .planning/codebase/CONCERNS.md create mode 100644 .planning/codebase/CONVENTIONS.md create mode 100644 .planning/codebase/INTEGRATIONS.md create mode 100644 .planning/codebase/STACK.md create mode 100644 .planning/codebase/STRUCTURE.md create mode 100644 .planning/codebase/TESTING.md diff --git a/.planning/codebase/ARCHITECTURE.md b/.planning/codebase/ARCHITECTURE.md new file mode 100644 index 00000000..74f8491b --- /dev/null +++ b/.planning/codebase/ARCHITECTURE.md @@ -0,0 +1,146 @@ +# Architecture + +**Analysis Date:** 2026-03-15 + +## Pattern Overview + +**Overall:** Layered client-server architecture with a separation between CLI management layer and device-side service layer. The device runs a scheduler-based job execution model that synchronizes state with a broker. + +**Key Characteristics:** +- CLI and service are separate binaries communicating through brokers (HTTP/Supabase) +- Device-side uses event-driven scheduler for responsive job execution +- State synchronization model where goal state is fetched from broker and containers/firmware are updated to match +- Pluggable broker architecture supporting multiple backend implementations +- Job-based execution model with priorities and runlevels for clean shutdown and state management + +## Layers + +**CLI Layer:** +- Purpose: Command-line interface for managing devices, fleets, pods, and configurations +- Location: `src/cli/` +- Contains: Command handlers, user-facing UI, broker clients, authentication +- Depends on: Broker implementations, shared constants and utilities +- Used by: End users via command-line invocation + +**Service Layer:** +- Purpose: Runs on devices to manage containers, firmware updates, and state synchronization +- Location: `src/service/` +- Contains: Job scheduler, container manager, broker connections, device state +- Depends on: Broker implementations, shared utilities, system services +- Used by: Device runtime environment + +**Shared Layer:** +- Purpose: Common utilities and constants used by both CLI and service +- Location: `src/shared/` +- Contains: Version info, server configuration, utilities, JSON diffing logic, patch tools +- Depends on: Third-party packages only +- Used by: Both CLI and service layers + +**Broker Layer:** +- Purpose: Communication abstraction between devices and management servers +- Location: `src/cli/brokers/` (CLI side) and `src/service/brokers/` (service side) +- Contains: HTTP broker implementation, Supabase broker implementation, broker interfaces +- Depends on: HTTP client libraries, encryption libraries +- Used by: CLI and service for device-broker communication + +## Data Flow + +**Device Synchronization Flow (Primary):** + +1. **Initialization**: Service starts on device via `run-artemis()` in `src/service/service.toit` +2. **Scheduler Setup**: `Scheduler` created with `Device`, `ContainerManager`, and `BrokerService` +3. **Synchronization Job**: `SynchronizeJob` connects to broker to fetch goal state +4. **Goal Processing**: Compares current device state with goal state from broker +5. **Updates Applied**: + - Container images downloaded and installed via `ContainerManager` + - Firmware updates applied via `FirmwareUpdateJob` + - Device state persisted to storage +6. **Report Back**: Device state and events reported back to broker +7. **Idle/Wait**: Scheduler waits until next job or state change + +**CLI Command Flow (Secondary):** + +1. User invokes command (e.g., `artemis device update`) +2. CLI command in `src/cli/cmds/` constructs request +3. Request routed through `Artemis` class or `BrokerCli` wrapper +4. API call made through configured broker (HTTP or Supabase) +5. Response returned and formatted for user output + +**State Management:** + +- Device state stored in `Device` class: current applications, firmware version, configuration +- Goal state received from broker as JSON map comparing containers and firmware +- State reconciliation via `json-diff` logic to compute minimal required changes +- State persistence: Job states serialized before deep sleep, restored on wake + +## Key Abstractions + +**Job System:** +- Purpose: Abstraction for schedulable, cancellable work units on device +- Examples: `src/service/jobs.toit` (base), `src/service/containers.toit` (container jobs), `src/service/synchronize.toit` (sync job) +- Pattern: Abstract `Job` class with lifecycle (`start`, `stop`, `schedule`). Subclasses implement scheduling logic. Scheduler tracks and runs jobs. + +**BrokerConnection Interface:** +- Purpose: Protocol abstraction for device-to-broker communication +- Examples: `src/service/brokers/http/http.toit` (HTTP implementation) +- Pattern: Interface defines methods for fetching goals, downloading images/firmware, reporting state and events. Implementations handle transport specifics. + +**ArtemisService Provider:** +- Purpose: Runtime API service for containers to access Artemis capabilities +- Location: `src/service/service.toit` (class `ArtemisServiceProvider`) +- Pattern: Implements Toit service protocol to expose device ID, reboot, container control methods at runtime + +**Pod:** +- Purpose: Container image with metadata and specification +- Location: `src/cli/pod.toit` and `src/cli/pod-specification.toit` +- Pattern: Contains container definition, triggers, environment variables, serialized as JSON + +**Firmware:** +- Purpose: Device firmware image with versioning and update capability +- Location: `src/cli/firmware.toit` and `src/service/firmware-update.toit` +- Pattern: Firmware identified by version string, downloaded in chunks, verified by SHA256 + +## Entry Points + +**CLI Entry Point:** +- Location: `src/cli/artemis.toit` (main function) +- Triggers: Command-line invocation with arguments +- Responsibilities: Sets up certificate roots, parses root command structure, routes to subcommands + +**Service Entry Point:** +- Location: `src/service/service.toit` (function `run-artemis`) +- Triggers: Device boots with Artemis service configured +- Responsibilities: Initializes scheduler, broker, container manager; runs main event loop + +**Artemis Server CLI (Device Management):** +- Location: `src/cli/artemis_servers/artemis-server.toit` +- Triggers: CLI needs to communicate with Artemis server API +- Responsibilities: Authenticates user, creates devices, lists SDK/service versions, downloads service images + +**Synchronizer Job (Device-side):** +- Location: `src/service/synchronize.toit` +- Triggers: Scheduled by the scheduler based on synchronization intervals +- Responsibilities: Connects to broker, fetches goal state, applies container/firmware updates, reports state + +## Error Handling + +**Strategy:** Try-catch blocks with graceful degradation. Critical errors trigger reboot or state reset. Network errors retry with backoff. + +**Patterns:** +- Broker connection failures: Retried with exponential backoff via synchronization job states (state machine with DISCONNECTED → CONNECTING → CONNECTED states) +- Container download failures: Logged with tags, container marked failed, synchronization continues +- Firmware update failures: Error reported back to broker, device remains in current state +- CLI errors: User-friendly messages via `cli.ui.abort()` or `cli.ui.error()` +- Service errors: Logged with context tags (device ID, version, job name) + +## Cross-Cutting Concerns + +**Logging:** Uses Toit `log` package. Loggers created with context via `.with-name` to track component (e.g., "scheduler", "containers", "synchronize"). Tags added to log entries for structured context. + +**Validation:** Pod specifications validated against schema in `src/cli/pod-specification.toit`. Device names and references validated in CLI command handlers. Configuration validated when loaded. + +**Authentication:** User authentication handled by `ArtemisServerCli` class. Credentials cached in local config. Tokens refreshed on demand via `ensure-authenticated` method. + +--- + +*Architecture analysis: 2026-03-15* diff --git a/.planning/codebase/CONCERNS.md b/.planning/codebase/CONCERNS.md new file mode 100644 index 00000000..e7d6ea71 --- /dev/null +++ b/.planning/codebase/CONCERNS.md @@ -0,0 +1,238 @@ +# Codebase Concerns + +**Analysis Date:** 2026-03-15 + +## Tech Debt + +**Artemis API Package Duplication:** +- Issue: Artemis API package has been temporarily copied from the official repository instead of imported as a dependency. Comments in `src/service/service.toit` (line 8-18) and `src/service/containers.toit` (line 10-18) explicitly state the API will be deleted once changes stabilize. +- Files: `src/service/service.toit`, `src/service/containers.toit`, `artemis-pkg-copy/` +- Impact: Maintenance burden when API changes - all changes must be synchronized with two copies. Risk of divergence between local copy and upstream. When deleted, import statements must be updated throughout codebase. +- Fix approach: Monitor when `toit-artemis` package reaches stable API, then switch imports from `artemis-pkg.api` back to `artemis.api` and remove `artemis-pkg-copy/` directory. Add pre-commit hook or CI check to prevent accidental modifications to copied package. + +**Container Image Bundled Detection Heuristic:** +- Issue: Determining if a container image is bundled relies on checking if `image.name != null` (see `src/service/containers.toit` lines 46-50). This is documented as "a bit of a hack" in TODO comment. +- Files: `src/service/containers.toit` (lines 46-50) +- Impact: Brittle logic that could break if API changes. Unclear contract between image name presence and bundled status. No explicit bundled property available. +- Fix approach: Add explicit bundled flag to container image API, or create dedicated method in API to check bundled status rather than inferring from name. + +**Container Lookup by GID Linear Search:** +- Issue: Finding a container by GID requires linear iteration through all jobs (see `src/service/containers.toit` lines 88-93). TODO suggests optimization via secondary map. +- Files: `src/service/containers.toit` (lines 88-93) +- Impact: O(n) lookup on potentially frequent GID lookups. If number of containers grows, performance degrades. +- Fix approach: Maintain secondary `jobs-by-gid_` map alongside existing `jobs_` map. Update map on container add/remove operations. + +**Firmware Part Matching by Index:** +- Issue: Firmware parts are matched between old and new firmware by array index rather than by name/type (see `src/service/firmware-update.toit` line 115 TODO). +- Files: `src/service/firmware-update.toit` (lines 115-116) +- Impact: Risk of corruption if firmware structure changes and parts reorder. Assumes rigid part ordering across firmware versions. +- Fix approach: Implement name/type-based matching for firmware parts. Store part metadata (name/type) and use for matching instead of array indices. + +**Container Image Installation Error Handling Gap:** +- Issue: Container loading in `src/service/containers.toit` (lines 59-63) has ambiguous error handling. If container image not found in flash, job is silently skipped but comments ask "Should we drop such an app from the current state?" +- Files: `src/service/containers.toit` (lines 52-67) +- Impact: Missing container images fail silently, potentially leaving device in inconsistent state. No logging or recovery mechanism. +- Fix approach: Explicitly log missing container images. Consider whether to treat as error vs silent skip. Possibly preserve container metadata for later recovery. + +**Container Required Status Management:** +- Issue: Required container marking logic (lines 69-77) iterates connections but assumes containers exist. No validation that required containers are actually installed. +- Files: `src/service/containers.toit` (lines 69-77) +- Impact: Can mark nonexistent containers as required without error. May cause synchronization to hang waiting for unavailable containers. +- Fix approach: Validate that all required containers exist before marking. Log warnings for missing required containers. Consider fallback strategy. + +**Container Image Reference Counting:** +- Issue: Image cleanup uses manual iteration through all jobs to check if image is still referenced (see `src/service/containers.toit` lines 135-138 TODO). Comment suggests reference counting would be better. +- Files: `src/service/containers.toit` (lines 135-138) +- Impact: O(n) cleanup on container uninstall. Scales poorly with number of containers. Easy to miss edge cases where image is still in use. +- Fix approach: Implement reference counting system where each job increments/decrements ref count on its image. Only uninstall when ref count reaches zero. + +**Synchronization Error Loop Prevention:** +- Issue: In `src/service/synchronize.toit` (lines 487-490), there's a comment about coding errors in `synchronize-step_` that could cause tight error loops with no fallback mechanism. Example log pattern shown suggests network lookup failures cascading. +- Files: `src/service/synchronize.toit` (lines 487-490) +- Impact: Coding errors in synchronization step handling could cause device to spin in error loop without recovery. Difficult to diagnose from device logs. +- Fix approach: Add explicit error classification in synchronize-step_ that distinguishes between transient network errors and coding errors. Implement exponential backoff with maximum retries. Consider safe-mode trigger for persistent errors. + +## Known Bugs + +**Unreachable Code in Connection Handling:** +- Issue: In `src/service/brokers/http/connection.toit` line 41, method `send-request` has `unreachable` statement after a call that never returns normally. The pattern catches result but then claims unreachable. +- Files: `src/service/brokers/http/connection.toit` (lines 38-41) +- Impact: Code is correct but confusing. If control flow changes, compiler won't catch the error because `unreachable` suppresses warnings. +- Fix approach: Refactor to use explicit return or exception. Remove misleading `unreachable` statement. + +**Assertion Failures on Checkpoint Misalignment:** +- Issue: Multiple assertions check checkpoint assumptions (see `src/service/firmware-update.toit` lines 112, 145). If checkpoints become misaligned due to corruption, assertions fail and crash instead of recovering gracefully. +- Files: `src/service/firmware-update.toit` (lines 112, 145) +- Impact: Firmware update can crash midway if checkpoint metadata corrupts. Device may be left in unrecoverable state. +- Fix approach: Replace assertions with proper error handling. Detect checkpoint corruption early and clear checkpoint to restart from beginning. + +## Security Considerations + +**TLS Session Caching in RTC Memory:** +- Risk: HTTP TLS session cache stored in RTC (RAM) memory that survives deep sleep but loses data on power loss (see `src/service/brokers/http/connection.toit` lines 89-92). +- Files: `src/service/brokers/http/connection.toit` (lines 89-92) +- Current mitigation: Sessions cleared on power loss, preventing replayed sessions from being persistent. TLS protocol provides forward secrecy. +- Recommendations: Document that session data is ephemeral. Consider adding integrity check to detect corrupted cached sessions. Periodically rotate session cache even if valid. + +**Firmware Checkpoint Validation:** +- Risk: Firmware update checkpoints contain old and new firmware checksums but don't verify checkpoint itself (see `src/service/firmware-update.toit` lines 23-25, 252-258). +- Files: `src/service/firmware-update.toit` +- Current mitigation: Checksums validate firmware content, device storage provides basic integrity. +- Recommendations: Add HMAC or signature to checkpoint structure to detect tampering. Validate checkpoint integrity before using it. + +**Network Retry Logic Credential Exposure:** +- Risk: HTTP connection retry logic (lines 56-73 in `src/service/brokers/http/connection.toit`) retries 3 times on 502/520/546 errors. Could theoretically retry with same credentials if broker is compromised. +- Files: `src/service/brokers/http/connection.toit` (lines 56-73) +- Current mitigation: Device headers configured per broker. Client-side secret not exposed. +- Recommendations: Add rate limiting to prevent excessive retry storms. Log all retry attempts for audit trail. + +## Performance Bottlenecks + +**Network Quarantine State Machine Complexity:** +- Problem: Network connection quarantine logic (see `src/service/network.toit` lines 28-148) involves timestamp checks and duration calculations on every connection attempt. Multiple temporary timers if network fails repeatedly. +- Files: `src/service/network.toit` (lines 28-148) +- Cause: Quarantine deadline stored as absolute monotonic microsecond timestamp, must compare against current time each attempt. No batch cleanup of expired quarantines. +- Improvement path: Implement connection quarantine using deadline queue or timer heap. Batch cleanup of expired quarantines on scheduler tick. + +**Scheduler Signal Monitor Custom Implementation:** +- Problem: Scheduler uses custom monitor-based signal mechanism (see `src/service/scheduler.toit` lines 128-137) instead of standard primitives. May not be optimally implemented. +- Files: `src/service/scheduler.toit` (lines 128-137, also TODO comment on line 128) +- Cause: Appears to be custom implementation for non-standard wait semantics. Comment suggests could use standard monitor but doesn't. +- Improvement path: Profile scheduler signal performance. Consider switching to standard Toit monitor primitives if available. Benchmark before/after. + +**Linear Iteration for GID Lookup:** +- Problem: Finding containers by GID requires linear scan (already noted in tech debt section). Called potentially during container RPC handlers. +- Files: `src/service/containers.toit` (lines 88-93) +- Cause: No secondary index by GID. +- Improvement path: Add `jobs-by-gid_` secondary map. Keep synchronized with primary jobs map. + +**Synchronization State Reporting Overhead:** +- Problem: Every synchronization step compares device state with goal state (see `src/service/synchronize.toit` line 504). Full state comparison for each step could be expensive with large state. +- Files: `src/service/synchronize.toit` (lines 486-546) +- Cause: `report-state-if-changed` function likely deep-compares entire state map. +- Improvement path: Implement incremental state tracking. Only report state deltas instead of full state. Cache last reported state. + +## Fragile Areas + +**Synchronization Step Error Handling:** +- Files: `src/service/synchronize.toit` (lines 486-546) +- Why fragile: Synchronization loop is complex with many state transitions. Comments explicitly acknowledge risk of coding errors causing tight error loops (lines 487-490). Log examples show network failures cascading into repeated errors. +- Safe modification: Add comprehensive logging at each state transition. Test error paths explicitly (network failures, timeouts, invalid responses). Consider simplifying state machine into smaller methods. +- Test coverage: Needs end-to-end tests simulating network failures at each step. Mock broker should test both success and failure paths. + +**Firmware Update Checkpoint System:** +- Files: `src/service/firmware-update.toit` +- Why fragile: Checkpoint tracks progress across firmware update but file corruption or bad timing could corrupt checkpoint state. Part matching by array index assumes firmware structure stability. Multiple writes and flushes with potential failure points. +- Safe modification: Add defensive checksum validation before using checkpoint. Handle missing/corrupt checkpoints gracefully by clearing and restarting. Add comprehensive logging of checkpoint lifecycle. +- Test coverage: Test checkpoint corruption scenarios. Test firmware updates interrupted at each checkpoint. Test part reordering in firmware structure. + +**Container Manager Image Lifecycle:** +- Files: `src/service/containers.toit` (lines 33-150) +- Why fragile: Image installation, uninstallation, and bundled status tracking have implicit assumptions about image names and availability. Manual iteration for reference counting. Silent failures on missing images. +- Safe modification: Add validation at each image operation. Explicit logging of image state changes. Consider immutable image metadata structures. Add consistency checks on startup. +- Test coverage: Test missing container images at load time. Test concurrent install/uninstall of same image. Test bundled image protection. + +**Network Manager Connection Quarantine:** +- Files: `src/service/network.toit` (lines 27-150) +- Why fragile: Quarantine deadline logic based on monotonic time comparisons. Multiple ways quarantine could persist incorrectly (time skew, stored value overflow). Iterates connections multiple times in sort. +- Safe modification: Add safeguards against time inconsistencies. Use saturating arithmetic for deadline calculations. Test quarantine expiration explicitly. +- Test coverage: Test time-based quarantine expiration. Test multiple connection failures and recovery. Test quarantine with network transitions. + +**Recovery URL Selection:** +- Files: `src/service/synchronize.toit` (lines 640-641, 652-671) +- Why fragile: Recovery URL picked randomly (line 641). If recovery service is temporarily down, no fallback to try others. Query result cached but minimal validation of response format. +- Safe modification: Validate recovery service response structure before using. Implement retry logic for recovery queries. Log all recovery attempts. +- Test coverage: Test recovery service unavailability. Test malformed recovery response. Test fallback to primary broker. + +## Scaling Limits + +**Container Lookup Performance:** +- Current capacity: O(n) lookup where n = number of containers. Practical limit ~100-1000 containers before noticeable latency. +- Limit: When containers >1000, GID lookups become observable bottleneck during container RPC calls. Synchronization could stall. +- Scaling path: Implement secondary GID index as noted in tech debt. Consider sharding containers if scale exceeds 10,000. + +**State Reporting Frequency:** +- Current capacity: Full state comparison on each synchronization step. With state size <10KB works fine. +- Limit: With large fleets (>10K devices) reporting full state repeatedly, cloud could see excessive traffic. State size growing with container configs could exceed network buffers. +- Scaling path: Implement state delta reporting. Compress state representation. Batch multiple state reports. + +**Firmware Update Bandwidth:** +- Current capacity: Binary patching works efficiently for moderate firmware updates (<100MB). Checkpoint system prevents total loss but adds I/O overhead. +- Limit: Very large firmware images (>500MB) could exhaust device storage for intermediate files. Checkpoint system adds latency to firmware writes. +- Scaling path: Stream firmware in smaller chunks. Implement progressive patching. Consider delta-sync for incremental updates. + +**Quarantine List Memory:** +- Current capacity: Network quarantine list likely <100 entries. Linear search acceptable. +- Limit: With complex network switching topology, quarantine list could grow large. Linear iteration becomes observable. +- Scaling path: Use time-indexed data structure (timer heap). Batch cleanup of expired entries. + +## Dependencies at Risk + +**Toit Language Evolution:** +- Risk: Codebase extensively uses Toit language features including custom monitors, task cancellation, and service providers. These could change in future versions. +- Impact: Major version upgrades of Toit SDK could require substantial refactoring, particularly scheduler and service provider code. +- Migration plan: Monitor Toit SDK changelog. Maintain compatibility shim layer for language features if possible. Plan major refactors around SDK upgrade cycles. + +**HTTP/TLS Library Stability:** +- Risk: HTTP client implementation in broker connection depends on `http` package. TLS session caching relies on undocumented RTC memory buckets. +- Impact: HTTP library changes could affect connection retry logic. TLS session format changes could break cached sessions. +- Migration plan: Monitor HTTP package updates. Abstract HTTP client creation into service provider. Test TLS session migration path explicitly. + +## Missing Critical Features + +**Firmware Rollback Recovery:** +- Problem: Firmware update can fail leaving device in incomplete state. Device has rollback capability but no automated recovery triggers it. Manual intervention required. +- Blocks: Can't reliably deploy faulty firmware updates. Devices can get stuck in validation-pending state. +- Recommendation: Implement automatic rollback trigger after N failed sync attempts post-update. Add metadata to track rollback history. + +**Container Migration Tool:** +- Problem: No tooling to migrate containers between devices or fleets. Container data is ephemeral. +- Blocks: Can't easily redistribute load or backup container state. Disaster recovery requires manual redeployment. +- Recommendation: Build container export/import tool. Document data migration patterns. + +**Network Failover Metrics:** +- Problem: No visibility into why network connections fail or why one connection preferred over another. +- Blocks: Hard to diagnose network configuration issues. Can't optimize connection priority based on actual performance. +- Recommendation: Track and report per-connection success metrics. Log network selection decisions. + +## Test Coverage Gaps + +**Firmware Update Edge Cases:** +- What's not tested: Checkpoint corruption, firmware download interruption at each part boundary, part reordering in firmware structure, concurrent firmware updates +- Files: `src/service/firmware-update.toit` +- Risk: Firmware updates could fail silently or corrupt device. No recovery from corruption scenarios. +- Priority: High + +**Synchronization Error Scenarios:** +- What's not tested: Network errors at each step of synchronization, broker unavailability during image download, timeout handling during state reporting, recovery server fallback +- Files: `src/service/synchronize.toit` +- Risk: Device could get stuck in error loops or miss updates during network issues. +- Priority: High + +**Container Lifecycle Management:** +- What's not tested: Missing container images at boot, concurrent install/uninstall of same image, bundled image protection, container reference counting +- Files: `src/service/containers.toit` +- Risk: Container state inconsistencies, orphaned images, incorrect cleanup. +- Priority: Medium + +**Network Quarantine System:** +- What's not tested: Quarantine deadline expiration, multiple connection failures, connection switching during quarantine, quarantine list memory growth +- Files: `src/service/network.toit` +- Risk: Connections could remain quarantined indefinitely or not quarantine properly. +- Priority: Medium + +**Watchdog Timer Integration:** +- What's not tested: Watchdog creation timeout (line 228), watchdog feeding during long operations, watchdog without service availability +- Files: `src/service/synchronize.toit` (lines 223-237) +- Risk: Watchdog could timeout due to bugs in watchdog integration rather than actual hangs. +- Priority: Medium + +**Task Cancellation Handling:** +- What's not tested: Synchronization cancellation during different states, container startup cancellation, proper cleanup on task.cancel +- Files: `src/service/synchronize.toit` (line 434) +- Risk: Incomplete cleanup on cancellation could leave locks held or connections open. +- Priority: Low + +--- + +*Concerns audit: 2026-03-15* diff --git a/.planning/codebase/CONVENTIONS.md b/.planning/codebase/CONVENTIONS.md new file mode 100644 index 00000000..a78406f3 --- /dev/null +++ b/.planning/codebase/CONVENTIONS.md @@ -0,0 +1,186 @@ +# Coding Conventions + +**Analysis Date:** 2026-03-15 + +## Naming Patterns + +**Files:** +- Kebab-case for file names: `json-diff.toit`, `pod-registry.toit`, `server-config.toit` +- Test files use suffix `-test.toit` or `-test-slow.toit`: `channel-test.toit`, `cmd-pod-delete-test.toit` +- No file extension variations; all source files are `.toit` + +**Functions:** +- Kebab-case for function names: `read-all`, `test-open`, `create-pods`, `ensure-authenticated` +- Helper/private functions use underscore suffix: `compute-cache-key_`, `install-root-certificates_`, `generate-envelope-path_` +- Test functions use `test-` prefix: `test-open`, `test-send`, `test-simple`, `test-neutering` +- Command handler functions use verb prefix: `sign-up`, `sign-in`, `create-auth-commands`, `ensure-available-artemis-service` + +**Variables:** +- Kebab-case for local variables and parameters: `tmp-dir`, `spec-ids`, `fleet-root`, `organization-id` +- Private instance fields use underscore suffix: `tmp-dir_`, `cache-key_`, `test-ui_`, `ders-already-installed_` +- Constants use SCREAMING-KEBAB-CASE: `DEFAULT-CAPACITY`, `MAGIC-NAME_`, `TEST-ORGANIZATION-UUID` +- UUID variables use `-id` or `-uuid` suffix: `organization-id`, `device-id`, `TEST-ORGANIZATION-UUID` + +**Types:** +- PascalCase for class names: `TestExit`, `TestPrinter`, `TestHumanPrinter`, `TestJsonPrinter`, `TestUi`, `TestCli` +- PascalCase for interface names: `Authenticatable`, `BrokerCli`, `ServerConfig` +- Private class fields use underscore suffix: `test-ui_`, `json_`, `quiet_`, `name_` + +## Code Style + +**Formatting:** +- No explicit formatter detected (no .prettierrc file) +- Consistent indentation of 2 spaces observed throughout codebase +- Line continuations use natural indentation +- Method/function definitions on single line with parameters indented on next lines + +**Linting:** +- No explicit linter configuration files detected (no eslint or similar) +- Code follows conventional Toit patterns with consistent style + +## Import Organization + +**Order:** +1. Standard library imports (system, core) +2. External package imports (cli, encoding, crypto, http, etc.) +3. Relative imports from parent packages (`..` imports) +4. Relative imports from sibling packages (`.` imports) +5. Export declarations + +**Examples from codebase:** +```toit +// File: src/cli/cli.toit +import certificate-roots +import cli show * +import core as core +import host.pipe show stderr +import io + +import .cmds.auth +import .cmds.config +import .cmds.device + +import ..shared.version +``` + +```toit +// File: src/cli/brokers/broker.toit +import cli show Cli +import host.file +import encoding.json +import net +import uuid show Uuid + +import ..auth +import ..config +import .supabase +import .http.base +``` + +**Path Aliases:** +- Direct imports of modules by name without aliases typically +- Aliasing used when name conflicts or for clarity: `import encoding.json as json-encoding` +- Re-exports using `show *` when module provides public API + +## Error Handling + +**Patterns:** +- Throw string literals with error descriptions: `throw "Unknown broker type"` +- Use assert for invariant checks: `assert: root-cmd.check; true` +- Try-finally blocks for cleanup operations: + ```toit + try: + block.call tmp-dir + finally: + directory.rmdir --recursive tmp-dir + ``` +- Try blocks with exception unwinding for control flow: + ```toit + exception = catch --unwind=(: not expect-exit-1 or (not allow-exception and it is not TestExit)): + artemis-pkg.main args --cli=run-cli + ``` +- Null checks using `if not var_:` pattern + +## Logging + +**Framework:** `log` standard library module used for structured logging + +**Patterns:** +- Logging is sparse in source code, mainly used in service/device code +- No universal logging pattern enforced across codebase +- Print statements used for CLI output via `cli.ui.emit` or `core.print` +- Test output via stdout/stderr captured in TestUi class + +## Comments + +**When to Comment:** +- Comments document non-obvious behavior and design decisions +- Interfaces and public functions have documentation comments +- Complex logic has inline comments explaining intent +- TODOs marked with `// TODO(author):` pattern + +**JSDoc/TSDoc:** +- Not applicable to Toit language +- Block comments using `/**` and `*/` for documentation: + ```toit + /** + Responsible for allowing the Artemis CLI to talk to Artemis services on devices. + */ + interface BrokerCli implements Authenticatable: + ``` +- Parameter documentation in comments: + ```toit + /** + The block is called with a $DeviceDetailed as argument: + The block must return a new goal state which replaces the actual goal state. + */ + update-goal --device-id/Uuid [block] -> none + ``` + +## Function Design + +**Size:** Functions are typically 5-50 lines, with test functions often longer due to test setup and assertions + +**Parameters:** +- Named parameters using `--parameter-name/type` syntax +- Block parameters with `[block]` or `[block/block-type]` syntax +- Required parameters without defaults +- Optional parameters with `?` type modifier: `--email/string?` + +**Return Values:** +- Explicit return types in function signatures: `-> string`, `-> List`, `-> none`, `-> bool` +- Functions return values implicitly (last expression) +- Null returns for operations that complete without producing values + +**Example patterns:** +```toit +// Function with named parameters and block +update-goal --device-id/Uuid [block] -> none + +// Function with optional parameters +constructor --quiet/bool=true --json/bool=false + +// Function with list parameters +update-goals --device-ids/List --goals/List -> none + +// Helper function returning computed value +cache-key -> string: + if not cache-key_: + cache-key_ = base64-lib.encode --url-mode (sha1.sha1 compute-cache-key_) + return cache-key_ +``` + +## Module Design + +**Exports:** +- Explicit export statements: `export Device` +- Modules export types, interfaces, and top-level functions +- Private implementation details use underscore suffix convention + +**Barrel Files:** +- Not commonly used; imports are specific and direct +- Relative imports reference exact modules needed + +--- + +*Convention analysis: 2026-03-15* diff --git a/.planning/codebase/INTEGRATIONS.md b/.planning/codebase/INTEGRATIONS.md new file mode 100644 index 00000000..0dc4eb35 --- /dev/null +++ b/.planning/codebase/INTEGRATIONS.md @@ -0,0 +1,170 @@ +# External Integrations + +**Analysis Date:** 2026-03-15 + +## APIs & External Services + +**Supabase (Primary Backend):** +- Service: Supabase Backend-as-a-Service (PostgREST API, Auth, Storage) + - SDK/Client: toit-supabase ^0.3.1 (`github.com/toitware/toit-supabase`) + - Authentication: JWT tokens with anon key + - Environment variables: `SUPABASE_URL`, `SUPABASE_ANON_KEY` + - Configuration: `ServerConfigSupabase` class in `src/shared/server-config.toit` + - Usage locations: + - `src/cli/brokers/supabase/supabase.toit` - CLI broker implementation + - `src/cli/artemis_servers/supabase/supabase.toit` - Artemis server API client + - `src/cli/utils/supabase.toit` - Utility functions + +**HTTP Brokers (Alternative Backend):** +- Service: Custom Toit HTTP broker protocol + - SDK/Client: pkg-http ^2.11.0 (`github.com/toitlang/pkg-http`) + - Configuration: `ServerConfigHttp` class in `src/shared/server-config.toit` + - Usage locations: + - `src/cli/brokers/http/base.toit` - HTTP broker implementation + - `src/cli/artemis_servers/http/base.toit` - HTTP server client + - `src/service/brokers/http` - Device-side HTTP broker + +**NTP Time Service:** +- Service: Network Time Protocol for system time synchronization + - SDK/Client: pkg-ntp ^1.1.0 (`github.com/toitlang/pkg-ntp`) + - Purpose: Ensuring accurate time on devices + - Usage: Time-critical device operations + +## Data Storage + +**Databases:** +- Provider: Supabase (PostgreSQL 15) + - Connection: Supabase REST API over HTTP/HTTPS + - Client: toit-supabase ^0.3.1 + - Schemas exposed: + - `public` - Main public schema + - `storage` - File storage metadata + - `graphql_public` - GraphQL API schema + - `toit_artemis` - Artemis-specific schema + - Configuration file: `supabase_artemis/supabase/config.toml` + - Migrations: Located in `supabase_artemis/supabase/migrations/` + - Seed data: `supabase_artemis/supabase/seed.sql` + +**File Storage:** +- Supabase Storage buckets + - Access: Through Supabase REST API + - Size limit: 50 MiB per file + - Support for public and private buckets + - Referenced in Edge Function `b/index.ts` for firmware and image uploads/downloads + +**Caching:** +- Configuration caching: Local filesystem in `~/.cache/artemis/` (device-specific) +- Server connection caching uses cache keys based on host/port/path hashing + +## Authentication & Identity + +**Auth Provider:** +- Supabase Auth (Email/Password and OAuth providers) + - Implementation: Supabase authentication service + - Methods supported: + - Email/password signup and sign-in (`sign-up`, `sign-in` with credentials) + - OAuth provider sign-in (Google, GitHub, etc.) - via Supabase OAuth providers + - JWT token-based authentication with 1-hour expiry (configurable) + - Session management: Browser-based with redirect URL handling + - OAuth Redirect URL: `https://toit.io/auth` for production + +**Authorization:** +- Row-Level Security (RLS) policies in PostgreSQL +- Organization-based access control +- Device ownership verification through organization membership +- Role-based access: User, Member, and organization-specific roles + +## Monitoring & Observability + +**Error Tracking:** +- Not detected as a dedicated service integration +- Error handling through application logging + +**Logs:** +- Console logging via `import log` in Toit code +- Supabase Edge Function logs available through Supabase dashboard +- Local development logs directed to console + +## CI/CD & Deployment + +**Hosting:** +- Supabase Cloud (primary) - API and database hosting +- Self-hosted option supported for local development and testing +- Public broker instance: `supabase.co` domain (e.g., `ezxwpyeoypvnnldpdotx.supabase.co`) +- Private Artemis instance: `artemis-api.toit.io` + +**Deployment:** +- Edge Functions deployed via Supabase CLI: `supabase functions deploy` +- Database migrations via Supabase CLI: `supabase db push` +- Service images managed through `tools/service_image_uploader/uploader.toit` + +**Build/Test Infrastructure:** +- Local Supabase instances: `make start-supabase` / `make start-supabase-no-config` +- Test targets: + - Unit tests: `make test` + - Serial tests: `make test-serial` + - Supabase integration tests: `make test-supabase` + +## Environment Configuration + +**Required env vars:** +- `SUPABASE_URL` - Supabase project URL (from config) +- `SUPABASE_ANON_KEY` - Supabase anonymous key for public access +- `ARTEMIS_CONFIG` - Path to local artemis configuration (default: `~/.config/artemis-dev/config`) +- `TOIT_PKG_AUTO_SYNC` - Whether to auto-sync packages (CMake option, ON by default) +- `DEFAULT_SDK_VERSION` - SDK version for compilation +- `ARTEMIS_GIT_VERSION` - Version string for builds (auto-computed from git) + +**Test Environment Vars:** +- `ARTEMIS_HOST` - Artemis server host (production: `artemis-api.toit.io`) +- `ARTEMIS_ANON` - Anonymous JWT token for Artemis server +- `ARTEMIS_TEST_HOST` - Test Supabase instance host +- `ARTEMIS_TEST_ANON` - Test Supabase instance anonymous key + +**Secrets location:** +- `.env` files (not committed) - Development secrets +- Supabase project settings - Production secrets +- Makefile JWT tokens (test credentials only, not for production) + +## Webhooks & Callbacks + +**Incoming:** +- Supabase Edge Function endpoint: `/functions/v1/b` + - POST endpoint receiving binary-encoded commands + - Commands handled: + - Device goal updates (`COMMAND_UPDATE_GOAL_`) + - Device state reporting (`COMMAND_REPORT_STATE_`) + - Event reporting (`COMMAND_REPORT_EVENT_`) + - Firmware/image downloads (`COMMAND_DOWNLOAD_`, `COMMAND_DOWNLOAD_PRIVATE_`) + - Pod registry operations (10+ commands for pod management) + - Authentication: JWT Bearer token in Authorization header or anon key + +**Outgoing:** +- Device -> Broker API calls for: + - State synchronization + - Event reporting + - Goal fetching + - Firmware/image downloading +- No webhook-style callbacks to external systems detected + +## Data Formats + +**Communication Protocols:** +- Binary protocol for device-broker communication (command-based) + - Command byte followed by JSON or binary payload + - Handled in Edge Function: `supabase_artemis/supabase/functions/b/index.ts` + - HTTP request body contains binary command + payload + +**REST API:** +- PostgREST API for database table operations +- supabase.rpc() for stored procedure calls +- Supabase Storage API for file operations + +**Serialization:** +- JSON for REST API payloads +- Binary/ArrayBuffer for firmware and image data +- Base64 encoding for certificate storage and transmission + +--- + +*Integration audit: 2026-03-15* diff --git a/.planning/codebase/STACK.md b/.planning/codebase/STACK.md new file mode 100644 index 00000000..c7ae8ff4 --- /dev/null +++ b/.planning/codebase/STACK.md @@ -0,0 +1,108 @@ +# Technology Stack + +**Analysis Date:** 2026-03-15 + +## Languages + +**Primary:** +- Toit v2.0.0-alpha.190 (development SDK version) - Core application logic, CLI, service, brokers, and tests +- TypeScript - Supabase Edge Functions for broker API endpoints + +**Secondary:** +- SQL - Database schemas and migrations for Supabase PostgreSQL +- CMake - Build system configuration +- Bash - Build scripts and development tooling + +## Runtime + +**Environment:** +- Toit Runtime (custom managed runtime for IoT/embedded systems) +- Deno - Runtime for Supabase Edge Functions (TypeScript) +- PostgreSQL 15 - Database backend + +**Package Manager:** +- Toit Package Manager (toit pkg) +- Lockfile: `package.lock` (present in root) + +## Frameworks + +**Core:** +- Toit Artemis Framework ^0.1.1 - Device and broker management system +- Toit CLI Framework ^2.6.0 - Command-line interface building +- Toit Supabase ^0.3.1 - Supabase client and authentication + +**Build/Dev:** +- CMake 3.23+ - Build orchestration +- Ninja - Build executor (used by CMake) +- Supabase CLI - Local development database and edge function management + +**Protocols & Networking:** +- Toit HTTP ^2.11.0 - HTTP client and server +- Toit NTP ^1.1.0 - Network Time Protocol for time synchronization +- TLS/HTTPS - Secure communication with root certificate support + +## Key Dependencies + +**Critical:** +- toit-supabase ^0.3.1 - Supabase client library with authentication and database access +- toit-artemis ^0.1.1 - Core Artemis device management library +- pkg-http ^2.11.0 - HTTP transport layer for broker communication +- toit-cert-roots ^1.11.0 - Root certificates for TLS/HTTPS connections + +**Infrastructure:** +- pkg-cli ^2.6.0 - CLI framework for command-line tools +- pkg-fs ^2.3.1 - Filesystem access +- pkg-host ^1.16.2 - Host system integration +- toit-watchdog ^1.4.1 - Watchdog timer functionality +- toit-partition-table-esp32 ^1.4.0 - ESP32 partition table management +- toit-semver ^1.1.0 - Semantic versioning utilities +- pkg-ar ^1.4.1 - Archive handling +- artemis-pkg-copy (local path) - Local artemis package copy + +**Development/Tooling:** +- snapshot (local path: tools/snapshot) - Snapshot building utility + +## Configuration + +**Environment:** +- Toit configuration stored in `~/.config/artemis-dev/config` (configurable via `ARTEMIS_CONFIG`) +- Broker credentials and configuration stored in local config files +- Support for both HTTP and Supabase broker configurations + +**Build:** +- `CMakeLists.txt` - Main build configuration +- `package.yaml` - Root package manifest with dependencies +- `Makefile` - Development workflow automation +- `supabase_artemis/supabase/config.toml` - Local Supabase configuration +- `public/supabase_broker/supabase/config.toml` - Broker Supabase configuration + +**Build Configuration Files:** +- PostgreSQL version pinned to 15 in `supabase_artemis/supabase/config.toml` +- API schemas: `public`, `storage`, `graphql_public`, `toit_artemis` +- Storage limit: 50MiB per file +- JWT expiry: 3600 seconds (1 hour) + +## Platform Requirements + +**Development:** +- Toit executable (toit CLI) +- CMake 3.23+ +- Ninja build system +- Supabase CLI (for local database development) +- Docker (for Supabase local development) +- GNU Make +- Git + +**Supported Hardware:** +- ESP32 (primary embedded target) +- ESP32-QEMU (for testing) +- Host systems (x86_64, ARM) + +**Production:** +- Supabase hosting infrastructure (cloud or self-hosted) +- PostgreSQL 15+ database +- Deno runtime (for Edge Functions) + +--- + +*Stack analysis: 2026-03-15* diff --git a/.planning/codebase/STRUCTURE.md b/.planning/codebase/STRUCTURE.md new file mode 100644 index 00000000..2aff4e65 --- /dev/null +++ b/.planning/codebase/STRUCTURE.md @@ -0,0 +1,219 @@ +# Codebase Structure + +**Analysis Date:** 2026-03-15 + +## Directory Layout + +``` +artemis/ +├── src/ # Main source code (Toit) +│ ├── cli/ # CLI tool +│ │ ├── artemis.toit # Main entry point +│ │ ├── cli.toit # CLI root command setup +│ │ ├── cmds/ # Command implementations +│ │ ├── artemis_servers/ # Artemis server API client +│ │ ├── brokers/ # Broker client implementations +│ │ ├── utils/ # CLI utilities +│ │ └── *.toit # Core CLI modules (config, device, fleet, pod, etc.) +│ ├── service/ # Device-side runtime service +│ │ ├── service.toit # Main service entry point +│ │ ├── scheduler.toit # Job scheduler +│ │ ├── containers.toit # Container management +│ │ ├── brokers/ # Broker connection implementations +│ │ ├── jobs.toit # Base job abstraction +│ │ ├── synchronize.toit # State synchronization job +│ │ ├── firmware-update.toit # Firmware update logic +│ │ ├── device.toit # Device state representation +│ │ ├── storage.toit # Persistent storage +│ │ └── *.toit # Supporting modules +│ └── shared/ # Shared code (CLI and service) +│ ├── version.toit # Version constants +│ ├── constants.toit # Global constants (commands) +│ ├── server-config.toit # Broker configuration +│ ├── json-diff.toit # JSON difference computation +│ └── utils/ # Shared utilities +├── tests/ # Test files +│ ├── *-test.toit # Individual test files +│ ├── gold/ # Expected output files for tests +│ └── spec_extends_tests/ # Test extensions +├── supabase_artemis/ # Supabase edge functions and migrations +│ └── supabase/ +│ ├── functions/ # Edge function code +│ ├── migrations/ # Database migrations +│ └── snippets/ # Code snippets +├── public/ # Public documentation and schemas +│ ├── docs/ # Documentation +│ ├── examples/ # Example configurations +│ └── schemas/ # JSON schemas for pod specs +├── tools/ # Utility tools +│ ├── http_servers/ # Test HTTP servers +│ ├── service_image_uploader/ # Upload service images +│ ├── snapshot/ # Snapshot tool +│ ├── lan_ip/ # LAN IP discovery +│ └── windows_installer/ # Windows installer +├── auth/ # Authentication utilities +├── benchmarks/ # Performance benchmarks +├── recovery/ # Recovery tools +├── artemis-pkg-copy/ # Temporary copy of artemis package +├── build/ # CMake build directory (generated) +├── .github/ # GitHub workflows and configuration +└── .planning/ # GSD planning documents +``` + +## Directory Purposes + +**src/cli:** +- Purpose: Command-line interface for managing devices, fleets, pods, and organizations +- Contains: Entry point, command handlers, broker clients, configuration management +- Key files: `artemis.toit` (CLI orchestrator), `cli.toit` (command structure), `cmds/` (individual commands) + +**src/service:** +- Purpose: Device-side runtime that manages containers and synchronization +- Contains: Scheduler, job implementations, container/firmware management, broker connections +- Key files: `service.toit` (entry point), `scheduler.toit` (job orchestration), `containers.toit` (app management), `synchronize.toit` (state sync) + +**src/shared:** +- Purpose: Code shared between CLI and service layers +- Contains: Version strings, command constants, server configuration, utilities +- Key files: `version.toit` (versioning), `constants.toit` (command codes), `server-config.toit` (broker config) + +**src/cli/brokers:** +- Purpose: CLI-side broker implementations for communicating with servers +- Contains: HTTP broker, Supabase broker, request/response handling +- Key files: `broker.toit` (interface), `http/base.toit` (HTTP implementation), `supabase/supabase.toit` (Supabase implementation) + +**src/service/brokers:** +- Purpose: Device-side broker implementations for communicating with management servers +- Contains: Connection handling, goal state fetching, state reporting +- Key files: `broker.toit` (interface), `http/http.toit` (HTTP implementation) + +**tests:** +- Purpose: Test suite for CLI commands and core functionality +- Contains: Command output tests, synchronization tests, JSON diff tests +- Key files: `*-test.toit` (individual tests), `gold/` (expected outputs for verification) + +**supabase_artemis:** +- Purpose: Supabase backend infrastructure (database and edge functions) +- Contains: Database schema migrations, serverless functions for API endpoints +- Key files: `migrations/` (database schema), `functions/` (API endpoints) + +## Key File Locations + +**Entry Points:** +- `src/cli/artemis.toit`: CLI main entry point and version handling +- `src/service/service.toit`: Device service entry point, exports `run-artemis` function +- `src/cli/cli.toit`: CLI command structure and routing + +**Configuration:** +- `src/shared/server-config.toit`: Broker connection configuration (HTTP/Supabase) +- `src/cli/config.toit`: User configuration management (profiles, brokers, cache) +- `package.yaml`: Toit package dependencies + +**Core Logic:** +- `src/service/scheduler.toit`: Job scheduler that drives device operation +- `src/service/containers.toit`: Container lifecycle management +- `src/service/synchronize.toit`: State synchronization with broker +- `src/cli/artemis.toit`: Device manager from CLI perspective +- `src/cli/fleet.toit`: Fleet management operations + +**Testing:** +- `tests/cmd-fleet-status-test.toit`: Fleet command test +- `tests/synchronizer.toit`: Synchronization logic test +- `tests/gold/`: Expected command output files + +## Naming Conventions + +**Files:** +- Toit source files: `lowercase-with-hyphens.toit` +- Executable or tool files: `lowercase-with-hyphens` (no extension) +- Test files: `*-test.toit` or `*_test.toit` (ending in -test or _test) +- Generated files: `*.generated.toit` or `version.toit.in` (template) + +**Directories:** +- Module directories: `lowercase-with-hyphens/` (e.g., `artemis_servers`, `brokers`) +- Command implementations: Inside `cmds/` with command name (e.g., `device.toit`, `fleet.toit`) +- Test support: `gold/` for expected outputs, `spec_extends_tests/` for test utilities + +**Classes:** +- PascalCase for classes: `ContainerManager`, `SynchronizeJob`, `ArtemisServerCli` +- Abstract base classes: `Job`, `TaskJob`, `BrokerConnection`, `BrokerService` + +**Functions/Methods:** +- snake-case with hyphens: `run-artemis`, `connect-network_`, `ensure-authenticated` +- Private methods: trailing underscore `_` before method name (e.g., `connected-artemis-server_`) +- Getter methods: simple names (e.g., `runlevel`, `is-running`) + +**Constants:** +- ALL-CAPS with hyphens: `RUNLEVEL-NORMAL`, `STATE-SYNCHRONIZED`, `COMMAND-UPDATE-GOALS_` +- Maps of constants: `ARTEMIS-COMMAND-TO-STRING`, `BROKER-COMMAND-TO-STRING` + +**Variables:** +- snake-case with hyphens: `max-offline-time`, `job-states`, `images_` +- Field names: lowercase: `name`, `id`, `tasks_` + +## Where to Add New Code + +**New CLI Command:** +1. Create handler in `src/cli/cmds/` (e.g., `src/cli/cmds/new-command.toit`) +2. Implement `create-new-command-commands() -> List` function +3. Import and call from `src/cli/cli.toit` in main function +4. Add command tests to `tests/` with corresponding gold output in `tests/gold/` + +**New Service Job:** +1. Create in `src/service/` (e.g., `src/service/new-job.toit`) +2. Extend `Job` or `TaskJob` abstract class from `src/service/jobs.toit` +3. Implement required methods: `is-running`, `schedule`, `start`, `stop` +4. Import and add to scheduler via `scheduler.add-job` in `src/service/service.toit` +5. Store/restore state via `scheduler-state` property if needed + +**New Broker Implementation:** +- CLI side: Create in `src/cli/brokers/` (e.g., `src/cli/brokers/custom/custom.toit`) +- Service side: Create in `src/service/brokers/` (e.g., `src/service/brokers/custom/custom.toit`) +- Implement `BrokerConnection` interface +- Add factory method in broker constructor in respective layer +- Update configuration to support new broker type + +**Shared Utilities:** +- General utilities: `src/shared/utils/utils.toit` or `src/shared/utils/specific-util.toit` +- Constants: Add to `src/shared/constants.toit` +- Protocols: Define interfaces in appropriate module or new dedicated file + +**Tests:** +- New test file: `tests/my-feature-test.toit` +- Expected output: `tests/gold/cmd-my-feature-test/` (directory with output files) +- Import test utilities from existing test files as reference + +## Special Directories + +**build/** +- Purpose: CMake build output directory (generated at build time) +- Generated: Yes +- Committed: No (in .gitignore) +- Contains: Compiled binaries, build artifacts, test executables + +**tests/gold/** +- Purpose: Expected output files for command tests (golden files) +- Generated: No (manually curated) +- Committed: Yes +- Usage: Tests compare actual output against files in this directory + +**supabase_artemis/supabase/** +- Purpose: Supabase infrastructure as code +- Generated: No +- Committed: Yes +- Structure: `migrations/` (numbered SQL files), `functions/` (edge functions) + +**.packages/** +- Purpose: Cached package dependencies (managed by Toit package manager) +- Generated: Yes (via `toit pkg` commands) +- Committed: No (in .gitignore) + +**artemis-pkg-copy/** +- Purpose: Temporary copy of the Artemis package API from public repository +- Generated: No (static copy) +- Committed: Yes +- Status: Marked for deletion when public package API stabilizes + +--- + +*Structure analysis: 2026-03-15* diff --git a/.planning/codebase/TESTING.md b/.planning/codebase/TESTING.md new file mode 100644 index 00000000..80d278bd --- /dev/null +++ b/.planning/codebase/TESTING.md @@ -0,0 +1,311 @@ +# Testing Patterns + +**Analysis Date:** 2026-03-15 + +## Test Framework + +**Runner:** +- CMake-based test runner using ctest +- Toit language runtime executes tests via `toit run` +- Config: `/home/flo/work/artemis/tests/CMakeLists.txt` + +**Assertion Library:** +- `expect` standard library module for assertions +- Provides functions like `expect-equals`, `expect-null`, `expect-throw`, `expect-bytes-equal`, `expect-identical` + +**Run Commands:** +```bash +make test # Run all tests +make test-serial # Run serial tests only +make test-supabase # Run Supabase-specific tests +``` + +CMake test execution: +```bash +cd build && ninja check # Run all tests +cd build && ninja check_serial # Serial tests +cd build && ninja check_supabase # Supabase tests +``` + +## Test File Organization + +**Location:** +- Tests located in `/home/flo/work/artemis/tests/` directory +- Co-located with main source but in separate `tests` directory +- Test files live at same hierarchy level as source modules + +**Naming:** +- Pattern: `*-test.toit` for regular tests +- Pattern: `*-test-slow.toit` for slow/long-running tests +- Pattern: `serial-*` for tests that must run sequentially +- Examples: `channel-test.toit`, `cmd-pod-delete-test.toit`, `supabase-artemis-broker-policies-test.toit` + +**Structure:** +``` +tests/ +├── *-test.toit # Standard unit/integration tests +├── *-test-slow.toit # Slow tests with longer timeouts +├── serial-*.toit # Serial tests (resource locks) +├── gold/ # Gold files for CLI output comparison +├── utils.toit # Test utilities and fixtures +├── CMakeLists.txt # Test configuration +└── .packages/ # Vendored dependencies +``` + +## Test Structure + +**Suite Organization:** + +Tests use a `main` entry point that spawns task(s) and may install/uninstall service providers: + +```toit +// File: tests/channel-test.toit +main: + provider := TestServiceProvider + provider.install + spawn:: test + provider.uninstall --wait + +test: + test-open "small" 32 * 1024 + test-open "medium" 64 * 1024 + 3.repeat: test-simple "fisk" + test-neutering "hest" + test-full "fisk" +``` + +**Patterns:** + +1. **Setup/Teardown:** + - `main` function handles initialization + - `spawn::` for concurrent test tasks + - Service provider `install`/`uninstall` for resource management + - Try-finally blocks for cleanup: + ```toit + try: + // test code + list.do: channel.send it + finally: + channel.close + ``` + +2. **Assertion Pattern:** + ```toit + expect-equals expected actual + expect-null value + expect-bytes-equal expected-bytes actual-bytes + expect-throw "ERROR_MESSAGE": function-call + expect-identical obj1 obj2 + ``` + +3. **CLI Test Pattern (from utils.toit):** + ```toit + run-gold test-name description args --ignore-spacing=false --expect-exit-1=false + ``` + Compares CLI output against gold files stored in `gold/` directory. + +4. **Test Helpers:** + ```toit + with-tmp-directory [block] // Create temp directory for test + with-tmp-config-cli [block] // Create test CLI with config + with-fleet [block] // Fleet testing context + with-server [block] // Server testing context + ``` + +## Mocking + +**Framework:** Manual mocking using test doubles and custom implementations + +**Patterns:** + +1. **Test Doubles:** + ```toit + // From tests/utils.toit + class TestExit: + + class TestPrinter extends cli-pkg.Printer: + print_ str/string: + test-ui_.stdout += "$str\n" + + class TestUi extends cli-pkg.Ui: + stdout/string := "" + stderr/string := "" + quiet_/bool + ``` + +2. **Service Provider Mocking:** + ```toit + provider := TestServiceProvider + provider.install + spawn:: test + provider.uninstall --wait + ``` + +3. **Capturing Output:** + ```toit + ui := TestUi --quiet=quiet --json=json + run-cli := cli.with --ui=ui + // Run code that uses ui + output := ui.stdout + ``` + +**What to Mock:** +- CLI output and printing (use TestUi) +- Service providers for isolated component testing +- File system operations (use temporary directories) +- External HTTP/Supabase servers (pre-configured with test fixtures) + +**What NOT to Mock:** +- Core language features +- Standard library functions +- Data structures (ByteArray, List, Map) +- File I/O when actual files needed for integration tests + +## Fixtures and Factories + +**Test Data:** + +Constants and fixture creation in `tests/utils.toit`: + +```toit +/** test@example.com is an admin of the $TEST-ORGANIZATION-UUID. */ +TEST-EXAMPLE-COM-EMAIL ::= "test@example.com" +TEST-EXAMPLE-COM-PASSWORD ::= "password" +TEST-EXAMPLE-COM-UUID ::= Uuid.parse "f76629c5-a070-4bbc-9918-64beaea48848" +TEST-EXAMPLE-COM-NAME ::= "Test User" + +TEST-ORGANIZATION-NAME ::= "Test Organization" +TEST-ORGANIZATION-UUID ::= Uuid.parse "4b6d9e35-cae9-44c0-8da0-6b0e485987e2" + +TEST-DEVICE-UUID ::= Uuid.parse "eb45c662-356c-4bea-ad8c-ede37688fddf" +TEST-POD-UUID ::= Uuid.parse "0e29c450-f802-49cc-b695-c5add71fdac3" +``` + +**Factory Functions:** + +```toit +// Create test CLI with temporary config +with-tmp-config-cli [block]: + with-tmp-directory: | directory | + config-path := "$directory/config" + app-name := "artemis-test" + config := cli-pkg.Config --app-name=app-name --path=config-path --data={:} + cli := cli-pkg.Cli app-name --config=config + block.call cli + +// Create pods for testing +create-pods name/string fleet/TestFleet --count/int -> List: + spec := """ + { "$schema": "https://toit.io/...", "name": "$name", ... } + """ + spec-path := "$fleet.fleet-dir/$(name).json" + write-blob-to-file spec-path spec + count.repeat: + fleet.run ["pod", "upload", spec-path] + return [description-id, spec-ids] +``` + +**Location:** +- Test utilities and fixtures in `tests/utils.toit` +- Test data constants defined at module level +- Fleet testing utilities in TestFleet class + +## Coverage + +**Requirements:** Not detected; no coverage enforcement found in configuration + +**View Coverage:** Not applicable; Toit test framework does not expose coverage metrics + +## Test Types + +**Unit Tests:** +- Scope: Individual functions and small modules +- Approach: Direct function calls with assertions +- Example: `test-open`, `test-send` in `channel-test.toit` +- Pattern: Parameterized helpers that run multiple scenarios + +**Integration Tests:** +- Scope: Multiple components interacting (CLI + server + broker) +- Approach: Full command execution with temporary servers +- Example: `cmd-pod-delete-test.toit`, broker policy tests +- Pattern: Use `with-fleet`, `with-server` context managers +- Configuration via `// ARTEMIS_TEST_FLAGS:` comments in test file + +**E2E Tests:** +- Framework: CMake-based with test flags for different server configurations +- Patterns: + - `// ARTEMIS_TEST_FLAGS: ARTEMIS` - requires Artemis server + - `// ARTEMIS_TEST_FLAGS: BROKER` - requires broker + - Multiple test variants per file with different flags +- Execution: Tests run with different server/broker combinations via CMake + +**Resource Locks:** +- Tests that need exclusive resources use CMake resource locks +- Lock names: `artemis_server`, `broker`, `artemis_broker`, `serial` +- Supabase tests automatically get locks when they use supabase flags + +## Common Patterns + +**Async Testing:** + +Concurrency via `spawn::` for parallel task execution: + +```toit +main: + provider.install + spawn:: test // Spawn test as concurrent task + provider.uninstall --wait +``` + +**Error Testing:** + +```toit +// Test that error is thrown with specific message +expect-throw "OUT_OF_RANGE: 209 > 200": + channel.acknowledge 209 + +// Test that specific exception type is thrown +expect-throw "ALREADY_IN_USE": + // code that throws +``` + +**Parameterized Tests:** + +Test helper functions called with different parameters: + +```toit +test-open "small" 32 * 1024 +test-open "medium" 64 * 1024 +test-open "large" 512 * 1024 + +test-neutering "hest" +[1, 2, 5, 127, 128, 129, 512, 1024, 3000].do: + test-neutering topic it +``` + +**Gold File Tests:** + +CLI output comparison against golden files: + +```toit +run-gold "BAA-delete-pod-revision" + "Delete a pod by revision" + [ + "pod", "delete", "$pod1-name#2" + ] + +// Compares output to gold/gold-dir-name/BAA-delete-pod-revision.txt +// Updates gold files if UPDATE_GOLD=1 environment variable set +``` + +**Test Timeouts:** + +Set via CMake based on test pattern: +- Default: `200` seconds +- Slow tests (`*-test-slow.toit`): `300` seconds +- Serial tests (`serial-*`): `1000` seconds +- QEMU tests (`qemu-*`): `300` seconds + +--- + +*Testing analysis: 2026-03-15* diff --git a/src/cli/broker.toit b/src/cli/broker.toit index 1a160518..90372022 100644 --- a/src/cli/broker.toit +++ b/src/cli/broker.toit @@ -11,7 +11,6 @@ import uuid show Uuid import encoding.base64 import encoding.ubjson -import .artemis import .cache import .config import .device @@ -24,6 +23,7 @@ import ..shared.version import ..shared.utils.patch show Patcher PatchObserver import .brokers.broker +import .organization import .event import .firmware import .pod-registry @@ -113,6 +113,46 @@ class Broker: ensure-authenticated: broker-connection_ + /** + Whether the broker supports administrative operations. + */ + supports-admin -> bool: + return broker-connection_ is AdminBrokerCli + + /** + Returns the admin interface of the broker. + + Throws if the broker does not support administrative operations. + */ + admin-connection -> AdminBrokerCli: + connection := broker-connection_ + if connection is not AdminBrokerCli: + throw "The configured broker does not support this operation." + return connection as AdminBrokerCli + + /** + Fetches the organization with the given $id. + + Returns null if the broker doesn't support administrative operations + or if the organization doesn't exist. + */ + get-organization --id/Uuid -> OrganizationDetailed?: + connection := broker-connection_ + if connection is not AdminBrokerCli: + return null + return (connection as AdminBrokerCli).get-organization id + + /** + Creates a device in the organization with the given $organization-id. + + The $device-id may be null in which case the broker creates an alias. + Throws if the broker does not support administrative operations. + */ + create-device --device-id/Uuid? --organization-id/Uuid -> Device: + return admin-connection.create-device-in-organization + --device-id=device-id + --organization-id=organization-id + /** Closes the broker. @@ -829,7 +869,6 @@ class Broker: /** Customizes a generic Toit envelope with the given $specification. - Also installs the Artemis service. The image is ready to be flashed together with the identity file. */ @@ -837,9 +876,7 @@ class Broker: --organization-id/Uuid --specification/PodSpecification --recovery-urls/List - --artemis/Artemis --output-path/string: - service-version := specification.artemis-version sdk-version := specification.sdk-version envelope-path := get-envelope @@ -856,7 +893,6 @@ class Broker: cli_.ui.abort "The envelope uses SDK version $envelope-sdk-version, but $sdk-version was requested." else: sdk-version = envelope-sdk-version - envelope-word-bit-size := Sdk.get-word-bit-size-from --envelope=envelope sdk := get-sdk sdk-version --cli=cli_ @@ -879,11 +915,10 @@ class Broker: connection.to-json device-config["connections"] = connections - // Create the assets for the Artemis service. + // Create the assets for the device service. // TODO(florian): share this code with the identity creation code. der-certificates := {:} broker-json := server-config-to-service-json server-config der-certificates - artemis-json := server-config-to-service-json artemis.server-config der-certificates with-tmp-directory: | tmp-dir | // Store the containers in the envelope. @@ -944,7 +979,7 @@ class Broker: }, "artemis.broker": { "format": "tison", - "json": artemis-json, + "json": broker-json, }, } der-certificates.do: | name/string value/ByteArray | @@ -969,61 +1004,6 @@ class Broker: artemis-assets-path := "$tmp-dir/artemis.assets" sdk.assets-create --output-path=artemis-assets-path artemis-assets - - // Build the Artemis service image. - artemis-container := get-artemis-container service-version --chip-family=envelope-chip-family --cli=cli_ - artemis-snapshot-path := "$tmp-dir/artemis.snapshot" - create-version-file := :: | repo-path/string | - // TODO(florian): share this code with the identity creation code. - version-path := "$repo-path/src/shared/version.toit" - if not file.is-file version-path: - // We already have the version form the container. - // We still need to extract a major and minor version. - artemis-version := artemis-container.git-ref - artemis-major/int := ? - artemis-minor/int := ? - if not artemis-version: - cli_.ui.abort "Local Artemis checkouts must have a version.toit file." - parts := (artemis-version.trim --left "v").split "." - if parts.size < 2: - // Probably just a commit hash or full ref. - // This should only happen during development. Use our version instead. - artemis-major = ARTEMIS-VERSION-MAJOR - artemis-minor = ARTEMIS-VERSION-MINOR - else: - had-error := false - artemis-major = int.parse parts[0] --on-error=: - had-error = true - 0 - artemis-minor = int.parse parts[1] --on-error=: - had-error = true - 0 - if had-error: - artemis-major = ARTEMIS-VERSION-MAJOR - artemis-minor = ARTEMIS-VERSION-MINOR - version-contents := """ - // This file is generated by the Toit SDK. - // Do not edit. - ARTEMIS-VERSION ::= "$artemis-version" - ARTEMIS-VERSION-MAJOR ::= $artemis-major - ARTEMIS-VERSION-MINOR ::= $artemis-minor - """ - file.write-contents --path=version-path version-contents - artemis-container.build-snapshot - --pre-compilation-hook=create-version-file - --relative-to=specification.relative-to - --sdk=sdk - --output-path=artemis-snapshot-path - --cli=cli_ - cli_.ui.emit --info "Added Artemis service container to envelope." - - sdk.firmware-add-container "artemis" - --envelope=output-path - --assets=artemis-assets-path - --program-path=artemis-snapshot-path - --trigger="boot" - --critical - // For convenience save all snapshots in the user's cache. cache-snapshots --envelope-path=output-path --cli=cli_ diff --git a/src/cli/brokers/broker.toit b/src/cli/brokers/broker.toit index 484b7800..f92cb0a1 100644 --- a/src/cli/brokers/broker.toit +++ b/src/cli/brokers/broker.toit @@ -10,6 +10,7 @@ import ..auth import ..config import ..event import ..device +import ..organization import ..pod-registry import ...shared.server-config import .supabase @@ -276,6 +277,84 @@ interface BrokerCli implements Authenticatable: --organization-id/Uuid --pod-id/Uuid +/** +Extends $BrokerCli with administrative operations. + +Not all brokers support administrative operations. For example, a + file-based broker (like a GitHub Pages broker) would not support + user or organization management. + +Use `broker is AdminBrokerCli` to check whether a broker supports + these operations. +*/ +interface AdminBrokerCli extends BrokerCli: + /** Returns the user-id of the authenticated user. */ + get-current-user-id -> Uuid + + /** + Fetches list of organizations the user has access to. + + The returned list contains instances of type $Organization. + */ + get-organizations -> List + + /** + Fetches the organization with the given $id. + + Returns null if the organization doesn't exist. + */ + get-organization id/Uuid -> OrganizationDetailed? + + /** Creates a new organization with the given $name. */ + create-organization name/string -> Organization + + /** Updates the given organization. */ + update-organization organization-id/Uuid --name/string -> none + + /** + Gets a list of members of the organization with the given $organization-id. + + Each entry is a map consisting of the "id" and "role". + */ + get-organization-members organization-id/Uuid -> List + + /** + Adds the user with $user-id as a new member to the organization + with $organization-id. + */ + organization-member-add --organization-id/Uuid --user-id/Uuid --role/string + + /** + Removes the user with $user-id from the organization with + $organization-id. + */ + organization-member-remove --organization-id/Uuid --user-id/Uuid + + /** + Updates the role of the user with $user-id in the organization + with $organization-id. + */ + organization-member-set-role --organization-id/Uuid --user-id/Uuid --role/string + + /** + Gets the profile of the user with the given $user-id. + + If no $user-id is given, the profile of the current user is returned. + Returns null if no user with the given ID exists. + */ + get-profile --user-id/Uuid?=null -> Map? + + /** Updates the profile of the current user. */ + update-profile --name/string -> none + + /** + Adds a new device to the organization with the given $organization-id. + + Takes a $device-id, representing the user's chosen name for the device. + The $device-id may be null in which case the server creates an alias. + */ + create-device-in-organization --organization-id/Uuid --device-id/Uuid? -> Device + with-broker server-config/ServerConfig --cli/Cli [block]: broker := BrokerCli server-config --cli=cli try: diff --git a/src/cli/brokers/supabase/supabase.toit b/src/cli/brokers/supabase/supabase.toit index 50081976..eb1373a4 100644 --- a/src/cli/brokers/supabase/supabase.toit +++ b/src/cli/brokers/supabase/supabase.toit @@ -3,10 +3,15 @@ import cli show Cli import http import supabase +import supabase.filter show equals import certificate-roots +import uuid show Uuid +import ..broker show AdminBrokerCli import ..http.base import ...config +import ...device +import ...organization import ...utils.supabase import ....shared.server-config @@ -38,7 +43,7 @@ create-broker-cli-supabase-http server-config/ServerConfigSupabase --cli/Cli -> return BrokerCliSupabase --id=id supabase-client http-config -class BrokerCliSupabase extends BrokerCliHttp: +class BrokerCliSupabase extends BrokerCliHttp implements AdminBrokerCli: supabase-client_/supabase.Client? := null constructor --id/string .supabase-client_ http-config/ServerConfigHttp: @@ -75,3 +80,89 @@ class BrokerCliSupabase extends BrokerCliHttp: return { "Authorization": "Bearer $bearer", } + + // AdminBrokerCli implementation. + + get-current-user-id -> Uuid: + return Uuid.parse supabase-client_.auth.get-current-user["id"] + + get-organizations -> List: + organizations := supabase-client_.rest.select "organizations" + return organizations.map: Organization.from-map it + + get-organization id/Uuid -> OrganizationDetailed?: + organizations := supabase-client_.rest.select "organizations" --filters=[ + equals "id" "$id" + ] + if organizations.is-empty: return null + return OrganizationDetailed.from-map organizations[0] + + create-organization name/string -> Organization: + inserted := supabase-client_.rest.insert "organizations" { "name": name } + return Organization.from-map inserted + + update-organization organization-id/Uuid --name/string -> none: + update := { + "name": name, + } + supabase-client_.rest.update "organizations" update --filters=[ + equals "id" "$organization-id" + ] + + get-organization-members organization-id/Uuid -> List: + members := supabase-client_.rest.select "roles" --filters=[ + equals "organization_id" "$organization-id" + ] + return members.map: { + "id": Uuid.parse it["user_id"], + "role": it["role"], + } + + organization-member-add --organization-id/Uuid --user-id/Uuid --role/string: + supabase-client_.rest.insert "roles" { + "organization_id": "$organization-id", + "user_id": "$user-id", + "role": role, + } + + organization-member-remove --organization-id/Uuid --user-id/Uuid: + supabase-client_.rest.delete "roles" --filters=[ + equals "organization_id" "$organization-id", + equals "user_id" "$user-id", + ] + + organization-member-set-role --organization-id/Uuid --user-id/Uuid --role/string: + supabase-client_.rest.update "roles" --filters=[ + equals "organization_id" "$organization-id", + equals "user_id" "$user-id", + ] { "role": role } + + get-profile --user-id/Uuid?=null -> Map?: + if not user-id: + current-user := supabase-client_.auth.get-current-user + user-id = Uuid.parse current-user["id"] + response := supabase-client_.rest.select "profiles_with_email" --filters=[ + equals "id" "$user-id", + ] + if response.is-empty: return null + result := response[0] + result["id"] = Uuid.parse result["id"] + return result + + update-profile --name/string -> none: + current-user := supabase-client_.auth.get-current-user + user-id := Uuid.parse current-user["id"] + supabase-client_.rest.update "profiles" { "name": name } --filters=[ + equals "id" "$user-id", + ] + + create-device-in-organization --organization-id/Uuid --device-id/Uuid? -> Device: + payload := { + "organization_id": "$organization-id", + } + if device-id: payload["alias"] = "$device-id" + inserted := supabase-client_.rest.insert "devices" payload + return Device + --hardware-id=Uuid.parse inserted["id"] + --id=Uuid.parse inserted["alias"] + --organization-id=Uuid.parse inserted["organization_id"] diff --git a/src/cli/cmds/auth.toit b/src/cli/cmds/auth.toit index 250686eb..456395dc 100644 --- a/src/cli/cmds/auth.toit +++ b/src/cli/cmds/auth.toit @@ -8,7 +8,6 @@ import ..cache import ..config import ..auth show Authenticatable import ..server-config -import ..artemis-servers.artemis-server show with-server ArtemisServerCli import ..brokers.broker show with-broker BrokerCli SIGNIN-OPTIONS ::= [ @@ -24,16 +23,15 @@ SIGNIN-OPTIONS ::= [ create-auth-commands -> List: auth-cmd := Command "auth" - --help="Authenticate against the Artemis server or a broker." + --help="Authenticate against a broker." sign-up-cmd := Command "signup" --aliases=["sign-up"] --help=""" - Sign up for an Artemis account with email and password. + Sign up for an account with email and password. - If '--broker' is provided, signs up for the default broker. If a server is provided with '--server', signs up for that server. - If neither is provided, signs up for the Artemis server. + Otherwise, signs up for the default broker. See 'list' for available servers. The usual way of signing up is to use oauth2. This command is only @@ -44,7 +42,9 @@ create-auth-commands -> List: are available. """ --options=[ - Flag "broker" --help="Sign up for the broker.", + // The --broker flag is kept for backward compatibility. It's now a no-op + // since all auth operations go through the broker. + Flag "broker" --hidden, OptionString "server" --help="Sign up for a specific server.", OptionString "email" --help="The email address for the account." @@ -54,7 +54,7 @@ create-auth-commands -> List: --required, ] --examples=[ - Example "Sign up for an Artemis account with email and password:" + Example "Sign up for an account with email and password:" --arguments="--email=test@example.com --password=secret", ] --run=:: sign-up it @@ -63,28 +63,27 @@ create-auth-commands -> List: login-cmd := Command "login" --aliases=["signin", "log-in", "sign-in"] --help=""" - Log in to the Artemis server or a broker. + Log in to a broker. - If '--broker' is provided, authenticates with the default broker. If a server is provided with '--server', authenticates with that server. - If neither is provided, authenticates with the Artemis server. + Otherwise, authenticates with the default broker. See 'list' for available servers. """ --options=[ - Flag "broker" --help="Log into the default broker.", + Flag "broker" --hidden, // Kept for backward compatibility (now a no-op). OptionString "server" --help="Log into a specific server.", ] + SIGNIN-OPTIONS --examples=[ - Example "Log in to the Artemis server using GitHub:" + Example "Log in to the default broker using GitHub:" --arguments="" --global-priority=10, Example """ - Log in to the Artemis server using GitHub without opening the link + Log in to the default broker using GitHub without opening the link in a browser:""" --arguments="--no-open-browser", - Example "Log in to the Artemis server using Google:" + Example "Log in to the default broker using Google:" --arguments="--provider=google", - Example "Log in to the Artemis server with email and password:" + Example "Log in with email and password:" --arguments="--email=test@example.com --password=secret", ] --run=:: sign-in it @@ -104,14 +103,13 @@ create-auth-commands -> List: --help=""" Updates the email or password for an account. - If '--broker' is provided, updates the account on the default broker. If a server is provided with '--server', updates the account on that server. - If neither is provided, updates the account on the Artemis server. + Otherwise, updates the account on the default broker. See 'list' for available servers. """ --options=[ - Flag "broker" --help="Update the account on a broker.", + Flag "broker" --hidden, // Kept for backward compatibility (now a no-op). OptionString "server" --help="Update the account on a specific server.", Option "email" --help="New email for the account.", Option "password" --help="New password for the account.", @@ -126,15 +124,14 @@ create-auth-commands -> List: logout-cmd := Command "logout" --aliases=["signout", "log-out", "sign-out"] --help=""" - Log out of the Artemis server or a broker. + Log out of a broker. - If '--broker' is provided, logs out of the default broker. If a server is provided with '--server', logs out of that server. - If neither is provided, logs out of the Artemis server. + Otherwise, logs out of the default broker. See 'list' for available servers. """ --options=[ - Flag "broker" --help="Log out of the the broker.", + Flag "broker" --hidden, // Kept for backward compatibility (now a no-op). OptionString "server" --help="Log out of a specific server.", ] --run=:: logout it @@ -163,26 +160,17 @@ update invocation/Invocation: with-authenticatable invocation/Invocation [block]: - broker := invocation["broker"] server := invocation["server"] cli := invocation.cli - ui := cli.ui - - if broker and server: - ui.abort "Cannot specify both '--broker' and '--server'." server-config/ServerConfig := ? - if broker or server: - server-config = broker - ? get-server-from-config --cli=cli --key=CONFIG-BROKER-DEFAULT-KEY - : get-server-from-config --cli=cli --name=server - with-broker --cli=cli server-config: | broker/BrokerCli | - block.call server-config.name broker + if server: + server-config = get-server-from-config --cli=cli --name=server else: - server-config = get-server-from-config --cli=cli --key=CONFIG-ARTEMIS-DEFAULT-KEY - with-server server-config --cli=cli: | server/ArtemisServerCli | - block.call server-config.name server + server-config = get-server-from-config --cli=cli --key=CONFIG-BROKER-DEFAULT-KEY + with-broker --cli=cli server-config: | broker/BrokerCli | + block.call server-config.name broker sign-in invocation/Invocation: with-authenticatable invocation: | name/string authenticatable/Authenticatable | diff --git a/src/cli/cmds/config.toit b/src/cli/cmds/config.toit index d5f75569..b7068050 100644 --- a/src/cli/cmds/config.toit +++ b/src/cli/cmds/config.toit @@ -232,7 +232,10 @@ default-server invocation/Invocation: config := cli.config ui := cli.ui - config-key/string := invocation["artemis"] ? CONFIG-ARTEMIS-DEFAULT-KEY : CONFIG-BROKER-DEFAULT-KEY + if invocation["artemis"]: + ui.emit --warning "The '--artemis' flag is deprecated. Using the default broker instead." + + config-key := CONFIG-BROKER-DEFAULT-KEY if invocation["clear"]: config.remove config-key diff --git a/src/cli/cmds/device-container.toit b/src/cli/cmds/device-container.toit index 9e73737d..64b3d089 100644 --- a/src/cli/cmds/device-container.toit +++ b/src/cli/cmds/device-container.toit @@ -5,7 +5,6 @@ import uuid show Uuid import .device import .utils_ -import ..artemis import ..cache import ..config import ..fleet diff --git a/src/cli/cmds/device.toit b/src/cli/cmds/device.toit index 5e892448..503214b9 100644 --- a/src/cli/cmds/device.toit +++ b/src/cli/cmds/device.toit @@ -9,7 +9,6 @@ import uuid show Uuid import .utils_ import .device-container import .serial show PARTITION-OPTION -import ..artemis import ..cache import ..config import ..device @@ -254,7 +253,7 @@ pod-for_ -> Pod? return Pod.from-file local --organization-id=fleet.organization-id --recovery-urls=fleet.recovery-urls - --artemis=fleet.artemis + --tmp-directory=fleet.tmp-directory --broker=fleet.broker --cli=cli @@ -342,7 +341,7 @@ show invocation/Invocation: if devices.is-empty: ui.abort "Device '$device-reference' does not exist on the broker." broker-device := devices[fleet-device.id] - organization := fleet.artemis.get-organization --id=broker-device.organization-id + organization := fleet.broker.get-organization --id=broker-device.organization-id events/List? := null if max-events != 0: events-map := broker.get-events @@ -420,8 +419,6 @@ extract-device fleet-device/DeviceFleet --cli/Cli: ui := cli.ui - artemis := fleet.artemis - device/Device := identity-path ? FleetWithDevices.device-from --identity-path=identity-path : fleet.broker.device-for --id=fleet-device.id diff --git a/src/cli/cmds/fleet.toit b/src/cli/cmds/fleet.toit index 45b3d91b..e39cdedb 100644 --- a/src/cli/cmds/fleet.toit +++ b/src/cli/cmds/fleet.toit @@ -13,8 +13,8 @@ import .device show import .auth as auth-cmd import .serial show PARTITION-OPTION import .utils_ -import ..artemis -import ..brokers.broker show BrokerCli +import ..broker show Broker +import ..brokers.broker show AdminBrokerCli BrokerCli with-broker import ..config import ..cache import ..device @@ -640,25 +640,34 @@ init invocation/Invocation: default-recovery-urls := (cli.config.get CONFIG-RECOVERY-SERVERS-KEY) or [] fleet-root := compute-fleet-root-or-ref invocation - with-artemis invocation: | artemis/Artemis | - fleet-file := FleetWithDevices.init fleet-root artemis - --organization-id=organization-id - --broker-config=broker-config - --recovery-url-prefixes=default-recovery-urls - --cli=cli - - fleet-file.recovery-urls.do: | url/string | - ui.emit --info "Added recovery URL: $url" - ui.emit --info "Fleet root '$fleet-root' initialized." - ui.emit --kind=Ui.RESULT - --structured=: { - "id": "$fleet-file.id", - "broker": fleet-file.broker-config.to-json --base64 --der-serializer=: unreachable, - "recovery-urls": fleet-file.recovery-urls, - } - --text=: - // Don't print anything. - null + // Validate the organization before creating the fleet files. + with-broker broker-config --cli=cli: | broker/BrokerCli | + if broker is AdminBrokerCli: + broker.ensure-authenticated: | error-message | + ui.abort "$error-message (broker)." + admin := broker as AdminBrokerCli + org := admin.get-organization organization-id + if not org: + ui.abort "Organization $organization-id does not exist or is not accessible." + + fleet-file := FleetWithDevices.init fleet-root + --organization-id=organization-id + --broker-config=broker-config + --recovery-url-prefixes=default-recovery-urls + --cli=cli + + fleet-file.recovery-urls.do: | url/string | + ui.emit --info "Added recovery URL: $url" + ui.emit --info "Fleet root '$fleet-root' initialized." + ui.emit --kind=Ui.RESULT + --structured=: { + "id": "$fleet-file.id", + "broker": fleet-file.broker-config.to-json --base64 --der-serializer=: unreachable, + "recovery-urls": fleet-file.recovery-urls, + } + --text=: + // Don't print anything. + null login invocation/Invocation: cli := invocation.cli @@ -762,7 +771,7 @@ roll-out invocation/Invocation: with-devices-fleet invocation: | fleet/FleetWithDevices | pod-diff-bases := diff-bases.map: | file-or-ref/string | if file.is-file file-or-ref: - Pod.parse file-or-ref --tmp-directory=fleet.artemis.tmp-directory --cli=cli + Pod.parse file-or-ref --tmp-directory=fleet.tmp-directory --cli=cli else: fleet.download (PodReference.parse file-or-ref --cli=cli) fleet.roll-out --diff-bases=pod-diff-bases diff --git a/src/cli/cmds/org.toit b/src/cli/cmds/org.toit index a1bdc98d..bd4d20a2 100644 --- a/src/cli/cmds/org.toit +++ b/src/cli/cmds/org.toit @@ -9,7 +9,7 @@ import ..config import ..cache import ..server-config import ..organization -import ..artemis-servers.artemis-server show with-server ArtemisServerCli +import ..brokers.broker show with-broker AdminBrokerCli BrokerCli import ..utils create-org-commands -> List: @@ -219,19 +219,21 @@ create-org-commands -> List: return [org-cmd] -with-org-server invocation/Invocation [block]: +with-org-admin invocation/Invocation [block]: cli := invocation.cli ui := cli.ui - server-config/ServerConfig := ? - server-config = get-server-from-config --key=CONFIG-ARTEMIS-DEFAULT-KEY --cli=cli + server-config/ServerConfig := get-server-from-config --key=CONFIG-BROKER-DEFAULT-KEY --cli=cli - with-server server-config --cli=cli: | server/ArtemisServerCli | - server.ensure-authenticated: | error-message | - ui.abort "$error-message (artemis)." - block.call server + with-broker server-config --cli=cli: | broker/BrokerCli | + if broker is not AdminBrokerCli: + ui.abort "The configured broker does not support organization management." + admin := broker as AdminBrokerCli + broker.ensure-authenticated: | error-message | + ui.abort "$error-message (broker)." + block.call admin -with-org-server-id invocation/Invocation [block]: +with-org-admin-id invocation/Invocation [block]: org-id := invocation["organization-id"] cli := invocation.cli @@ -242,12 +244,12 @@ with-org-server-id invocation/Invocation [block]: if not org-id: ui.abort "No default organization set." - with-org-server invocation: | server | - block.call server org-id + with-org-admin invocation: | admin | + block.call admin org-id list-orgs invocation/Invocation -> none: - with-org-server invocation: | server/ArtemisServerCli | - orgs := server.get-organizations + with-org-admin invocation: | admin/AdminBrokerCli | + orgs := admin.get-organizations invocation.cli.ui.emit-table --result --header={"id": "ID", "name": "Name"} orgs.map: | org/Organization | { @@ -257,18 +259,18 @@ list-orgs invocation/Invocation -> none: add-org invocation/Invocation -> none: should-make-default := invocation["default"] - with-org-server invocation: | server/ArtemisServerCli | - org := server.create-organization invocation["name"] + with-org-admin invocation: | admin/AdminBrokerCli | + org := admin.create-organization invocation["name"] invocation.cli.ui.emit --info "Added organization $org.id - $org.name." if should-make-default: make-default_ org --cli=invocation.cli show-org invocation/Invocation -> none: - with-org-server-id invocation: | server/ArtemisServerCli org-id/Uuid | - print-org org-id server --cli=invocation.cli + with-org-admin-id invocation: | admin/AdminBrokerCli org-id/Uuid | + print-org org-id admin --cli=invocation.cli -print-org org-id/Uuid server/ArtemisServerCli --cli/Cli -> none: +print-org org-id/Uuid admin/AdminBrokerCli --cli/Cli -> none: ui := cli.ui - org := server.get-organization org-id + org := admin.get-organization org-id if not org: ui.abort "Organization $org-id not found." if ui.wants-structured --kind=Ui.RESULT: @@ -309,14 +311,14 @@ default-org invocation/Invocation -> none: ui.emit --result "$org-id" return - with-org-server invocation: | server/ArtemisServerCli | - print-org org-id server --cli=cli + with-org-admin invocation: | admin/AdminBrokerCli | + print-org org-id admin --cli=cli return - with-org-server-id invocation: | server/ArtemisServerCli org-id/Uuid | + with-org-admin-id invocation: | admin/AdminBrokerCli org-id/Uuid | org/OrganizationDetailed? := null - exception := catch: org = server.get-organization org-id + exception := catch: org = admin.get-organization org-id if exception or not org: ui.abort "Organization not found." @@ -337,21 +339,21 @@ update-org invocation/Invocation -> none: if not name: ui.abort "No name provided." if name == "": ui.abort "Name cannot be empty." - with-org-server-id invocation: | server/ArtemisServerCli org-id/Uuid | - server.update-organization org-id --name=name + with-org-admin-id invocation: | admin/AdminBrokerCli org-id/Uuid | + admin.update-organization org-id --name=name ui.emit --info "Updated organization $org-id." member-list invocation/Invocation -> none: ui := invocation.cli.ui - with-org-server-id invocation: | server/ArtemisServerCli org-id/Uuid | - members := server.get-organization-members org-id + with-org-admin-id invocation: | admin/AdminBrokerCli org-id/Uuid | + members := admin.get-organization-members org-id if invocation["id-only"]: member-ids := members.map: "$it["id"]" member-ids.sort --in-place ui.emit-list member-ids --kind=Ui.RESULT return - profiles := members.map: server.get-profile --user-id=it["id"] + profiles := members.map: admin.get-profile --user-id=it["id"] unsorted-result := List members.size: { "id": "$members[it]["id"]", "role": members[it]["role"], @@ -375,11 +377,11 @@ member-add invocation/Invocation -> none: user-id := invocation["user-id"] role := invocation["role"] - with-org-server-id invocation: | server/ArtemisServerCli org-id/Uuid| - existing-members := server.get-organization-members org-id + with-org-admin-id invocation: | admin/AdminBrokerCli org-id/Uuid| + existing-members := admin.get-organization-members org-id if (existing-members.any: it["id"] == user-id): ui.abort "User $user-id is already a member of organization $org-id." - server.organization-member-add + admin.organization-member-add --organization-id=org-id --user-id=user-id --role=role @@ -391,12 +393,12 @@ member-remove invocation/Invocation -> none: user-id := invocation["user-id"] force := invocation["force"] - with-org-server-id invocation: | server/ArtemisServerCli org-id/Uuid | + with-org-admin-id invocation: | admin/AdminBrokerCli org-id/Uuid | if not force: - current-user-id := server.get-current-user-id + current-user-id := admin.get-current-user-id if user-id == current-user-id: ui.abort "Use '--force' to remove yourself from an organization." - server.organization-member-remove --organization-id=org-id --user-id=user-id + admin.organization-member-remove --organization-id=org-id --user-id=user-id ui.emit --info "Removed user $user-id from organization $org-id." member-set-role invocation/Invocation -> none: @@ -405,8 +407,8 @@ member-set-role invocation/Invocation -> none: user-id := invocation["user-id"] role := invocation["role"] - with-org-server-id invocation: | server/ArtemisServerCli org-id/Uuid| - server.organization-member-set-role + with-org-admin-id invocation: | admin/AdminBrokerCli org-id/Uuid| + admin.organization-member-set-role --organization-id=org-id --user-id=user-id --role=role diff --git a/src/cli/cmds/pod.toit b/src/cli/cmds/pod.toit index b0520ddd..0ed5c4e6 100644 --- a/src/cli/cmds/pod.toit +++ b/src/cli/cmds/pod.toit @@ -4,7 +4,6 @@ import cli show * import host.file import .utils_ -import ..artemis import ..config import ..cache import ..fleet @@ -302,14 +301,12 @@ create-pod invocation/Invocation: output := invocation["output"] with-pod-fleet invocation: | fleet/Fleet | - artemis := fleet.artemis - broker := fleet.broker pod := Pod.from-specification --organization-id=fleet.organization-id --recovery-urls=fleet.recovery-urls --path=specification-path - --artemis=artemis - --broker=broker + --tmp-directory=fleet.tmp-directory + --broker=fleet.broker --cli=cli pod.write output --cli=cli @@ -333,14 +330,12 @@ upload invocation/Invocation: tags.add "latest" with-pod-fleet invocation: | fleet/Fleet | - artemis := fleet.artemis - broker := fleet.broker pod-paths.do: | pod-path/string | pod := Pod.from-file pod-path --organization-id=fleet.organization-id --recovery-urls=fleet.recovery-urls - --artemis=artemis - --broker=broker + --tmp-directory=fleet.tmp-directory + --broker=fleet.broker --cli=cli upload-result := fleet.upload --pod=pod --tags=tags --force-tags=force if ui.wants-structured --kind=Ui.RESULT: diff --git a/src/cli/cmds/profile.toit b/src/cli/cmds/profile.toit index b14d05eb..afafa843 100644 --- a/src/cli/cmds/profile.toit +++ b/src/cli/cmds/profile.toit @@ -6,7 +6,7 @@ import net import ..config import ..cache import ..server-config -import ..artemis-servers.artemis-server show with-server ArtemisServerCli +import ..brokers.broker show with-broker AdminBrokerCli BrokerCli create-profile-commands -> List: profile-cmd := Command "profile" @@ -35,20 +35,23 @@ create-profile-commands -> List: return [profile-cmd] -with-profile-server invocation/Invocation [block]: +with-profile-admin invocation/Invocation [block]: cli := invocation.cli - server-config := get-server-from-config --key=CONFIG-ARTEMIS-DEFAULT-KEY --cli=cli + server-config := get-server-from-config --key=CONFIG-BROKER-DEFAULT-KEY --cli=cli - with-server server-config --cli=cli: | server/ArtemisServerCli | - server.ensure-authenticated: | error-message | - cli.ui.abort "$error-message (artemis)." - block.call server + with-broker server-config --cli=cli: | broker/BrokerCli | + if broker is not AdminBrokerCli: + cli.ui.abort "The configured broker does not support profile management." + admin := broker as AdminBrokerCli + broker.ensure-authenticated: | error-message | + cli.ui.abort "$error-message (broker)." + block.call admin show-profile invocation/Invocation: ui := invocation.cli.ui - with-profile-server invocation: | server/ArtemisServerCli | - profile := server.get-profile + with-profile-admin invocation: | admin/AdminBrokerCli | + profile := admin.get-profile if ui.wants-structured --kind=Ui.RESULT: // We recreate the map, so we don't show unnecessary entries. ui.emit @@ -73,6 +76,6 @@ update-profile invocation/Invocation: if not name: ui.abort "No name specified." - with-profile-server invocation: | server/ArtemisServerCli | - server.update-profile --name=name + with-profile-admin invocation: | admin/AdminBrokerCli | + admin.update-profile --name=name ui.emit --info "Profile updated." diff --git a/src/cli/cmds/sdk.toit b/src/cli/cmds/sdk.toit index d8390612..c5ff5f8d 100644 --- a/src/cli/cmds/sdk.toit +++ b/src/cli/cmds/sdk.toit @@ -1,43 +1,18 @@ // Copyright (C) 2023 Toitware ApS. All rights reserved. import cli show * -import net -import semver - -import ..config -import ..cache -import ..fleet -import ..server-config -import ..artemis-servers.artemis-server show with-server ArtemisServerCli -import .utils_ create-sdk-commands -> List: sdk-cmd := Command "sdk" --help="Information about supported SDKs." - --options=[ - Option "server" --hidden --help="The server to use.", - ] list-cmd := Command "list" --aliases=["ls"] - --help="List supported SDKs. REMOVED." - --options=[ - Option "sdk-version" --help="The SDK version to list.", - Option "service-version" --help="The service version to list.", - ] - --examples=[ - Example "List all available SDKs/service versions:" - --arguments="", - Example "List all available service versions for a the SDK version v2.0.0-alpha.139:" - --arguments="--sdk-version=v2.0.0-alpha.139", - Example "List all available SDK versions for the service version v0.5.5:" - --arguments="--service-version=v0.5.5", - ] + --help="List supported SDKs." --run=:: list-sdks it sdk-cmd.add list-cmd return [sdk-cmd] list-sdks invocation/Invocation: - print "Artemis now compiles the service locally, and services are not downloaded from the server anymore." - invocation.cli.ui.abort "UNSUPPORTED" + invocation.cli.ui.abort "The 'sdk list' command is no longer supported." diff --git a/src/cli/cmds/serial.toit b/src/cli/cmds/serial.toit index c689c0f7..1b5456ed 100644 --- a/src/cli/cmds/serial.toit +++ b/src/cli/cmds/serial.toit @@ -5,7 +5,6 @@ import host.file import uuid show Uuid import .utils_ -import ..artemis import ..broker import ..cache import ..config @@ -184,9 +183,6 @@ flash invocation/Invocation: ui.abort "Cannot specify both a local pod file and a remote pod reference." with-devices-fleet invocation: | fleet/FleetWithDevices | - artemis := fleet.artemis - broker := fleet.broker - with-tmp-directory: | tmp-dir/string | device-id := random-uuid identity-path := fleet.create-identity @@ -203,8 +199,8 @@ flash invocation/Invocation: pod = Pod.from-file local --organization-id=fleet.organization-id --recovery-urls=fleet.recovery-urls - --artemis=artemis - --broker=broker + --tmp-directory=fleet.tmp-directory + --broker=fleet.broker --cli=cli reference = PodReference --id=pod.id else: @@ -250,7 +246,7 @@ flash invocation/Invocation: else: ui.emit --info "Simulating flash." ui.emit --info "Using the local Artemis service and not the one specified in the specification." - old-default := cli.config.get CONFIG-ARTEMIS-DEFAULT-KEY + old-default := cli.config.get CONFIG-BROKER-DEFAULT-KEY if should-make-default: make-default_ --device-id=device-id --cli=cli run-host --pod=pod @@ -278,8 +274,8 @@ flash --station/bool invocation/Invocation: partitions := build-partitions_ params["partition"] --cli=cli force := params["force"] - with-artemis invocation: | artemis/Artemis | - pod := Pod.parse pod-path --tmp-directory=artemis.tmp-directory --cli=cli + with-tmp-directory: | tmp-directory/string | + pod := Pod.parse pod-path --tmp-directory=tmp-directory --cli=cli with-tmp-directory: | tmp-dir/string | // Make unique for the given device. config-bytes := pod.compute-device-specific-data diff --git a/src/cli/cmds/utils_.toit b/src/cli/cmds/utils_.toit index f1edae30..de23a5d3 100644 --- a/src/cli/cmds/utils_.toit +++ b/src/cli/cmds/utils_.toit @@ -5,7 +5,6 @@ import host.os import partition-table show PartitionTable import uuid show Uuid -import ..artemis import ..config import ..cache import ..fleet @@ -14,20 +13,6 @@ import ..sdk import ..server-config import ..utils -with-artemis invocation/Invocation [block]: - cli := invocation.cli - artemis-config := get-server-from-config --cli=cli --key=CONFIG-ARTEMIS-DEFAULT-KEY - - with-tmp-directory: | tmp-directory/string | - artemis := Artemis - --cli=invocation.cli - --tmp-directory=tmp-directory - --server-config=artemis-config - try: - block.call artemis - finally: - artemis.close - default-device-from-config --cli/Cli -> Uuid?: config := cli.config device-id-string := config.get CONFIG-DEVICE-DEFAULT-KEY @@ -47,9 +32,10 @@ with-devices-fleet invocation/Invocation [block]: // the constructor call below will throw. fleet-root := compute-fleet-root-or-ref invocation - with-artemis invocation: | artemis/Artemis | + with-tmp-directory: | tmp-directory/string | default-broker-config := get-server-from-config --cli=cli --key=CONFIG-BROKER-DEFAULT-KEY - fleet := FleetWithDevices fleet-root artemis + fleet := FleetWithDevices fleet-root + --tmp-directory=tmp-directory --default-broker-config=default-broker-config --cli=cli block.call fleet @@ -59,9 +45,10 @@ with-pod-fleet invocation/Invocation [block]: fleet-root-or-ref := compute-fleet-root-or-ref invocation - with-artemis invocation: | artemis/Artemis | + with-tmp-directory: | tmp-directory/string | default-broker-config := get-server-from-config --cli=cli --key=CONFIG-BROKER-DEFAULT-KEY - fleet := Fleet fleet-root-or-ref artemis + fleet := Fleet fleet-root-or-ref + --tmp-directory=tmp-directory --default-broker-config=default-broker-config --cli=cli block.call fleet diff --git a/src/cli/fleet.toit b/src/cli/fleet.toit index 9f12a6c4..102729c0 100644 --- a/src/cli/fleet.toit +++ b/src/cli/fleet.toit @@ -8,7 +8,6 @@ import encoding.base64 import host.file import uuid show Uuid -import .artemis import .broker import .cache import .config @@ -330,24 +329,26 @@ class Fleet: id/Uuid organization-id/Uuid - artemis/Artemis broker/Broker + tmp-directory/string cli_/Cli fleet-root-or-ref_/string fleet-file_/FleetFile constructor fleet-root-or-ref/string - artemis/Artemis + --tmp-directory/string --default-broker-config/ServerConfig --cli/Cli: fleet-file := load-fleet-file fleet-root-or-ref --default-broker-config=default-broker-config --cli=cli - return Fleet fleet-root-or-ref artemis + return Fleet fleet-root-or-ref + --tmp-directory=tmp-directory --fleet-file=fleet-file --cli=cli - constructor .fleet-root-or-ref_ .artemis + constructor .fleet-root-or-ref_ + --.tmp-directory --fleet-file/FleetFile --short-strings/Map?=null --cli/Cli: @@ -359,13 +360,13 @@ class Fleet: --server-config=fleet-file.broker-config --fleet-id=id --organization-id=organization-id - --tmp-directory=artemis.tmp-directory + --tmp-directory=tmp-directory --short-strings=short-strings --cli=cli - // TODO(florian): should we always do this check? - org := artemis.get-organization --id=organization-id - if not org: + // Validate the organization if the broker supports it. + org := broker.get-organization --id=organization-id + if org == null and broker.supports-admin: cli.ui.abort "Organization $organization-id does not exist or is not accessible." static load-fleet-file -> FleetFile @@ -520,7 +521,8 @@ class FleetWithDevices extends Fleet: /** Map from name, device-id, alias to index in $devices_. */ aliases_/Map := {:} - constructor fleet-root/string artemis/Artemis + constructor fleet-root/string + --tmp-directory/string --default-broker-config/ServerConfig --cli/Cli: if not file.is-directory fleet-root and file.is-file fleet-root: @@ -539,12 +541,13 @@ class FleetWithDevices extends Fleet: devices_.do: | device/DeviceFleet | device-short-strings_[device.id] = device.short-string aliases_ = build-alias-map_ devices_ --cli=cli - super fleet-root artemis + super fleet-root + --tmp-directory=tmp-directory --fleet-file=fleet-file --short-strings=device-short-strings_ --cli=cli - static init fleet-root/string artemis/Artemis -> FleetFile + static init fleet-root/string -> FleetFile --organization-id/Uuid --broker-config/ServerConfig --recovery-url-prefixes/List @@ -560,10 +563,6 @@ class FleetWithDevices extends Fleet: if file.is-file "$fleet-root/$DEVICES-FILE_": ui.abort "Fleet root '$fleet-root' already contains a $DEVICES-FILE_ file." - org := artemis.get-organization --id=organization-id - if not org: - ui.abort "Organization $organization-id does not exist or is not accessible." - broker-name := broker-config.name fleet-id := random-uuid recovery-urls := recovery-url-prefixes.map: | prefix | @@ -703,7 +702,7 @@ class FleetWithDevices extends Fleet: --short-strings=device-short-strings_ --fleet-id=id --organization-id=organization-id - --tmp-directory=artemis.tmp-directory + --tmp-directory=tmp-directory --cli=cli_ old-broker.update --device-id=device-id --pod=pod @@ -747,7 +746,7 @@ class FleetWithDevices extends Fleet: --short-strings=device-short-strings_ --fleet-id=id --organization-id=organization-id - --tmp-directory=artemis.tmp-directory + --tmp-directory=tmp-directory --cli=cli_ // We could filter out devices that were already known in the new broker, but // it's easier and more robust to update all devices. @@ -843,7 +842,7 @@ class FleetWithDevices extends Fleet: --organization-id=organization-id --short-strings=device-short-strings_ --cli=cli_ - --tmp-directory=artemis.tmp-directory + --tmp-directory=tmp-directory all-brokers := [broker] + migrating-from-brokers @@ -1025,26 +1024,20 @@ class FleetWithDevices extends Fleet: /** Provisions a device. - Contacts the Artemis server and creates a new device entry with the - given $device-id (used as "alias" on the server side) in the - organization with the given $organization-id. + Creates a new device entry on the broker with the given $device-id + (used as "alias" on the server side) in the organization with the + given $organization-id. Writes the identity file to $out-path. */ provision --device-id/Uuid? --out-path/string: - // Ensure that we are authenticated with both the Artemis server and the broker. - // We don't want to create a device on Artemis and then have an error with the broker. - artemis.ensure-authenticated broker.ensure-authenticated - device := artemis.create-device + device := broker.create-device --device-id=device-id --organization-id=organization-id assert: device.id == device-id - hardware-id := device.hardware-id - // Insert an initial event mostly for testing purposes. - artemis.notify-created --hardware-id=hardware-id broker.notify-created device write-identity-file device --out-path=out-path @@ -1068,7 +1061,7 @@ class FleetWithDevices extends Fleet: --short-strings=device-short-strings_ --fleet-id=id --organization-id=organization-id - --tmp-directory=artemis.tmp-directory + --tmp-directory=tmp-directory --cli=cli_ if new-broker.server-config.name == broker.server-config.name: @@ -1123,7 +1116,7 @@ class FleetWithDevices extends Fleet: --short-strings=device-short-strings_ --fleet-id=id --organization-id=organization-id - --tmp-directory=artemis.tmp-directory + --tmp-directory=tmp-directory --cli=cli_ current-detailed-devices := current-broker.get-devices --device-ids=device-ids current-ids := current-detailed-devices.keys diff --git a/src/cli/pod.toit b/src/cli/pod.toit index ee22d1fc..453e1a6f 100644 --- a/src/cli/pod.toit +++ b/src/cli/pod.toit @@ -10,7 +10,6 @@ import host.file import io import uuid show Uuid -import .artemis import .broker import .cache import .device @@ -60,7 +59,7 @@ class Pod: --organization-id/Uuid --recovery-urls/List --path/string - --artemis/Artemis + --tmp-directory/string --broker/Broker --cli/Cli: specification := parse-pod-specification-file path --cli=cli @@ -68,7 +67,7 @@ class Pod: --organization-id=organization-id --recovery-urls=recovery-urls --specification=specification - --artemis=artemis + --tmp-directory=tmp-directory --broker=broker --cli=cli @@ -76,15 +75,14 @@ class Pod: --organization-id/Uuid --recovery-urls/List --specification/PodSpecification + --tmp-directory/string --broker/Broker - --artemis/Artemis --cli/Cli: - envelope-path := generate-envelope-path_ --tmp-directory=artemis.tmp-directory + envelope-path := generate-envelope-path_ --tmp-directory=tmp-directory broker.customize-envelope --organization-id=organization-id --recovery-urls=recovery-urls --output-path=envelope-path - --artemis=artemis --specification=specification envelope := file.read-contents envelope-path id := random-uuid @@ -95,7 +93,7 @@ class Pod: return Pod --id=id --name=specification.name - --tmp-directory=artemis.tmp-directory + --tmp-directory=tmp-directory --envelope=envelope --envelope-path=envelope-path --partition-table=partition-table @@ -163,7 +161,7 @@ class Pod: path/string --organization-id/Uuid --recovery-urls/List - --artemis/Artemis + --tmp-directory/string --broker/Broker --cli/Cli: if not file.is-file path: @@ -179,13 +177,13 @@ class Pod: stream.close pod/Pod := ? if is-compiled-pod: - return Pod.parse path --tmp-directory=artemis.tmp-directory --cli=cli + return Pod.parse path --tmp-directory=tmp-directory --cli=cli else: return Pod.from-specification --organization-id=organization-id --recovery-urls=recovery-urls --path=path - --artemis=artemis + --tmp-directory=tmp-directory --broker=broker --cli=cli diff --git a/src/cli/server-config.toit b/src/cli/server-config.toit index e40375c6..2be3f53a 100644 --- a/src/cli/server-config.toit +++ b/src/cli/server-config.toit @@ -34,7 +34,7 @@ get-server-from-config --cli/Cli --key/string -> ServerConfig?: return DEFAULT-ARTEMIS-SERVER-CONFIG if not server-name: - if key == CONFIG-ARTEMIS-DEFAULT-KEY or key == CONFIG-BROKER-DEFAULT-KEY: + if key == CONFIG-BROKER-DEFAULT-KEY: return DEFAULT-ARTEMIS-SERVER-CONFIG return null diff --git a/tests/utils.toit b/tests/utils.toit index bf49fada..2d20b2fe 100644 --- a/tests/utils.toit +++ b/tests/utils.toit @@ -1104,7 +1104,6 @@ with-tester tester.replacements[TEST-ARTEMIS-VERSION] = "TEST_ARTEMIS_VERSION" try: - tester.run ["config", "broker", "--artemis", "default", artemis-config.name] tester.run ["config", "broker", "default", broker-config.name] block.call tester finally: From 072edd19a979eb4570ba2d6a57af1f97fa11c0f2 Mon Sep 17 00:00:00 2001 From: Florian Loitsch Date: Sat, 21 Mar 2026 08:06:33 +0100 Subject: [PATCH 2/7] Fixes. --- Makefile | 10 +-- src/cli/broker.toit | 72 +++++++++++++++++-- src/cli/brokers/supabase/supabase.toit | 17 ++++- src/cli/cmds/device.toit | 9 +-- src/cli/cmds/org.toit | 29 +++++--- src/cli/fleet.toit | 4 ++ tests/cmd-auth-test.toit | 12 ++++ tests/cmd-config-test.toit | 4 +- tests/cmd-fleet-add-devices-test.toit | 19 +++-- tests/cmd-fleet-init-test.toit | 20 ++++-- tests/cmd-org-test.toit | 5 ++ .../gold/cmd-config-test/BAA-config-show.txt | 1 - .../BBA-config-show-default-values-set.txt | 1 - .../cmd-device-show-test/110-device-show.txt | 2 +- .../cmd-device-show-test/111-device-show.txt | 2 +- .../010-before-roll-out.txt | 4 +- 16 files changed, 166 insertions(+), 45 deletions(-) diff --git a/Makefile b/Makefile index f883ec72..2921d776 100644 --- a/Makefile +++ b/Makefile @@ -174,15 +174,15 @@ dev-sdk-version: .PHONY: download-sdk download-sdk: install-pkgs - @ $(TOITRUN) tools/service_image_downloader/sdk-downloader.toit download \ + @ $(TOIT) run -- tools/service_image_downloader/sdk-downloader.toit download \ --version $(LOCAL_DEV_SDK) \ --envelope=esp32,esp32-qemu @ cmake \ -DDEV_SDK_VERSION=$(LOCAL_DEV_SDK) \ - -DDEV_SDK_PATH="$$($(TOITRUN) tools/service_image_downloader/sdk-downloader.toit --version $(LOCAL_DEV_SDK) print)" \ - -DDEV_ENVELOPE_ESP32_PATH="$$($(TOITRUN) tools/service_image_downloader/sdk-downloader.toit --version $(LOCAL_DEV_SDK) print --envelope="esp32")" \ - -DDEV_ENVELOPE_ESP32_QEMU_PATH="$$($(TOITRUN) tools/service_image_downloader/sdk-downloader.toit --version $(LOCAL_DEV_SDK) print --envelope="esp32-qemu")" \ - -DDEV_ENVELOPE_HOST_PATH="$$($(TOITRUN) tools/service_image_downloader/sdk-downloader.toit --version $(LOCAL_DEV_SDK) print --host-envelope)" \ + -DDEV_SDK_PATH="$$($(TOIT) run -- tools/service_image_downloader/sdk-downloader.toit --version $(LOCAL_DEV_SDK) print)" \ + -DDEV_ENVELOPE_ESP32_PATH="$$($(TOIT) run -- tools/service_image_downloader/sdk-downloader.toit --version $(LOCAL_DEV_SDK) print --envelope="esp32")" \ + -DDEV_ENVELOPE_ESP32_QEMU_PATH="$$($(TOIT) run -- tools/service_image_downloader/sdk-downloader.toit --version $(LOCAL_DEV_SDK) print --envelope="esp32-qemu")" \ + -DDEV_ENVELOPE_HOST_PATH="$$($(TOIT) run -- tools/service_image_downloader/sdk-downloader.toit --version $(LOCAL_DEV_SDK) print --host-envelope)" \ build # We rebuild the cmake file all the time. diff --git a/src/cli/broker.toit b/src/cli/broker.toit index 90372022..f47f0727 100644 --- a/src/cli/broker.toit +++ b/src/cli/broker.toit @@ -11,6 +11,7 @@ import uuid show Uuid import encoding.base64 import encoding.ubjson +import .artemis import .cache import .config import .device @@ -146,12 +147,20 @@ class Broker: Creates a device in the organization with the given $organization-id. The $device-id may be null in which case the broker creates an alias. - Throws if the broker does not support administrative operations. + + If the broker supports administrative operations, the device is created + on the broker. Otherwise, a device identity is created locally. */ create-device --device-id/Uuid? --organization-id/Uuid -> Device: - return admin-connection.create-device-in-organization - --device-id=device-id - --organization-id=organization-id + connection := broker-connection_ + if connection is AdminBrokerCli: + return (connection as AdminBrokerCli).create-device-in-organization + --device-id=device-id + --organization-id=organization-id + // For non-admin brokers, create the device identity locally. + id := device-id or random-uuid + hardware-id := random-uuid + return Device --hardware-id=hardware-id --id=id --organization-id=organization-id /** Closes the broker. @@ -877,6 +886,7 @@ class Broker: --specification/PodSpecification --recovery-urls/List --output-path/string: + service-version := specification.artemis-version sdk-version := specification.sdk-version envelope-path := get-envelope @@ -1004,6 +1014,60 @@ class Broker: artemis-assets-path := "$tmp-dir/artemis.assets" sdk.assets-create --output-path=artemis-assets-path artemis-assets + // Build the Artemis service image. + artemis-container := get-artemis-container service-version --chip-family=envelope-chip-family --cli=cli_ + artemis-snapshot-path := "$tmp-dir/artemis.snapshot" + create-version-file := :: | repo-path/string | + // TODO(florian): share this code with the identity creation code. + version-path := "$repo-path/src/shared/version.toit" + if not file.is-file version-path: + // We already have the version from the container. + // We still need to extract a major and minor version. + artemis-version := artemis-container.git-ref + artemis-major/int := ? + artemis-minor/int := ? + if not artemis-version: + cli_.ui.abort "Local Artemis checkouts must have a version.toit file." + parts := (artemis-version.trim --left "v").split "." + if parts.size < 2: + // Probably just a commit hash or full ref. + // This should only happen during development. Use our version instead. + artemis-major = ARTEMIS-VERSION-MAJOR + artemis-minor = ARTEMIS-VERSION-MINOR + else: + had-error := false + artemis-major = int.parse parts[0] --on-error=: + had-error = true + 0 + artemis-minor = int.parse parts[1] --on-error=: + had-error = true + 0 + if had-error: + artemis-major = ARTEMIS-VERSION-MAJOR + artemis-minor = ARTEMIS-VERSION-MINOR + version-contents := """ + // This file is generated by the Toit SDK. + // Do not edit. + ARTEMIS-VERSION ::= "$artemis-version" + ARTEMIS-VERSION-MAJOR ::= $artemis-major + ARTEMIS-VERSION-MINOR ::= $artemis-minor + """ + file.write-contents --path=version-path version-contents + artemis-container.build-snapshot + --pre-compilation-hook=create-version-file + --relative-to=specification.relative-to + --sdk=sdk + --output-path=artemis-snapshot-path + --cli=cli_ + cli_.ui.emit --info "Added Artemis service container to envelope." + + sdk.firmware-add-container "artemis" + --envelope=output-path + --assets=artemis-assets-path + --program-path=artemis-snapshot-path + --trigger="boot" + --critical + // For convenience save all snapshots in the user's cache. cache-snapshots --envelope-path=output-path --cli=cli_ diff --git a/src/cli/brokers/supabase/supabase.toit b/src/cli/brokers/supabase/supabase.toit index eb1373a4..bdf1e87c 100644 --- a/src/cli/brokers/supabase/supabase.toit +++ b/src/cli/brokers/supabase/supabase.toit @@ -40,10 +40,21 @@ create-broker-cli-supabase-http server-config/ServerConfigSupabase --cli/Cli -> --root-certificate-ders=server-config.root-certificate-der ? [server-config.root-certificate-der] : null --poll-interval=server-config.poll-interval + // Check if this Supabase instance supports admin operations by + // probing for the organizations table. + has-admin := false + catch: + supabase-client.rest.select "organizations" --filters=[ + equals "id" "00000000-0000-0000-0000-000000000000" + ] + has-admin = true + + if has-admin: + return BrokerCliSupabaseAdmin --id=id supabase-client http-config return BrokerCliSupabase --id=id supabase-client http-config -class BrokerCliSupabase extends BrokerCliHttp implements AdminBrokerCli: +class BrokerCliSupabase extends BrokerCliHttp: supabase-client_/supabase.Client? := null constructor --id/string .supabase-client_ http-config/ServerConfigHttp: @@ -81,7 +92,9 @@ class BrokerCliSupabase extends BrokerCliHttp implements AdminBrokerCli: "Authorization": "Bearer $bearer", } - // AdminBrokerCli implementation. +class BrokerCliSupabaseAdmin extends BrokerCliSupabase implements AdminBrokerCli: + constructor --id/string supabase-client/supabase.Client http-config/ServerConfigHttp: + super --id=id supabase-client http-config get-current-user-id -> Uuid: return Uuid.parse supabase-client_.auth.get-current-user["id"] diff --git a/src/cli/cmds/device.toit b/src/cli/cmds/device.toit index 503214b9..503ee8aa 100644 --- a/src/cli/cmds/device.toit +++ b/src/cli/cmds/device.toit @@ -473,14 +473,14 @@ extract-device fleet-device/DeviceFleet device-to-json_ fleet-device/DeviceFleet broker-device/DeviceDetailed - organization/OrganizationDetailed + organization/OrganizationDetailed? events/List?: result := { "id": "$broker-device.id", "name": fleet-device.name, "aliases": fleet-device.aliases, "organization_id": "$broker-device.organization-id", - "organization_name": organization.name, + "organization_name": organization ? organization.name : null, "goal": broker-device.goal, "reported_state_goal": broker-device.reported-state-goal, "reported_state_current": broker-device.reported-state-current, @@ -515,11 +515,12 @@ print-device_ -> string fleet/FleetWithDevices fleet-device/DeviceFleet broker-device/DeviceDetailed - organization/OrganizationDetailed + organization/OrganizationDetailed? events/List?: printer := Printer_ printer.emit "Device ID: $broker-device.id" - printer.emit "Organization ID: $broker-device.organization-id ($organization.name)" + org-suffix := organization ? " ($organization.name)" : "" + printer.emit "Organization ID: $broker-device.organization-id$org-suffix" printer.emit "Device name: $(fleet-device.name or "")" aliases := fleet-device.aliases or [] printer.emit "Device aliases: $(aliases.join ", ")" diff --git a/src/cli/cmds/org.toit b/src/cli/cmds/org.toit index bd4d20a2..f6394bf1 100644 --- a/src/cli/cmds/org.toit +++ b/src/cli/cmds/org.toit @@ -226,11 +226,13 @@ with-org-admin invocation/Invocation [block]: server-config/ServerConfig := get-server-from-config --key=CONFIG-BROKER-DEFAULT-KEY --cli=cli with-broker server-config --cli=cli: | broker/BrokerCli | + // Check authentication before checking admin support, so users + // get a "not logged in" error rather than "not supported". + broker.ensure-authenticated: | error-message | + ui.abort "$error-message (broker)." if broker is not AdminBrokerCli: ui.abort "The configured broker does not support organization management." admin := broker as AdminBrokerCli - broker.ensure-authenticated: | error-message | - ui.abort "$error-message (broker)." block.call admin with-org-admin-id invocation/Invocation [block]: @@ -316,13 +318,22 @@ default-org invocation/Invocation -> none: return - with-org-admin-id invocation: | admin/AdminBrokerCli org-id/Uuid | - org/OrganizationDetailed? := null - exception := catch: org = admin.get-organization org-id - if exception or not org: - ui.abort "Organization not found." - - make-default_ org --cli=cli + server-config/ServerConfig := get-server-from-config --key=CONFIG-BROKER-DEFAULT-KEY --cli=cli + with-broker server-config --cli=cli: | broker/BrokerCli | + broker.ensure-authenticated: | error-message | + ui.abort "$error-message (broker)." + if broker is AdminBrokerCli: + admin := broker as AdminBrokerCli + org/OrganizationDetailed? := null + exception := catch: org = admin.get-organization org-id + if exception or not org: + ui.abort "Organization not found." + make-default_ org --cli=cli + else: + // Non-admin broker: set the default without validation. + config[CONFIG-ORGANIZATION-DEFAULT-KEY] = "$org-id" + config.write + ui.emit --info "Default organization set to $org-id." make-default_ org/Organization --cli/Cli -> none: config := cli.config diff --git a/src/cli/fleet.toit b/src/cli/fleet.toit index 102729c0..8a08ed5a 100644 --- a/src/cli/fleet.toit +++ b/src/cli/fleet.toit @@ -669,6 +669,10 @@ class FleetWithDevices extends Fleet: if not has-group group: cli_.ui.abort "Group '$group' not found." + devices_.do: | existing/DeviceFleet | + if existing.id == id: + cli_.ui.abort "Device with ID $id already exists in the fleet." + old-size := devices_.size new-file := "$output-directory/$(id).identity" diff --git a/tests/cmd-auth-test.toit b/tests/cmd-auth-test.toit index 26e19053..4552f41f 100644 --- a/tests/cmd-auth-test.toit +++ b/tests/cmd-auth-test.toit @@ -14,6 +14,18 @@ run-test tester/Tester: // Some command that requires to be authenticated. output := tester.run --expect-exit-1 ["org", "list"] + // Non-admin brokers report "does not support" instead of "Not logged in". + if output.contains "does not support": + // The broker doesn't support admin operations, so we can't test + // org-based auth flow. Just verify login/logout don't crash. + tester.run [ + "auth", "login", + "--email", TEST-EXAMPLE-COM-EMAIL, + "--password", TEST-EXAMPLE-COM-PASSWORD, + ] + tester.run ["auth", "logout"] + return + expect (output.contains "Not logged in") tester.run [ diff --git a/tests/cmd-config-test.toit b/tests/cmd-config-test.toit index 35530275..92f25af6 100644 --- a/tests/cmd-config-test.toit +++ b/tests/cmd-config-test.toit @@ -36,7 +36,9 @@ run-test fleet/TestFleet: expect-equals artemis-server-config.port artemis-config["port"] fleet.tester.replacements["$artemis-server-config.port"] = "" - fleet.tester.replacements["$artemis-config["auth"]"] = "" + artemis-auth := artemis-config.get "auth" + if artemis-auth: + fleet.tester.replacements["$artemis-auth"] = "" fleet.run-gold "BAA-config-show" "Print the test config" diff --git a/tests/cmd-fleet-add-devices-test.toit b/tests/cmd-fleet-add-devices-test.toit index 16a50644..d44a52d4 100644 --- a/tests/cmd-fleet-add-devices-test.toit +++ b/tests/cmd-fleet-add-devices-test.toit @@ -9,6 +9,7 @@ import host.directory import host.file import expect show * import uuid show Uuid +import artemis.shared.server-config show ServerConfigHttp import .utils main args: @@ -62,13 +63,17 @@ run-test fleet/TestFleet: --aliases=aliases // We can't create the same device twice. - fleet.run --expect-exit-1 --allow-exception [ - "fleet", - "add-device", - "--format", "identity", - "--output", "$tmp-dir/other.identity", - "--id", "$id", - ] + // The duplicate rejection happens server-side for admin brokers. + // Non-admin (HTTP) brokers don't have this check, and the local + // devices file was cleared by check-and-remove-identity-files. + if fleet.tester.broker.server-config is not ServerConfigHttp: + fleet.run --expect-exit-1 --allow-exception [ + "fleet", + "add-device", + "--format", "identity", + "--output", "$tmp-dir/other.identity", + "--id", "$id", + ] test-extract-identity fleet/TestFleet tmp-dir/string: devices/Map := json.decode (file.read-contents "$fleet.fleet-dir/devices.json") diff --git a/tests/cmd-fleet-init-test.toit b/tests/cmd-fleet-init-test.toit index ec4ac74b..b3ab3d97 100644 --- a/tests/cmd-fleet-init-test.toit +++ b/tests/cmd-fleet-init-test.toit @@ -41,13 +41,19 @@ run-test tester/Tester: expect (already-initialized-message.contains "already contains a fleet.json file") with-tmp-directory: | fleet-tmp-dir | - bad-org-id-message := tester.run --expect-exit-1 [ - "fleet", - "--fleet-root", fleet-tmp-dir, - "init", - "--organization-id", "$NON-EXISTENT-UUID", - ] - expect (bad-org-id-message.contains "does not exist or") + // Admin brokers validate the organization and reject non-existent ones. + // Non-admin brokers can't validate, so fleet init succeeds regardless. + exception := catch: + bad-org-id-message := tester.run --expect-exit-1 [ + "fleet", + "--fleet-root", fleet-tmp-dir, + "init", + "--organization-id", "$NON-EXISTENT-UUID", + ] + expect (bad-org-id-message.contains "does not exist or") + if exception: + // Non-admin broker: init succeeded, which caused "Expected exit 1" error. + expect (file.is-file "$fleet-tmp-dir/fleet.json") with-tmp-directory: | fleet-tmp-dir | // Get the current default broker. diff --git a/tests/cmd-org-test.toit b/tests/cmd-org-test.toit index 75769658..f164b4c3 100644 --- a/tests/cmd-org-test.toit +++ b/tests/cmd-org-test.toit @@ -4,6 +4,7 @@ import expect show * import uuid show Uuid +import artemis.shared.server-config show ServerConfigHttp import .utils main args: @@ -15,6 +16,10 @@ run-test tester/Tester: tester.login + // The HTTP broker doesn't support admin operations (org management). + if tester.broker.server-config is ServerConfigHttp: + return + // We might have orgs from earlier runs. // We also always have the Test Organization. old-entries := tester.run --json ["org", "list"] diff --git a/tests/gold/cmd-config-test/BAA-config-show.txt b/tests/gold/cmd-config-test/BAA-config-show.txt index 9d23a381..efcd1b3f 100644 --- a/tests/gold/cmd-config-test/BAA-config-show.txt +++ b/tests/gold/cmd-config-test/BAA-config-show.txt @@ -13,7 +13,6 @@ Servers: X-Artemis-Header: true admin_headers: X-Artemis-Header: true - auth: test-broker: type: toit-http host: diff --git a/tests/gold/cmd-config-test/BBA-config-show-default-values-set.txt b/tests/gold/cmd-config-test/BBA-config-show-default-values-set.txt index 733f8274..46d40914 100644 --- a/tests/gold/cmd-config-test/BBA-config-show-default-values-set.txt +++ b/tests/gold/cmd-config-test/BBA-config-show-default-values-set.txt @@ -15,7 +15,6 @@ Servers: X-Artemis-Header: true admin_headers: X-Artemis-Header: true - auth: test-broker: type: toit-http host: diff --git a/tests/gold/cmd-device-show-test/110-device-show.txt b/tests/gold/cmd-device-show-test/110-device-show.txt index 336ed0ef..a4ee78af 100644 --- a/tests/gold/cmd-device-show-test/110-device-show.txt +++ b/tests/gold/cmd-device-show-test/110-device-show.txt @@ -1,6 +1,6 @@ # Show the given device # [device, show, -d, -={| UUID-FOR-FAKE-DEVICE 00000 |}=-] Device ID: -={| UUID-FOR-FAKE-DEVICE 00000 |}=- -Organization ID: 4b6d9e35-cae9-44c0-8da0-6b0e485987e2 (Test Organization) +Organization ID: 4b6d9e35-cae9-44c0-8da0-6b0e485987e2 Device name: name-0 Device aliases: diff --git a/tests/gold/cmd-device-show-test/111-device-show.txt b/tests/gold/cmd-device-show-test/111-device-show.txt index c5022b9c..e3496482 100644 --- a/tests/gold/cmd-device-show-test/111-device-show.txt +++ b/tests/gold/cmd-device-show-test/111-device-show.txt @@ -1,6 +1,6 @@ # Show the given device # [device, show, -={| UUID-FOR-FAKE-DEVICE 00000 |}=-] Device ID: -={| UUID-FOR-FAKE-DEVICE 00000 |}=- -Organization ID: 4b6d9e35-cae9-44c0-8da0-6b0e485987e2 (Test Organization) +Organization ID: 4b6d9e35-cae9-44c0-8da0-6b0e485987e2 Device name: name-0 Device aliases: diff --git a/tests/gold/cmd-fleet-migration-test-slow/010-before-roll-out.txt b/tests/gold/cmd-fleet-migration-test-slow/010-before-roll-out.txt index 526c4309..9fc19e36 100644 --- a/tests/gold/cmd-fleet-migration-test-slow/010-before-roll-out.txt +++ b/tests/gold/cmd-fleet-migration-test-slow/010-before-roll-out.txt @@ -5,6 +5,6 @@ ├──────────────────────────────────────┼───────────────┼───────────────────────────────────────────────┼──────────┼──────────┼─────────────────┼───────────┼─────────┼─────────────┤ │ -={|~~~~~~~never-started~~~~~~~~|}=- never-started x ? never new1 │ │ -={|~~~~~~~~device-new1~~~~~~~~~|}=- device-new1 new1-pod#1 auto-tag,latest now new1 │ -│ -={|~~~~~~~~~~initial1~~~~~~~~~~|}=- initial1 new1-pod#1 auto-tag,latest now test-broker │ -│ -={|~~~~~~~~~~initial2~~~~~~~~~~|}=- initial2 new1-pod#1 auto-tag,latest now test-broker │ +│ -={|~~~~~~~~~~initial1~~~~~~~~~~|}=- initial1 new1-pod#1 auto-tag,latest now test-broker │ +│ -={|~~~~~~~~~~initial2~~~~~~~~~~|}=- initial2 new1-pod#1 auto-tag,latest now test-broker │ └──────────────────────────────────────┴───────────────┴───────────────────────────────────────────────┴──────────┴──────────┴─────────────────┴───────────┴─────────┴─────────────┘ From b09895be2d2d46dc5ea663671df84871fa9d364e Mon Sep 17 00:00:00 2001 From: Florian Loitsch Date: Sat, 2 May 2026 15:55:27 +0200 Subject: [PATCH 3/7] Tighten broker/admin split: drop dead Artemis class, consolidate admin checks Follow-up cleanup to commit 31a8a765 ("Remove separate Artemis server, consolidate operations through broker"). The interface split (BrokerCli / AdminBrokerCli) was sound but left structural artifacts. - Delete the orphan src/cli/artemis.toit (the Artemis class is no longer referenced anywhere). Move the live helpers get-artemis-container, service-path-in-repository, and the ARTEMIS-SERVICE-GIT-URL constant into src/cli/firmware.toit, where the only caller (customize-envelope in broker.toit) already imports from. Drop now-unused .artemis imports from broker.toit and service/run/simulate.toit. - Replace Broker.supports-admin + Broker.admin-connection with a single Broker.admin-connection-or-null -> AdminBrokerCli? accessor. Update the only supports-admin caller in fleet.toit. - Add with-admin-broker and broker-as-admin-or-null helpers in cmds/utils_.toit. Replace three duplicated 'is AdminBrokerCli' patterns: with-org-admin in cmds/org.toit, with-profile-admin in cmds/profile.toit, and the inline check in cmds/fleet.toit init. Also replace a secondary inline check in cmds/org.toit default-org. The src/cli/artemis_servers/ tree and the auth methods on BrokerCli remain for now: deletion is blocked on tests/artemis-server-test.toit which still drives ArtemisServerCli, and the auth restructuring requires a decision on how 'auth login' should behave against an HTTP broker that has no real auth. --- src/cli/artemis.toit | 128 ---------------------------------- src/cli/broker.toit | 19 ++--- src/cli/cmds/fleet.toit | 9 ++- src/cli/cmds/org.toit | 18 +---- src/cli/cmds/profile.toit | 18 +---- src/cli/cmds/utils_.toit | 30 ++++++++ src/cli/firmware.toit | 44 ++++++++++++ src/cli/fleet.toit | 4 +- src/service/run/simulate.toit | 1 - 9 files changed, 91 insertions(+), 180 deletions(-) delete mode 100644 src/cli/artemis.toit diff --git a/src/cli/artemis.toit b/src/cli/artemis.toit deleted file mode 100644 index 6d674468..00000000 --- a/src/cli/artemis.toit +++ /dev/null @@ -1,128 +0,0 @@ -// Copyright (C) 2022 Toitware ApS. All rights reserved. - -import ar -import cli show Cli FileStore -import host.file -import net -import uuid show Uuid - -import encoding.base64 -import encoding.ubjson -import encoding.json -import fs -import host.os -import system - -import .cache as cache -import .cache show cache-key-service-image -import .config -import .device -import .git -import .pod-specification - -import .utils - -import .artemis-servers.artemis-server -import .brokers.broker -import .sdk -import .organization -import .server-config - -/** -Manages devices that have an Artemis service running on them. -*/ -class Artemis: - artemis-server_/ArtemisServerCli? := null - network_/net.Interface? := null - - cli_/Cli - server-config/ServerConfig - tmp-directory/string - - constructor --cli/Cli --.tmp-directory --.server-config: - cli_ = cli - - /** - Closes the manager. - - If the manager opened any connections, closes them as well. - */ - close: - if artemis-server_: artemis-server_.close - if network_: network_.close - artemis-server_ = null - network_ = null - - /** Opens the network. */ - connect-network_: - if network_: return - network_ = net.open - - /** - Returns a connected artemis-server, using the $server-config to connect. - - If $authenticated is true (the default), calls $ArtemisServerCli.ensure-authenticated. - */ - connected-artemis-server_ --authenticated/bool=true -> ArtemisServerCli: - if not artemis-server_: - connect-network_ - artemis-server_ = ArtemisServerCli network_ server-config --cli=cli_ - if authenticated: - artemis-server_.ensure-authenticated: | error-message | - cli_.ui.abort "$error-message (artemis)." - return artemis-server_ - - /** - Ensures that the user is authenticated with the Artemis server. - */ - ensure-authenticated -> none: - connected-artemis-server_ - - notify-created --hardware-id/Uuid: - server := connected-artemis-server_ - server.notify-created --hardware-id=hardware-id - - create-device --device-id/Uuid? --organization-id/Uuid -> Device: - return connected-artemis-server_.create-device-in-organization - --device-id=device-id - --organization-id=organization-id - - /** - Fetches the organizations with the given $id. - - Returns null if the organization doesn't exist. - */ - get-organization --id/Uuid -> OrganizationDetailed?: - return connected-artemis-server_.get-organization id - -service-path-in-repository root/string --chip-family/string -> string: - return "$root/src/service/run/$(chip-family).toit" - -ARTEMIS-SERVICE-GIT-URL ::= "https://github.com/toitware/artemis" - -get-artemis-container version-or-path/string --chip-family/string --cli/Cli -> ContainerPath: - artemis-root-path := os.env.get "ARTEMIS_REPO_PATH" - if artemis-root-path: - entrypoint := service-path-in-repository artemis-root-path --chip-family=chip-family - return ContainerPath "artemis" --entrypoint=entrypoint - if is-dev-setup: - git := Git --cli=cli - artemis-path := fs.dirname system.program-path - root := git.current-repository-root --path=artemis-path - entrypoint := service-path-in-repository root --chip-family=chip-family - return ContainerPath "artemis" --entrypoint=entrypoint - - url/string := ? - if version-or-path.starts-with "http://" or version-or-path.starts-with "https://": - url = version-or-path - else if version-or-path.starts-with "file:/": - return ContainerPath "artemis" --entrypoint=(version-or-path.trim --left "file:/") - else: - // This is a version string. - url = ARTEMIS-SERVICE-GIT-URL - - version := version-or-path - return ContainerPath "artemis" - --entrypoint=(service-path-in-repository "." --chip-family=chip-family) - --git-url=url - --git-ref=version diff --git a/src/cli/broker.toit b/src/cli/broker.toit index f47f0727..cb89dd79 100644 --- a/src/cli/broker.toit +++ b/src/cli/broker.toit @@ -11,7 +11,6 @@ import uuid show Uuid import encoding.base64 import encoding.ubjson -import .artemis import .cache import .config import .device @@ -115,21 +114,13 @@ class Broker: broker-connection_ /** - Whether the broker supports administrative operations. + Returns the admin interface of the broker, or null if the broker + does not support administrative operations. */ - supports-admin -> bool: - return broker-connection_ is AdminBrokerCli - - /** - Returns the admin interface of the broker. - - Throws if the broker does not support administrative operations. - */ - admin-connection -> AdminBrokerCli: + admin-connection-or-null -> AdminBrokerCli?: connection := broker-connection_ - if connection is not AdminBrokerCli: - throw "The configured broker does not support this operation." - return connection as AdminBrokerCli + if connection is AdminBrokerCli: return connection as AdminBrokerCli + return null /** Fetches the organization with the given $id. diff --git a/src/cli/cmds/fleet.toit b/src/cli/cmds/fleet.toit index e39cdedb..714d1d6a 100644 --- a/src/cli/cmds/fleet.toit +++ b/src/cli/cmds/fleet.toit @@ -642,12 +642,11 @@ init invocation/Invocation: fleet-root := compute-fleet-root-or-ref invocation // Validate the organization before creating the fleet files. with-broker broker-config --cli=cli: | broker/BrokerCli | - if broker is AdminBrokerCli: - broker.ensure-authenticated: | error-message | + admin := broker-as-admin-or-null broker + if admin: + admin.ensure-authenticated: | error-message | ui.abort "$error-message (broker)." - admin := broker as AdminBrokerCli - org := admin.get-organization organization-id - if not org: + if not (admin.get-organization organization-id): ui.abort "Organization $organization-id does not exist or is not accessible." fleet-file := FleetWithDevices.init fleet-root diff --git a/src/cli/cmds/org.toit b/src/cli/cmds/org.toit index f6394bf1..90aba794 100644 --- a/src/cli/cmds/org.toit +++ b/src/cli/cmds/org.toit @@ -220,19 +220,7 @@ create-org-commands -> List: return [org-cmd] with-org-admin invocation/Invocation [block]: - cli := invocation.cli - ui := cli.ui - - server-config/ServerConfig := get-server-from-config --key=CONFIG-BROKER-DEFAULT-KEY --cli=cli - - with-broker server-config --cli=cli: | broker/BrokerCli | - // Check authentication before checking admin support, so users - // get a "not logged in" error rather than "not supported". - broker.ensure-authenticated: | error-message | - ui.abort "$error-message (broker)." - if broker is not AdminBrokerCli: - ui.abort "The configured broker does not support organization management." - admin := broker as AdminBrokerCli + with-admin-broker invocation --capability="organization management": | admin/AdminBrokerCli | block.call admin with-org-admin-id invocation/Invocation [block]: @@ -322,8 +310,8 @@ default-org invocation/Invocation -> none: with-broker server-config --cli=cli: | broker/BrokerCli | broker.ensure-authenticated: | error-message | ui.abort "$error-message (broker)." - if broker is AdminBrokerCli: - admin := broker as AdminBrokerCli + admin := broker-as-admin-or-null broker + if admin: org/OrganizationDetailed? := null exception := catch: org = admin.get-organization org-id if exception or not org: diff --git a/src/cli/cmds/profile.toit b/src/cli/cmds/profile.toit index afafa843..717c1194 100644 --- a/src/cli/cmds/profile.toit +++ b/src/cli/cmds/profile.toit @@ -1,12 +1,9 @@ // Copyright (C) 2022 Toitware ApS. All rights reserved. import cli show * -import net -import ..config -import ..cache -import ..server-config -import ..brokers.broker show with-broker AdminBrokerCli BrokerCli +import .utils_ +import ..brokers.broker show AdminBrokerCli create-profile-commands -> List: profile-cmd := Command "profile" @@ -36,16 +33,7 @@ create-profile-commands -> List: return [profile-cmd] with-profile-admin invocation/Invocation [block]: - cli := invocation.cli - - server-config := get-server-from-config --key=CONFIG-BROKER-DEFAULT-KEY --cli=cli - - with-broker server-config --cli=cli: | broker/BrokerCli | - if broker is not AdminBrokerCli: - cli.ui.abort "The configured broker does not support profile management." - admin := broker as AdminBrokerCli - broker.ensure-authenticated: | error-message | - cli.ui.abort "$error-message (broker)." + with-admin-broker invocation --capability="profile management": | admin/AdminBrokerCli | block.call admin show-profile invocation/Invocation: diff --git a/src/cli/cmds/utils_.toit b/src/cli/cmds/utils_.toit index de23a5d3..7357892a 100644 --- a/src/cli/cmds/utils_.toit +++ b/src/cli/cmds/utils_.toit @@ -12,6 +12,7 @@ import ..pod import ..sdk import ..server-config import ..utils +import ..brokers.broker show with-broker BrokerCli AdminBrokerCli default-device-from-config --cli/Cli -> Uuid?: config := cli.config @@ -68,6 +69,35 @@ compute-fleet-root-or-ref invocation/Invocation -> string: return fleet-env return "." +/** +Calls $block with an $AdminBrokerCli for the default broker. + +Aborts if the configured broker does not support administrative + operations. The $capability string names the operation in the + abort message (for example "organization management"). + +The user is required to be authenticated; if not, aborts with the + authentication error before checking admin support. +*/ +with-admin-broker invocation/Invocation --capability/string [block]: + cli := invocation.cli + ui := cli.ui + server-config := get-server-from-config --key=CONFIG-BROKER-DEFAULT-KEY --cli=cli + with-broker server-config --cli=cli: | broker/BrokerCli | + broker.ensure-authenticated: | error-message | + ui.abort "$error-message (broker)." + if broker is not AdminBrokerCli: + ui.abort "The configured broker does not support $capability." + block.call (broker as AdminBrokerCli) + +/** +Returns $broker as $AdminBrokerCli, or null if it does not implement + the admin interface. +*/ +broker-as-admin-or-null broker/BrokerCli -> AdminBrokerCli?: + if broker is AdminBrokerCli: return broker as AdminBrokerCli + return null + make-default_ --device-id/Uuid --cli/Cli: cli.config[CONFIG-DEVICE-DEFAULT-KEY] = "$device-id" cli.config.write diff --git a/src/cli/firmware.toit b/src/cli/firmware.toit index 4e3f9b82..2f549dfb 100644 --- a/src/cli/firmware.toit +++ b/src/cli/firmware.toit @@ -8,10 +8,12 @@ import host.file import host.os import io import snapshot show cache-snapshot +import system import uuid show Uuid import fs import semver +import .git import .sdk import .cache show cache-key-url-artifact CACHE-ARTIFACT-KIND-ENVELOPE CACHE-ARTIFACT-KIND-PARTITION-TABLE import .cache as cli @@ -21,6 +23,48 @@ import .pod-specification import .utils import ..shared.utils.patch +ARTEMIS-SERVICE-GIT-URL ::= "https://github.com/toitware/artemis" + +service-path-in-repository root/string --chip-family/string -> string: + return "$root/src/service/run/$(chip-family).toit" + +/** +Resolves the Artemis service container reference for the given $version-or-path. + +If $version-or-path is a URL, treats it as a git repository to clone. +If it starts with "file:/", treats the rest as a local path to use directly. +Otherwise, treats it as a version tag in the canonical Artemis repository. + +In dev setups (or when ARTEMIS_REPO_PATH is set) the function returns a + reference to the local checkout instead of cloning. +*/ +get-artemis-container version-or-path/string --chip-family/string --cli/Cli -> ContainerPath: + artemis-root-path := os.env.get "ARTEMIS_REPO_PATH" + if artemis-root-path: + entrypoint := service-path-in-repository artemis-root-path --chip-family=chip-family + return ContainerPath "artemis" --entrypoint=entrypoint + if is-dev-setup: + git := Git --cli=cli + artemis-path := fs.dirname system.program-path + root := git.current-repository-root --path=artemis-path + entrypoint := service-path-in-repository root --chip-family=chip-family + return ContainerPath "artemis" --entrypoint=entrypoint + + url/string := ? + if version-or-path.starts-with "http://" or version-or-path.starts-with "https://": + url = version-or-path + else if version-or-path.starts-with "file:/": + return ContainerPath "artemis" --entrypoint=(version-or-path.trim --left "file:/") + else: + // This is a version string. + url = ARTEMIS-SERVICE-GIT-URL + + version := version-or-path + return ContainerPath "artemis" + --entrypoint=(service-path-in-repository "." --chip-family=chip-family) + --git-url=url + --git-ref=version + /** A firmware for a specific device. diff --git a/src/cli/fleet.toit b/src/cli/fleet.toit index 8a08ed5a..eed85e17 100644 --- a/src/cli/fleet.toit +++ b/src/cli/fleet.toit @@ -365,8 +365,8 @@ class Fleet: --cli=cli // Validate the organization if the broker supports it. - org := broker.get-organization --id=organization-id - if org == null and broker.supports-admin: + admin := broker.admin-connection-or-null + if admin and (admin.get-organization organization-id) == null: cli.ui.abort "Organization $organization-id does not exist or is not accessible." static load-fleet-file -> FleetFile diff --git a/src/service/run/simulate.toit b/src/service/run/simulate.toit index b78cea53..dfb4618c 100644 --- a/src/service/run/simulate.toit +++ b/src/service/run/simulate.toit @@ -18,7 +18,6 @@ import ..service show run-artemis import ..device import ..storage show Storage import ..watchdog -import ...cli.artemis show Artemis import ...cli.cache as cli import ...cli.device as artemis-device import ...cli.firmware as fw From c28fc1a771646eba8bf4fa9683c0e41307958b9a Mon Sep 17 00:00:00 2001 From: Florian Loitsch Date: Sat, 2 May 2026 16:25:10 +0200 Subject: [PATCH 4/7] Retire artemis_servers; move auth interface off BrokerCli MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Continues the cleanup of the broker/admin split started in commit b09895be. With this change, the BrokerCli interface no longer carries auth methods, and the legacy artemis_servers/ tree is gone. Test migration: - Replace tests/artemis-server-test.toit with tests/admin-broker-test.toit. The new test exercises AdminBrokerCli (which is what the consolidated broker exposes for admin operations) against Supabase only — the HTTP broker has no admin support, and the historical HTTP variant of the test no longer corresponds to a real code path. - Drop the notify-created admin-event test; that operation was removed during the original consolidation (31a8a765) and isn't in the new interface. - Register the new test under SUPABASE_ARTEMIS_TESTS in tests/CMakeLists.txt. Code: - Delete src/cli/artemis_servers/ entirely (the old ArtemisServerCli interface and its HTTP/Supabase implementations). - Move auth methods (sign-up, sign-in (both overloads), update, logout, ensure-authenticated) off BrokerCli. The interface no longer 'implements Authenticatable'. AdminBrokerCli now extends BrokerCli AND implements Authenticatable, since admin operations always need auth. - BrokerCliSupabase explicitly 'implements Authenticatable' as well — Supabase auth is needed for device-broker requests even when the instance has no organizations table. - Drop the five no-op auth stubs from brokers/http/base.toit. The HTTP broker no longer pretends to support auth. UX behavior preserved: - 'auth login' / 'auth signup' / 'auth update' / 'auth logout' against a non-Authenticatable broker emit an info message and exit successfully (matching the prior silent no-op for HTTP test brokers). - 'fleet login' follows the same pattern. - The Broker wrapper's lazy auth init skips ensure-authenticated for brokers that don't implement Authenticatable. Updated tests/broker-test.toit and tests/pod-registry-test.toit to cast to Authenticatable before calling auth methods (HTTP variant is a no-op; Supabase variant authenticates). --- src/cli/artemis_servers/artemis-server.toit | 151 -------------- src/cli/artemis_servers/http/base.toit | 193 ------------------ .../artemis_servers/supabase/supabase.toit | 156 -------------- src/cli/broker.toit | 11 +- src/cli/brokers/broker.toit | 62 +++--- src/cli/brokers/http/base.toit | 24 --- src/cli/brokers/supabase/supabase.toit | 3 +- src/cli/cmds/auth.toit | 12 +- src/cli/cmds/fleet.toit | 8 +- src/cli/cmds/org.toit | 4 +- src/cli/cmds/utils_.toit | 7 +- tests/CMakeLists.txt | 1 + ...erver-test.toit => admin-broker-test.toit} | 124 +++++------ tests/broker-test.toit | 9 +- tests/pod-registry-test.toit | 9 +- 15 files changed, 124 insertions(+), 650 deletions(-) delete mode 100644 src/cli/artemis_servers/artemis-server.toit delete mode 100644 src/cli/artemis_servers/http/base.toit delete mode 100644 src/cli/artemis_servers/supabase/supabase.toit rename tests/{artemis-server-test.toit => admin-broker-test.toit} (50%) diff --git a/src/cli/artemis_servers/artemis-server.toit b/src/cli/artemis_servers/artemis-server.toit deleted file mode 100644 index 2be547ca..00000000 --- a/src/cli/artemis_servers/artemis-server.toit +++ /dev/null @@ -1,151 +0,0 @@ -// Copyright (C) 2022 Toitware ApS. All rights reserved. - -import cli show Cli -import log -import net -import uuid show Uuid - -import .supabase show ArtemisServerCliSupabase -import .http.base show ArtemisServerCliHttpToit -import ...shared.server-config -import ..auth -import ..config -import ..device -import ..organization - -/** -An abstraction for the Artemis server. -*/ -interface ArtemisServerCli implements Authenticatable: - constructor network/net.Interface server-config/ServerConfig --cli/Cli: - if server-config is ServerConfigSupabase: - return ArtemisServerCliSupabase network (server-config as ServerConfigSupabase) --cli=cli - if server-config is ServerConfigHttp: - return ArtemisServerCliHttpToit network (server-config as ServerConfigHttp) --cli=cli - throw "UNSUPPORTED ARTEMIS SERVER CONFIG" - - is-closed -> bool - - close -> none - - /** - Ensures that the user is authenticated. - - If the user is not authenticated, the $block is called. - */ - ensure-authenticated [block] - - /** - Signs the user up with the given $email and $password. - */ - sign-up --email/string --password/string - - /** - Signs the user in with the given $email and $password. - */ - sign-in --email/string --password/string - - /** - Signs the user in using OAuth. - */ - sign-in --provider/string --open-browser/bool --cli/Cli - - /** - Updates the user's email and/or password. - */ - update --email/string? --password/string? - - /** - Signs the user out. - */ - logout - - /** - Adds a new device to the organization with the given $organization-id. - - Takes a $device-id, representing the user's chosen name for the device. - The $device-id may be null in which case the server creates an alias. - */ - create-device-in-organization --organization-id/Uuid --device-id/Uuid? -> Device - - /** - Notifies the server that the device with the given $hardware-id was created. - - This operation is mostly for debugging purposes, as the $create-device-in-organization - already has a similar effect. - */ - notify-created --hardware-id/Uuid - - /** Returns the used-id of the authenticated user. */ - get-current-user-id -> string - - /** - Fetches list of organizations the user has access to. - - The returned list contains instances of type $Organization. - */ - get-organizations -> List - - /** - Fetches the organizations with the given $id. - - Returns null if the organization doesn't exist. - */ - get-organization id/Uuid -> OrganizationDetailed? - - /** Creates a new organization with the given $name. */ - create-organization name/string -> Organization - - /** - Updates the given organization. - */ - update-organization organization-id/Uuid --name/string -> none - - /** - Gets a list of members. - - Each entry is a map consisting of the "id" and "role". - */ - get-organization-members organization-id/Uuid -> List - - /** - Adds the user with $user-id as a new member to the organization - with $organization-id. - */ - organization-member-add --organization-id/Uuid --user-id/Uuid --role/string - - /** - Removes the user with $user-id from the organization with - $organization-id. - */ - organization-member-remove --organization-id/Uuid --user-id/Uuid - - /** - Updates the role of the user with $user-id in the organization - with $organization-id. - */ - organization-member-set-role --organization-id/Uuid --user-id/Uuid --role/string - - /** - Gets the profile of the user with the given ID. - - If no user ID is given, the profile of the current user is returned. - Returns null, if no user with the given ID exists. - */ - get-profile --user-id/Uuid?=null -> Map? - - /** - Updates the profile of the current user. - */ - // TODO(florian): add support for changing the email. - update-profile --name/string - -with-server server-config/ServerConfig --cli/Cli [block]: - network := net.open - server/ArtemisServerCli? := null - try: - server = ArtemisServerCli network server-config --cli=cli - block.call server - finally: - if server: server.close - network.close diff --git a/src/cli/artemis_servers/http/base.toit b/src/cli/artemis_servers/http/base.toit deleted file mode 100644 index 9269b61e..00000000 --- a/src/cli/artemis_servers/http/base.toit +++ /dev/null @@ -1,193 +0,0 @@ -// Copyright (C) 2022 Toitware ApS. All rights reserved. - -import certificate-roots -import cli show Cli -import encoding.ubjson -import http -import log -import net -import encoding.json -import encoding.base64 -import uuid show Uuid - -import ..artemis-server -import ...config -import ...device -import ...organization - -import ....shared.server-config -import ....shared.utils as utils -import ....shared.constants show * - -class ArtemisServerCliHttpToit implements ArtemisServerCli: - client_/http.Client? := ? - server-config_/ServerConfigHttp - current-user-id_/Uuid? := null - cli_/Cli - - constructor network/net.Interface .server-config_/ServerConfigHttp --cli/Cli: - client_ = http.Client network - cli_ = cli - - is-closed -> bool: - return client_ == null - - close -> none: - if not client_: return - client_.close - client_ = null - - ensure-authenticated [block]: - if current-user-id_: return - user-id := cli_.config.get "$(CONFIG-SERVER-AUTHS-KEY).$(server-config_.name)" - if user-id: - current-user-id_ = Uuid.parse user-id - return - block.call "Not logged in" - - sign-up --email/string --password/string: - send-request_ COMMAND-SIGN-UP_ { - "email": email, - "password": password, - } - - sign-in --email/string --password/string: - id := send-request_ COMMAND-SIGN-IN_ { - "email": email, - "password": password, - } - current-user-id_ = Uuid.parse id - cli_.config["$(CONFIG-SERVER-AUTHS-KEY).$(server-config_.name)"] = id - cli_.config.write - - sign-in --provider/string --open-browser/bool --cli/Cli: - throw "UNIMPLEMENTED" - - update --email/string? --password/string?: - payload := {:} - if email: payload["email"] = email - if password: payload["password"] = password - send-request_ COMMAND-UPDATE-CURRENT-USER_ payload - - logout: - if not current-user-id_: throw "Not logged in" - current-user-id_ = null - cli_.config.remove "$(CONFIG-SERVER-AUTHS-KEY).$(server-config_.name)" - cli_.config.write - - create-device-in-organization --organization-id/Uuid --device-id/Uuid? -> Device: - map := { - "organization_id": "$organization-id", - } - if device-id: map["alias"] = "$device-id" - - device-info := send-request_ COMMAND-CREATE-DEVICE-IN-ORGANIZATION_ map - return Device - --hardware-id=Uuid.parse device-info["id"] - --id=Uuid.parse device-info["alias"] - --organization-id=Uuid.parse device-info["organization_id"] - - notify-created --hardware-id/Uuid -> none: - send-request_ COMMAND-NOTIFY-ARTEMIS-CREATED_ { - "hardware_id": "$hardware-id", - "data": { "type": "created" }, - } - - get-current-user-id -> Uuid: - return current-user-id_ - - get-organizations -> List: - organizations := send-request_ COMMAND-GET-ORGANIZATIONS_ {:} - return organizations.map: Organization.from-map it - - get-organization id/Uuid -> OrganizationDetailed?: - organization := send-request_ COMMAND-GET-ORGANIZATION-DETAILS_ { - "id": "$id", - } - if organization == null: return null - return OrganizationDetailed.from-map organization - - create-organization name/string -> Organization: - organization := send-request_ COMMAND-CREATE-ORGANIZATION_ { - "name": name, - } - return Organization.from-map organization - - update-organization organization-id/Uuid --name/string -> none: - update := { - "name": name, - } - send-request_ COMMAND-UPDATE-ORGANIZATION_ { - "id": "$organization-id", - "update": update, - } - - get-organization-members id/Uuid -> List: - response := send-request_ COMMAND-GET-ORGANIZATION-MEMBERS_ { - "id": "$id", - } - return response.map: { - "id": Uuid.parse it["id"], - "role": it["role"], - } - - organization-member-add --organization-id/Uuid --user-id/Uuid --role/string: - send-request_ COMMAND-ORGANIZATION-MEMBER-ADD_ { - "organization_id": "$organization-id", - "user_id": "$user-id", - "role": role, - } - - organization-member-remove --organization-id/Uuid --user-id/Uuid: - send-request_ COMMAND-ORGANIZATION-MEMBER-REMOVE_ { - "organization_id": "$organization-id", - "user_id": "$user-id", - } - - organization-member-set-role --organization-id/Uuid --user-id/Uuid --role/string: - send-request_ COMMAND-ORGANIZATION-MEMBER-SET-ROLE_ { - "organization_id": "$organization-id", - "user_id": "$user-id", - "role": role, - } - - get-profile --user-id/Uuid?=null -> Map?: - result := send-request_ COMMAND-GET-PROFILE_ { - "id": user-id ? "$user-id" : null, - } - if not result: return null - result["id"] = Uuid.parse result["id"] - return result - - update-profile --name/string -> none: - send-request_ COMMAND-UPDATE-PROFILE_ { - "name": name, - } - - send-request_ command/int data/Map -> any: - encoded/ByteArray := #[command] + (json.encode data) - headers := http.Headers - if current-user-id_ != null: - headers.add "X-User-Id" "$current-user-id_" - - if server-config_.admin-headers: - server-config_.admin-headers.do: | key value | - headers.add key value - - response := client_.post encoded - --host=server-config_.host - --port=server-config_.port - --path="/" - --headers=headers - - if response.status-code != http.STATUS-OK and response.status-code != http.STATUS-IM-A-TEAPOT: - throw "HTTP error: $response.status-code $response.status-message" - - if (command == COMMAND-DOWNLOAD-SERVICE-IMAGE_) - and response.status-code != http.STATUS-IM-A-TEAPOT: - return utils.read-all response.body - - decoded := json.decode-stream response.body - if response.status-code == http.STATUS-IM-A-TEAPOT: - throw "Broker error: $decoded" - return decoded diff --git a/src/cli/artemis_servers/supabase/supabase.toit b/src/cli/artemis_servers/supabase/supabase.toit deleted file mode 100644 index 3e294b05..00000000 --- a/src/cli/artemis_servers/supabase/supabase.toit +++ /dev/null @@ -1,156 +0,0 @@ -// Copyright (C) 2022 Toitware ApS. All rights reserved. - -import certificate-roots -import cli show Cli -import http -import net -import encoding.json -import supabase -import supabase.filter show equals is-null orr -import uuid show Uuid - -import ..artemis-server -import ...config -import ...device -import ...organization -import ...utils.supabase - -import ....shared.server-config - -TOIT_IO_AUTH_REDIRECT_URL ::= "https://toit.io/auth" - -class ArtemisServerCliSupabase implements ArtemisServerCli: - client_/supabase.Client? := ? - server-config_/ServerConfigSupabase - - constructor network/net.Interface .server-config_/ServerConfigSupabase --cli/Cli: - local-storage := ConfigLocalStorage --auth-key="$(CONFIG-SERVER-AUTHS-KEY).$(server-config_.name)" --cli=cli - client_ = supabase.Client network --server-config=server-config_ --local-storage=local-storage - - is-closed -> bool: - return client_ == null - - close: - if not client_: return - client_.close - client_ = null - - ensure-authenticated [block]: - client_.ensure-authenticated block - - sign-up --email/string --password/string: - client_.auth.sign-up --email=email --password=password - - sign-in --email/string --password/string: - client_.auth.sign-in --email=email --password=password - - sign-in --provider/string --open-browser/bool --cli/Cli: - client_.auth.sign-in - --provider=provider - --open-browser=open-browser - --redirect-url=TOIT_IO_AUTH_REDIRECT_URL - --ui=SupabaseUi cli - - update --email/string? --password/string?: - payload := {:} - if email: payload["email"] = email - if password: payload["password"] = password - client_.auth.update-current-user payload - - logout: - client_.auth.logout - - create-device-in-organization --organization-id/Uuid --device-id/Uuid? -> Device: - payload := { - "organization_id": "$organization-id", - } - - if device-id: payload["alias"] = "$device-id" - - inserted := client_.rest.insert "devices" payload - return Device - --hardware-id=Uuid.parse inserted["id"] - --id=Uuid.parse inserted["alias"] - --organization-id=Uuid.parse inserted["organization_id"] - - notify-created --hardware-id/Uuid -> none: - client_.rest.insert "events" --no-return-inserted { - "device_id": "$hardware-id", - "data": { "type": "created" } - } - - get-current-user-id -> Uuid: - return Uuid.parse client_.auth.get-current-user["id"] - - get-organizations -> List: - // TODO(florian): we only need the id and the name. - organizations := client_.rest.select "organizations" - return organizations.map: Organization.from-map it - - get-organization id/Uuid -> OrganizationDetailed?: - organizations := client_.rest.select "organizations" --filters=[ - equals "id" "$id" - ] - if organizations.is-empty: return null - return OrganizationDetailed.from-map organizations[0] - - create-organization name/string -> Organization: - inserted := client_.rest.insert "organizations" { "name": name } - return Organization.from-map inserted - - update-organization organization-id/Uuid --name/string -> none: - update := { - "name": name, - } - client_.rest.update "organizations" update --filters=[ - equals "id" "$organization-id" - ] - - get-organization-members organization-id/Uuid -> List: - members := client_.rest.select "roles" --filters=[ - equals "organization_id" "$organization-id" - ] - return members.map: { - "id": Uuid.parse it["user_id"], - "role": it["role"], - } - - organization-member-add --organization-id/Uuid --user-id/Uuid --role/string: - client_.rest.insert "roles" { - "organization_id": "$organization-id", - "user_id": "$user-id", - "role": role, - } - - organization-member-remove --organization-id/Uuid --user-id/Uuid: - client_.rest.delete "roles" --filters=[ - equals "organization_id" "$organization-id", - equals "user_id" "$user-id", - ] - - organization-member-set-role --organization-id/Uuid --user-id/Uuid --role/string: - client_.rest.update "roles" --filters=[ - equals "organization_id" "$organization-id", - equals "user_id" "$user-id", - ] { "role": role } - - get-profile --user-id/Uuid?=null -> Map?: - if not user-id: - // TODO(florian): we should have the current user cached. - current-user := client_.auth.get-current-user - user-id = Uuid.parse current-user["id"] - response := client_.rest.select "profiles_with_email" --filters=[ - equals "id" "$user-id", - ] - if response.is-empty: return null - result := response[0] - result["id"] = Uuid.parse result["id"] - return result - - update-profile --name/string -> none: - // TODO(florian): we should have the current user cached. - current-user := client_.auth.get-current-user - user-id := Uuid.parse current-user["id"] - client_.rest.update "profiles" { "name": name } --filters=[ - equals "id" "$user-id", - ] diff --git a/src/cli/broker.toit b/src/cli/broker.toit index cb89dd79..c51cf7e0 100644 --- a/src/cli/broker.toit +++ b/src/cli/broker.toit @@ -22,6 +22,7 @@ import .utils.patch-build show build-diff-patch build-trivial-patch import ..shared.version import ..shared.utils.patch show Patcher PatchObserver +import .auth show Authenticatable import .brokers.broker import .organization import .event @@ -98,9 +99,11 @@ class Broker: broker-connection_ -> BrokerCli: if not broker-connection__: - broker-connection__ = BrokerCli server-config --cli=cli_ - broker-connection__.ensure-authenticated: | error-message | - cli_.ui.abort "$error-message (broker)." + connection := BrokerCli server-config --cli=cli_ + if connection is Authenticatable: + (connection as Authenticatable).ensure-authenticated: | error-message | + cli_.ui.abort "$error-message (broker)." + broker-connection__ = connection return broker-connection__ short-string-for_ --device-id/Uuid -> string: @@ -109,6 +112,8 @@ class Broker: /** Ensures that the broker is authenticated. + + Has no effect for brokers that don't require authentication. */ ensure-authenticated: broker-connection_ diff --git a/src/cli/brokers/broker.toit b/src/cli/brokers/broker.toit index f92cb0a1..a2c6e523 100644 --- a/src/cli/brokers/broker.toit +++ b/src/cli/brokers/broker.toit @@ -18,8 +18,13 @@ import .http.base /** Responsible for allowing the Artemis CLI to talk to Artemis services on devices. + +User authentication is not part of this interface. Brokers that require + authentication (for example Supabase) implement $Authenticatable in + addition. Use `broker is Authenticatable` to check before triggering + auth flows. */ -interface BrokerCli implements Authenticatable: +interface BrokerCli: // TODO(florian): we probably want to add a `connect` function to this interface. // At the moment we require the connection to be open when artemis receives the // broker. @@ -43,38 +48,6 @@ interface BrokerCli implements Authenticatable: */ id -> string - /** - Ensures that the user is authenticated. - - If the user is not authenticated, the $block is called. - */ - ensure-authenticated [block] - - /** - Signs the user up with the given $email and $password. - */ - sign-up --email/string --password/string - - /** - Signs the user in with the given $email and $password. - */ - sign-in --email/string --password/string - - /** - Signs the user in using OAuth. - */ - sign-in --provider/string --cli/Cli --open-browser - - /** - Updates the user's email and/or password. - */ - update --email/string? --password/string? - - /** - Signs the user out. - */ - logout - /** Updates the goal state of the device with the given $device-id. @@ -284,10 +257,31 @@ Not all brokers support administrative operations. For example, a file-based broker (like a GitHub Pages broker) would not support user or organization management. +Admin brokers always require user authentication; the interface + therefore extends $Authenticatable. + Use `broker is AdminBrokerCli` to check whether a broker supports these operations. */ -interface AdminBrokerCli extends BrokerCli: +interface AdminBrokerCli extends BrokerCli implements Authenticatable: + /** See $Authenticatable.ensure-authenticated. */ + ensure-authenticated [block] + + /** See $Authenticatable.sign-up. */ + sign-up --email/string --password/string + + /** See $(Authenticatable.sign-in --email --password). */ + sign-in --email/string --password/string + + /** See $(Authenticatable.sign-in --provider --cli --open-browser). */ + sign-in --provider/string --cli/Cli --open-browser + + /** See $Authenticatable.update. */ + update --email/string? --password/string? + + /** See $Authenticatable.logout. */ + logout + /** Returns the user-id of the authenticated user. */ get-current-user-id -> Uuid diff --git a/src/cli/brokers/http/base.toit b/src/cli/brokers/http/base.toit index ec4a1b2b..426684ea 100644 --- a/src/cli/brokers/http/base.toit +++ b/src/cli/brokers/http/base.toit @@ -44,30 +44,6 @@ class BrokerCliHttp implements BrokerCli: is-closed -> bool: return network_ == null - ensure-authenticated [block]: - // For simplicity do nothing. - // This way we can use the same tests for all brokers. - - sign-up --email/string --password/string: - // For simplicity do nothing. - // This way we can use the same tests for all brokers. - - sign-in --email/string --password/string: - // For simplicity do nothing. - // This way we can use the same tests for all brokers. - - sign-in --provider/string --cli/Cli --open-browser/bool: - // For simplicity do nothing. - // This way we can use the same tests for all brokers. - - update --email/string? --password/string?: - // For simplicity do nothing. - // This way we can use the same tests for all brokers. - - logout: - // For simplicity do nothing. - // This way we can use the same tests for all brokers. - send-request_ command/int data/any -> any: if is-closed: throw "CLOSED" encoded/ByteArray := ? diff --git a/src/cli/brokers/supabase/supabase.toit b/src/cli/brokers/supabase/supabase.toit index bdf1e87c..16b7fc17 100644 --- a/src/cli/brokers/supabase/supabase.toit +++ b/src/cli/brokers/supabase/supabase.toit @@ -9,6 +9,7 @@ import uuid show Uuid import ..broker show AdminBrokerCli import ..http.base +import ...auth show Authenticatable import ...config import ...device import ...organization @@ -54,7 +55,7 @@ create-broker-cli-supabase-http server-config/ServerConfigSupabase --cli/Cli -> return BrokerCliSupabase --id=id supabase-client http-config -class BrokerCliSupabase extends BrokerCliHttp: +class BrokerCliSupabase extends BrokerCliHttp implements Authenticatable: supabase-client_/supabase.Client? := null constructor --id/string .supabase-client_ http-config/ServerConfigHttp: diff --git a/src/cli/cmds/auth.toit b/src/cli/cmds/auth.toit index 456395dc..a7329099 100644 --- a/src/cli/cmds/auth.toit +++ b/src/cli/cmds/auth.toit @@ -159,6 +159,13 @@ update invocation/Invocation: if exception: ui.abort exception +/** +Calls $block with the broker's $Authenticatable for the configured server. + +If the broker does not require authentication (for example a plain HTTP + broker), emits an informational message and returns without invoking + $block. This keeps tests and scripts portable across broker kinds. +*/ with-authenticatable invocation/Invocation [block]: server := invocation["server"] @@ -170,7 +177,10 @@ with-authenticatable invocation/Invocation [block]: else: server-config = get-server-from-config --cli=cli --key=CONFIG-BROKER-DEFAULT-KEY with-broker --cli=cli server-config: | broker/BrokerCli | - block.call server-config.name broker + if broker is not Authenticatable: + cli.ui.emit --info "Broker '$server-config.name' does not require authentication." + return + block.call server-config.name (broker as Authenticatable) sign-in invocation/Invocation: with-authenticatable invocation: | name/string authenticatable/Authenticatable | diff --git a/src/cli/cmds/fleet.toit b/src/cli/cmds/fleet.toit index 714d1d6a..a95aa81f 100644 --- a/src/cli/cmds/fleet.toit +++ b/src/cli/cmds/fleet.toit @@ -11,6 +11,7 @@ import .device show EXTRACT-FORMATS-COMMAND-HELP import .auth as auth-cmd +import ..auth show Authenticatable import .serial show PARTITION-OPTION import .utils_ import ..broker show Broker @@ -675,9 +676,12 @@ login invocation/Invocation: with-pod-fleet invocation: | fleet/Fleet | broker := fleet.broker broker-name := broker.server-config.name + broker-cli := BrokerCli broker.server-config --cli=cli + if broker-cli is not Authenticatable: + ui.emit --info "Broker '$broker-name' does not require authentication." + return ui.emit --info "Logging in to broker '$broker-name'." - broker-authenticatable := BrokerCli broker.server-config --cli=cli - auth-cmd.sign-in invocation --name=broker-name --authenticatable=broker-authenticatable + auth-cmd.sign-in invocation --name=broker-name --authenticatable=(broker-cli as Authenticatable) add-devices invocation/Invocation: cli := invocation.cli diff --git a/src/cli/cmds/org.toit b/src/cli/cmds/org.toit index 90aba794..79e69fd9 100644 --- a/src/cli/cmds/org.toit +++ b/src/cli/cmds/org.toit @@ -308,10 +308,10 @@ default-org invocation/Invocation -> none: server-config/ServerConfig := get-server-from-config --key=CONFIG-BROKER-DEFAULT-KEY --cli=cli with-broker server-config --cli=cli: | broker/BrokerCli | - broker.ensure-authenticated: | error-message | - ui.abort "$error-message (broker)." admin := broker-as-admin-or-null broker if admin: + admin.ensure-authenticated: | error-message | + ui.abort "$error-message (broker)." org/OrganizationDetailed? := null exception := catch: org = admin.get-organization org-id if exception or not org: diff --git a/src/cli/cmds/utils_.toit b/src/cli/cmds/utils_.toit index 7357892a..e4321fe9 100644 --- a/src/cli/cmds/utils_.toit +++ b/src/cli/cmds/utils_.toit @@ -84,11 +84,12 @@ with-admin-broker invocation/Invocation --capability/string [block]: ui := cli.ui server-config := get-server-from-config --key=CONFIG-BROKER-DEFAULT-KEY --cli=cli with-broker server-config --cli=cli: | broker/BrokerCli | - broker.ensure-authenticated: | error-message | - ui.abort "$error-message (broker)." if broker is not AdminBrokerCli: ui.abort "The configured broker does not support $capability." - block.call (broker as AdminBrokerCli) + admin := broker as AdminBrokerCli + admin.ensure-authenticated: | error-message | + ui.abort "$error-message (broker)." + block.call admin /** Returns $broker as $AdminBrokerCli, or null if it does not implement diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index c7f85824..138d8ae4 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -43,6 +43,7 @@ set(TEST_PREFIX "") include(fail.cmake OPTIONAL) set (SUPABASE_ARTEMIS_TESTS + "/tests/admin-broker-test.toit" "/tests/supabase-policies-test.toit" ) diff --git a/tests/artemis-server-test.toit b/tests/admin-broker-test.toit similarity index 50% rename from tests/artemis-server-test.toit rename to tests/admin-broker-test.toit index 50da7d0f..fb5f0f76 100644 --- a/tests/artemis-server-test.toit +++ b/tests/admin-broker-test.toit @@ -1,80 +1,56 @@ -// Copyright (C) 2022 Toitware ApS. All rights reserved. +// Copyright (C) 2026 Toit contributors. -// ARTEMIS_TEST_FLAGS: ARTEMIS +// Exercises the AdminBrokerCli interface against a Supabase broker that +// provides admin operations (org/profile/member management). Replaces +// the historical artemis-server-test, which exercised the old standalone +// ArtemisServerCli interface that has since been folded into the broker. import cli show Cli import expect show * -import host.directory -import log -import net import uuid show Uuid import .artemis-server import .utils -import artemis.cli.artemis-servers.artemis-server show ArtemisServerCli -import artemis.shared.server-config show ServerConfig -import artemis.cli.auth as cli-auth +import artemis.cli.brokers.broker show BrokerCli AdminBrokerCli with-broker main args: - server-type := ? - if args.is-empty: - server-type = "http" - else if args[0] == "--http-server": - server-type = "http" - else if args[0] == "--supabase-server": - server-type = "supabase" - else: - throw "Unknown server server type: $args[0]" - with-artemis-server --args=args --type=server-type: | artemis-server/TestArtemisServer | - run-test artemis-server --authenticate=: | server/ArtemisServerCli | - server.sign-in + with-artemis-server --args=args --type="supabase": | artemis-server/TestArtemisServer | + run-test artemis-server --authenticate=: | admin/AdminBrokerCli | + admin.sign-in --email=TEST-EXAMPLE-COM-EMAIL --password=TEST-EXAMPLE-COM-PASSWORD run-test artemis-server/TestArtemisServer [--authenticate]: server-config := artemis-server.server-config - backdoor := artemis-server.backdoor with-tmp-config-cli: | cli/Cli | - network := net.open - server-cli := ArtemisServerCli network server-config --cli=cli - authenticate.call server-cli - hardware-id := test-create-device-in-organization server-cli backdoor - test-notify-created server-cli backdoor --hardware-id=hardware-id - - test-organizations server-cli backdoor - test-profile server-cli backdoor - -test-create-device-in-organization server-cli/ArtemisServerCli backdoor/ArtemisServerBackdoor -> Uuid: + with-broker server-config --cli=cli: | broker/BrokerCli | + if broker is not AdminBrokerCli: + throw "Test broker does not support admin operations." + admin := broker as AdminBrokerCli + authenticate.call admin + test-create-device-in-organization admin + test-organizations admin + test-profile admin + +test-create-device-in-organization admin/AdminBrokerCli -> Uuid: // Test without and with alias. - device1 := server-cli.create-device-in-organization + device1 := admin.create-device-in-organization --device-id=null --organization-id=TEST-ORGANIZATION-UUID - hardware-id1 := device1.hardware-id - data := backdoor.fetch-device-information --hardware-id=hardware-id1 - expect-equals hardware-id1 data[0] - expect-equals TEST-ORGANIZATION-UUID data[1] + expect-equals TEST-ORGANIZATION-UUID device1.organization-id alias-id := random-uuid - device2 := server-cli.create-device-in-organization + device2 := admin.create-device-in-organization --device-id=alias-id --organization-id=TEST-ORGANIZATION-UUID - sleep --ms=200 - hardware-id2 := device2.hardware-id - data = backdoor.fetch-device-information --hardware-id=hardware-id2 - expect-equals hardware-id2 data[0] - expect-equals TEST-ORGANIZATION-UUID data[1] - expect-equals alias-id data[2] - - return hardware-id2 + expect-equals TEST-ORGANIZATION-UUID device2.organization-id + expect-equals alias-id device2.id -test-notify-created server-cli/ArtemisServerCli backdoor/ArtemisServerBackdoor --hardware-id/Uuid: - expect-not (backdoor.has-event --hardware-id=hardware-id --type="created") - server-cli.notify-created --hardware-id=hardware-id - expect (backdoor.has-event --hardware-id=hardware-id --type="created") + return device2.hardware-id -test-organizations server-cli/ArtemisServerCli backdoor/ArtemisServerBackdoor: - original-orgs := server-cli.get-organizations +test-organizations admin/AdminBrokerCli: + original-orgs := admin.get-organizations // For now we can't be sure that there aren't other organizations from // previous runs of the test. @@ -82,40 +58,40 @@ test-organizations server-cli/ArtemisServerCli backdoor/ArtemisServerBackdoor: expect original-orgs.size >= 1 // The prefilled organization. expect (original-orgs.any: it.id == TEST-ORGANIZATION-UUID) - org := server-cli.create-organization "Testy" + org := admin.create-organization "Testy" expect-equals "Testy" org.name expect-not-equals "" org.id expect-not (original-orgs.any: it.id == org.id) - new-orgs := server-cli.get-organizations + new-orgs := admin.get-organizations expect-equals (original-orgs.size + 1) new-orgs.size original-orgs.do: | old-org | expect (new-orgs.any: it.id == old-org.id) expect (new-orgs.any: it.id == org.id) - detailed := server-cli.get-organization org.id + detailed := admin.get-organization org.id expect-equals org.id detailed.id expect-equals org.name detailed.name expect (detailed.created-at < Time.now) - non-existent := server-cli.get-organization NON-EXISTENT-UUID + non-existent := admin.get-organization NON-EXISTENT-UUID expect-null non-existent // Test member functions. current-user-id := TEST-EXAMPLE-COM-UUID demo-user-id := DEMO-EXAMPLE-COM-UUID - members := server-cli.get-organization-members org.id + members := admin.get-organization-members org.id expect-equals 1 members.size expect-equals current-user-id members[0]["id"] expect-equals "admin" members[0]["role"] // Add a new member. - server-cli.organization-member-add + admin.organization-member-add --organization-id=org.id --user-id=demo-user-id --role="member" - members = server-cli.get-organization-members org.id + members = admin.get-organization-members org.id expect-equals 2 members.size expect members[0]["id"] != members[1]["id"] members.do: | member | @@ -126,11 +102,11 @@ test-organizations server-cli/ArtemisServerCli backdoor/ArtemisServerBackdoor: expect-equals "member" member["role"] // Update the role of the new member. - server-cli.organization-member-set-role + admin.organization-member-set-role --organization-id=org.id --user-id=demo-user-id --role="admin" - members = server-cli.get-organization-members org.id + members = admin.get-organization-members org.id expect-equals 2 members.size expect members[0]["id"] != members[1]["id"] members.do: | member | @@ -139,21 +115,21 @@ test-organizations server-cli/ArtemisServerCli backdoor/ArtemisServerBackdoor: expect-equals "admin" member["role"] // Remove the new member. - server-cli.organization-member-remove + admin.organization-member-remove --organization-id=org.id --user-id=demo-user-id - members = server-cli.get-organization-members org.id + members = admin.get-organization-members org.id expect-equals 1 members.size expect-equals current-user-id members[0]["id"] expect-equals "admin" members[0]["role"] // Add the new member with admin role. - server-cli.organization-member-add + admin.organization-member-add --organization-id=org.id --user-id=demo-user-id --role="admin" - members = server-cli.get-organization-members org.id + members = admin.get-organization-members org.id expect-equals 2 members.size expect members[0]["id"] != members[1]["id"] members.do: | member | @@ -162,32 +138,32 @@ test-organizations server-cli/ArtemisServerCli backdoor/ArtemisServerBackdoor: expect-equals "admin" member["role"] // Keep the demo user in the same organization as the test user, - // so we can read the user's profile in 'test_profile' + // so we can read the user's profile in 'test-profile'. -test-profile server-cli/ArtemisServerCli backdoor/ArtemisServerBackdoor: - profile := server-cli.get-profile +test-profile admin/AdminBrokerCli: + profile := admin.get-profile - profile = server-cli.get-profile + profile = admin.get-profile expect-equals "Test User" profile["name"] id := profile["id"] - server-cli.update-profile --name="Test User updated" - profile = server-cli.get-profile + admin.update-profile --name="Test User updated" + profile = admin.get-profile expect-equals "Test User updated" profile["name"] - profile2 := server-cli.get-profile --user-id=id + profile2 := admin.get-profile --user-id=id expect-equals profile["id"] profile2["id"] expect-equals profile["name"] profile2["name"] expect-equals profile["email"] profile2["email"] // Change it back. // Other tests might need the profile to be in a certain state. - server-cli.update-profile --name="Test User" + admin.update-profile --name="Test User" - profile-non-existent := server-cli.get-profile --user-id=NON-EXISTENT-UUID + profile-non-existent := admin.get-profile --user-id=NON-EXISTENT-UUID expect-null profile-non-existent // The following test requires that we have added the demo user // and test user into the same organization. - profile-demo := server-cli.get-profile --user-id=DEMO-EXAMPLE-COM-UUID + profile-demo := admin.get-profile --user-id=DEMO-EXAMPLE-COM-UUID expect-equals DEMO-EXAMPLE-COM-NAME profile-demo["name"] diff --git a/tests/broker-test.toit b/tests/broker-test.toit index 2b9b6d63..d1f73b7a 100644 --- a/tests/broker-test.toit +++ b/tests/broker-test.toit @@ -8,6 +8,7 @@ import log import monitor import net import artemis.cli.brokers.broker +import artemis.cli.auth show Authenticatable import artemis.cli.device show DeviceDetailed import artemis.service.device show Device import artemis.cli.event show Event @@ -56,9 +57,11 @@ run-test // We are going to reuse the cli for all tests (and only authenticate once). // However, we will need multiple services. test-broker.with-cli: | broker-cli/broker.BrokerCli | - // Make sure we are authenticated. - broker-cli.ensure-authenticated: - broker-cli.sign-in --email=TEST-EXAMPLE-COM-EMAIL --password=TEST-EXAMPLE-COM-PASSWORD + // Make sure we are authenticated. HTTP test brokers don't require auth. + if broker-cli is Authenticatable: + auth := broker-cli as Authenticatable + auth.ensure-authenticated: + auth.sign-in --email=TEST-EXAMPLE-COM-EMAIL --password=TEST-EXAMPLE-COM-PASSWORD if broker-name == "supabase-local-artemis": // Make sure the device is in the database. diff --git a/tests/pod-registry-test.toit b/tests/pod-registry-test.toit index 6038a9a8..cb24ec07 100644 --- a/tests/pod-registry-test.toit +++ b/tests/pod-registry-test.toit @@ -7,6 +7,7 @@ import expect show * import log import net import artemis.cli.brokers.broker +import artemis.cli.auth show Authenticatable import artemis.cli.pod-registry show * import artemis.service.brokers.broker @@ -23,9 +24,11 @@ run-test test-broker/TestBroker: test-broker.with-cli: | broker-cli/broker.BrokerCli | - // Make sure we are authenticated. - broker-cli.ensure-authenticated: - broker-cli.sign-in --email=TEST-EXAMPLE-COM-EMAIL --password=TEST-EXAMPLE-COM-PASSWORD + // Make sure we are authenticated. HTTP test brokers don't require auth. + if broker-cli is Authenticatable: + auth := broker-cli as Authenticatable + auth.ensure-authenticated: + auth.sign-in --email=TEST-EXAMPLE-COM-EMAIL --password=TEST-EXAMPLE-COM-PASSWORD test-pod-registry --test-broker=test-broker broker-cli test-pods --test-broker=test-broker broker-cli From 643535565317585532a4d49bb2e0dc79d9e03f0f Mon Sep 17 00:00:00 2001 From: Florian Loitsch Date: Sat, 2 May 2026 16:44:06 +0200 Subject: [PATCH 5/7] Extract FirmwarePatchUploader into its own module The patch-upload helpers (upload-trivial-patches, diff-and-upload, the static id and compute-applied-hash helpers) are used both when uploading a pod (so devices can pull trivial patches) and when rolling out an update (so devices can pull diff patches against their current firmware). Moving them out of broker.toit into a small dedicated module prepares for the upcoming PodStore extraction, where pod uploads need to trigger trivial patch uploads without dragging the rest of the broker god-object along. The new FirmwarePatchUploader is bound to a (broker, organization, server-config) triple at construction time and is reused via a lazy accessor on the Broker wrapper. No behavior change: callers still hit the broker's firmware bucket with the same patch IDs and cache keys. --- src/cli/broker.toit | 136 +++------------------------- src/cli/firmware-patches.toit | 165 ++++++++++++++++++++++++++++++++++ 2 files changed, 178 insertions(+), 123 deletions(-) create mode 100644 src/cli/firmware-patches.toit diff --git a/src/cli/broker.toit b/src/cli/broker.toit index c51cf7e0..9e9f4a37 100644 --- a/src/cli/broker.toit +++ b/src/cli/broker.toit @@ -4,11 +4,9 @@ import cli show Cli FileStore DirectoryStore import crypto.sha256 import host.file import host.os -import io import net import uuid show Uuid -import encoding.base64 import encoding.ubjson import .cache @@ -18,12 +16,11 @@ import .pod import .pod-specification import .utils -import .utils.patch-build show build-diff-patch build-trivial-patch import ..shared.version -import ..shared.utils.patch show Patcher PatchObserver import .auth show Authenticatable import .brokers.broker +import .firmware-patches show FirmwarePatchUploader import .organization import .event import .firmware @@ -181,7 +178,7 @@ class Broker: Also uploads the trivial patches. */ upload --pod/Pod --tags/List --force-tags/bool -> UploadResult: - upload-trivial-patches_ --pod=pod + patch-uploader_.upload-trivial-patches --pod=pod pod.split: | manifest/Map parts/Map | parts.do: | id/string contents/ByteArray | @@ -240,108 +237,16 @@ class Broker: --tags=sorted-uploaded-tags --tag-errors=tag-errors - upload-trivial-patches_ --pod/Pod -> none: - firmware-contents := FirmwareContents.from-envelope pod.envelope-path --cli=cli_ - upload_ --firmware-contents=firmware-contents + patch-uploader__/FirmwarePatchUploader? := null - upload_ --firmware-contents/FirmwareContents: - firmware-contents.trivial-patches.do: - upload-patch_ it - - /** - Uploads the given $patch to the server under the given $organization-id. - */ - upload-patch_ patch/FirmwarePatch: - diff-and-upload_ patch - - /** - Computes patches and uploads them to the broker. - */ - diff-and-upload_ patch/FirmwarePatch -> none: - // Unless it is already cached, always create/upload the trivial one. - trivial-id := id_ --to=patch.to_ - cache-key := cache-key-patch - --broker-config=server-config - --organization-id=organization-id - --patch-id=trivial-id - cli_.cache.get cache-key: | store/FileStore | - trivial := build-trivial-patch patch.bits_ - broker-connection_.upload-firmware trivial + patch-uploader_ -> FirmwarePatchUploader: + if not patch-uploader__: + patch-uploader__ = FirmwarePatchUploader + --broker-connection=broker-connection_ + --server-config=server-config --organization-id=organization-id - --firmware-id=trivial-id - store.save-via-writer: | writer/io.Writer | - trivial.do: writer.write it - - if not patch.from_: return - - // Attempt to fetch the old trivial patch and use it to construct - // the old bits so we can compute a diff from them. - old-id := id_ --to=patch.from_ - cache-key = cache-key-patch - --broker-config=server-config - --organization-id=organization-id - --patch-id=old-id - trivial-old := cli_.cache.get cache-key: | store/FileStore | - downloaded := null - catch: downloaded = broker-connection_.download-firmware - --organization-id=organization-id - --id=old-id - if not downloaded: - cli_.ui.emit --warning "Failed to download old firmware for patch $old-id -> $trivial-id." - return - store.with-tmp-directory: | tmp-dir | - file.write-contents downloaded --path="$tmp-dir/patch" - // TODO(florian): we don't have the chunk-size when downloading from the broker. - store.move "$tmp-dir/patch" - - bitstream := io.Reader trivial-old - patcher := Patcher bitstream null - patch-writer := PatchWriter - if not patcher.patch patch-writer: return - // Build the old bits and check that we get the correct hash. - old := patch-writer.buffer.bytes - if old.size < patch-writer.size: old += ByteArray (patch-writer.size - old.size) - sha := sha256.Sha256 - sha.add old - if patch.from_ != sha.get: return - - diff-id := id_ --from=patch.from_ --to=patch.to_ - cache-key = cache-key-patch - --broker-config=server-config - --organization-id=organization-id - --patch-id=diff-id - cli_.cache.get cache-key: | store/FileStore | - // Build the diff and verify that we can apply it and get the - // correct hash out before uploading it. - diff := build-diff-patch old patch.bits_ - if patch.to_ != (compute-applied-hash_ diff old): return - diff-size-bytes := diff.reduce --initial=0: | size chunk | size + chunk.size - diff-size := diff-size-bytes > 4096 - ? "$((diff-size-bytes + 1023) / 1024) KB" - : "$diff-size-bytes B" - from64 := base64.encode patch.from_ --url-mode - to64 := base64.encode patch.to_ --url-mode - cli_.ui.emit --info "Uploading patch $from64 -> $to64 ($diff-size)." - broker-connection_.upload-firmware diff - --organization-id=organization-id - --firmware-id=diff-id - store.save-via-writer: | writer/io.Writer | - diff.do: writer.write it - - static id_ --from/ByteArray?=null --to/ByteArray -> string: - folder := base64.encode to --url-mode - entry := from ? (base64.encode from --url-mode) : "none" - return "$folder/$entry" - - static compute-applied-hash_ diff/List old/ByteArray -> ByteArray?: - combined := diff.reduce --initial=#[]: | acc chunk | acc + chunk - bitstream := io.Reader combined - patcher := Patcher bitstream old - writer := PatchWriter - if not patcher.patch writer: return null - sha := sha256.Sha256 - sha.add writer.buffer.bytes - return sha.get + --cli=cli_ + return patch-uploader__ is-cached --pod-id/Uuid -> bool: manifest-key := cache-key-pod-manifest @@ -555,9 +460,7 @@ class Broker: FirmwareContents.from-envelope diff-base.envelope-path --cli=cli_ base-firmwares.do: | contents/FirmwareContents | - trivial-patches := extract-trivial-patches_ contents - trivial-patches.do: | _ patch/FirmwarePatch | - upload-patch_ patch + patch-uploader_.upload-trivial-patches --firmware-contents=contents update-bulk_ --devices=devices @@ -565,19 +468,6 @@ class Broker: --base-firmwares=base-firmwares --warn-only-trivial=warn-only-trivial /** - Extracts the trivial patches from the given $firmware-contents. - - Returns a mapping from patch-id (as used when diffing to the part) and - the patch itself. - */ - extract-trivial-patches_ firmware-contents/FirmwareContents -> Map: - result := {:} - firmware-contents.trivial-patches.do: | patch/FirmwarePatch | - patch-id := id_ --to=patch.to_ - result[patch-id] = patch - return result - - /** Update the given $devices. The lists $devices and $pods must have the same size. @@ -633,7 +523,7 @@ class Broker: --base-firmwares/List --warn-only-trivial/bool=true: device-id := device.id - upload_ --firmware-contents=unconfigured-contents + patch-uploader_.upload-trivial-patches --firmware-contents=unconfigured-contents known-encoded-firmwares := {} [ @@ -689,7 +579,7 @@ class Broker: --cli=cli_ upgrade-from.do: | old-firmware-contents/FirmwareContents | patches := upgrade-to.contents.patches old-firmware-contents - patches.do: diff-and-upload_ it + patches.do: patch-uploader_.diff-and-upload it // Build the updated goal and return it. sdk := get-sdk pod.sdk-version --cli=cli_ diff --git a/src/cli/firmware-patches.toit b/src/cli/firmware-patches.toit new file mode 100644 index 00000000..90ebb328 --- /dev/null +++ b/src/cli/firmware-patches.toit @@ -0,0 +1,165 @@ +// Copyright (C) 2026 Toit contributors. + +import cli show Cli FileStore +import crypto.sha256 +import encoding.base64 +import host.file +import io +import uuid show Uuid + +import .brokers.broker show BrokerCli +import .cache +import .firmware show FirmwareContents FirmwarePatch PatchWriter +import .pod show Pod +import .utils.patch-build show build-diff-patch build-trivial-patch +import ..shared.server-config +import ..shared.utils.patch show Patcher + +/** +Uploads firmware patches (trivial and diff) to a broker's firmware + storage on behalf of pod uploads and device roll-outs. + +The uploader is bound to a specific (broker, organization) pair. Patch + IDs are derived from the from/to firmware checksums; uploads are + cached locally so repeated runs are cheap. +*/ +class FirmwarePatchUploader: + broker-connection_/BrokerCli + server-config_/ServerConfig + organization-id_/Uuid + cli_/Cli + + constructor + --broker-connection/BrokerCli + --server-config/ServerConfig + --organization-id/Uuid + --cli/Cli: + broker-connection_ = broker-connection + server-config_ = server-config + organization-id_ = organization-id + cli_ = cli + + /** + Uploads the trivial patches contained in $pod's firmware envelope. + */ + upload-trivial-patches --pod/Pod -> none: + firmware-contents := FirmwareContents.from-envelope pod.envelope-path --cli=cli_ + upload-trivial-patches --firmware-contents=firmware-contents + + /** + Uploads the trivial patches contained in $firmware-contents. + */ + upload-trivial-patches --firmware-contents/FirmwareContents -> none: + firmware-contents.trivial-patches.do: diff-and-upload it + + /** + Returns the trivial patches in $firmware-contents keyed by patch id. + */ + static extract-trivial-patches firmware-contents/FirmwareContents -> Map: + result := {:} + firmware-contents.trivial-patches.do: | patch/FirmwarePatch | + result[(id --to=patch.to_)] = patch + return result + + /** + Computes patches for the given $patch and uploads them to the broker. + + Always uploads the trivial patch (unless cached). If $patch carries + a "from" hash, additionally attempts to compute and upload a diff + patch from the previous trivial patch. + */ + diff-and-upload patch/FirmwarePatch -> none: + // Unless it is already cached, always create/upload the trivial one. + trivial-id := id --to=patch.to_ + cache-key := cache-key-patch + --broker-config=server-config_ + --organization-id=organization-id_ + --patch-id=trivial-id + cli_.cache.get cache-key: | store/FileStore | + trivial := build-trivial-patch patch.bits_ + broker-connection_.upload-firmware trivial + --organization-id=organization-id_ + --firmware-id=trivial-id + store.save-via-writer: | writer/io.Writer | + trivial.do: writer.write it + + if not patch.from_: return + + // Attempt to fetch the old trivial patch and use it to construct + // the old bits so we can compute a diff from them. + old-id := id --to=patch.from_ + cache-key = cache-key-patch + --broker-config=server-config_ + --organization-id=organization-id_ + --patch-id=old-id + trivial-old := cli_.cache.get cache-key: | store/FileStore | + downloaded := null + catch: downloaded = broker-connection_.download-firmware + --organization-id=organization-id_ + --id=old-id + if not downloaded: + cli_.ui.emit --warning "Failed to download old firmware for patch $old-id -> $trivial-id." + return + store.with-tmp-directory: | tmp-dir | + file.write-contents downloaded --path="$tmp-dir/patch" + // TODO(florian): we don't have the chunk-size when downloading from the broker. + store.move "$tmp-dir/patch" + + bitstream := io.Reader trivial-old + patcher := Patcher bitstream null + patch-writer := PatchWriter + if not patcher.patch patch-writer: return + // Build the old bits and check that we get the correct hash. + old := patch-writer.buffer.bytes + if old.size < patch-writer.size: old += ByteArray (patch-writer.size - old.size) + sha := sha256.Sha256 + sha.add old + if patch.from_ != sha.get: return + + diff-id := id --from=patch.from_ --to=patch.to_ + cache-key = cache-key-patch + --broker-config=server-config_ + --organization-id=organization-id_ + --patch-id=diff-id + cli_.cache.get cache-key: | store/FileStore | + // Build the diff and verify that we can apply it and get the + // correct hash out before uploading it. + diff := build-diff-patch old patch.bits_ + if patch.to_ != (compute-applied-hash diff old): return + diff-size-bytes := diff.reduce --initial=0: | size chunk | size + chunk.size + diff-size := diff-size-bytes > 4096 + ? "$((diff-size-bytes + 1023) / 1024) KB" + : "$diff-size-bytes B" + from64 := base64.encode patch.from_ --url-mode + to64 := base64.encode patch.to_ --url-mode + cli_.ui.emit --info "Uploading patch $from64 -> $to64 ($diff-size)." + broker-connection_.upload-firmware diff + --organization-id=organization-id_ + --firmware-id=diff-id + store.save-via-writer: | writer/io.Writer | + diff.do: writer.write it + + /** + Computes the patch identifier for a (from, to) firmware checksum pair. + + If $from is null, returns the trivial-patch identifier for $to. + */ + static id --from/ByteArray?=null --to/ByteArray -> string: + folder := base64.encode to --url-mode + entry := from ? (base64.encode from --url-mode) : "none" + return "$folder/$entry" + + /** + Applies $diff to $old and returns the SHA-256 of the result. + + Returns null if the diff doesn't apply cleanly. + */ + static compute-applied-hash diff/List old/ByteArray -> ByteArray?: + combined := diff.reduce --initial=#[]: | acc chunk | acc + chunk + bitstream := io.Reader combined + patcher := Patcher bitstream old + writer := PatchWriter + if not patcher.patch writer: return null + sha := sha256.Sha256 + sha.add writer.buffer.bytes + return sha.get From 89c13542c6355d80f5502e8c3bf6be5c9a71d80b Mon Sep 17 00:00:00 2001 From: Florian Loitsch Date: Sat, 2 May 2026 16:59:17 +0200 Subject: [PATCH 6/7] Extract PodStore interface and BrokerPodStore implementation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pod-registry operations (descriptions, entries, tags, parts, manifests) were 14 of the ~20 methods on the Broker wrapper class. They form a self-contained concern: a store of pods scoped to a (fleet, organization) pair. Pulling them out lets future implementations (filesystem-backed for git-managed fleets, HTTP-only for public release pages) plug in without touching the device-broker code. - New interface PodStore in src/cli/pod-store.toit. Moves UploadResult here too, and renames PodBroker -> PodInfo (the old name was misleading after extraction; it's a single pod's metadata, not a broker-of-pods). - New class BrokerPodStore in src/cli/brokers/broker-pod-store.toit implements PodStore by delegating to a BrokerCli plus a FirmwarePatchUploader (for the side-effect of pre-uploading trivial firmware patches when a pod is uploaded). - Broker wrapper exposes a lazy `pod-store -> BrokerPodStore` accessor and drops the 14 pod methods plus their UploadResult/PodBroker types. Net diff in broker.toit: -307 lines. - Fleet's pod-related methods (upload, download, list-pods, delete, add-tags, remove-tags, pod, get-pod-id, pod-exists) and the inline pod-entry/description lookup in `status` now go through broker.pod-store instead of broker. The fleet-id and organization-id parameters that used to thread through every pod-registry call are now constructor parameters of BrokerPodStore. This matches the pod-registry SQL schema, where (fleet_id, name) is a hard uniqueness constraint — not incidental plumbing. No behavior change: cache keys, broker requests, and CLI output are identical. --- src/cli/broker.toit | 329 ++------------------------ src/cli/brokers/broker-pod-store.toit | 319 +++++++++++++++++++++++++ src/cli/fleet.toit | 31 +-- src/cli/pod-store.toit | 154 ++++++++++++ 4 files changed, 503 insertions(+), 330 deletions(-) create mode 100644 src/cli/brokers/broker-pod-store.toit create mode 100644 src/cli/pod-store.toit diff --git a/src/cli/broker.toit b/src/cli/broker.toit index 9e9f4a37..492de547 100644 --- a/src/cli/broker.toit +++ b/src/cli/broker.toit @@ -1,14 +1,12 @@ // Copyright (C) 2024 Toitware ApS. All rights reserved. -import cli show Cli FileStore DirectoryStore +import cli show Cli DirectoryStore import crypto.sha256 import host.file import host.os import net import uuid show Uuid -import encoding.ubjson - import .cache import .config import .device @@ -20,45 +18,15 @@ import ..shared.version import .auth show Authenticatable import .brokers.broker +import .brokers.broker-pod-store show BrokerPodStore import .firmware-patches show FirmwarePatchUploader import .organization import .event import .firmware -import .pod-registry import .program import .sdk import .server-config -class UploadResult: - fleet-id/Uuid - id/Uuid - name/string - revision/int - tags/List - tag-errors/List - - constructor --.fleet-id --.id --.name --.revision --.tags --.tag-errors: - - to-json -> Map: - result := { - "fleet-id": "$fleet-id", - "id": "$id", - "name": name, - "revision": revision, - "tags": tags, - } - if not tag-errors.is-empty: - result["tag-errors"] = tag-errors - return result - -class PodBroker: - id/Uuid - name/string? - revision/int? - tags/List? - - constructor --.id --.name --.revision --.tags: - /** Manages devices that have an Artemis service running on them. */ @@ -168,75 +136,6 @@ class Broker: network_.close network_ = null - is-existing-tag-error_ error -> bool: - if error is not string: return false - return error.contains "duplicate key value" or error.contains "already exists" - - /** - Uploads the given $pod to the broker for the given $fleet-id in $organization-id. - - Also uploads the trivial patches. - */ - upload --pod/Pod --tags/List --force-tags/bool -> UploadResult: - patch-uploader_.upload-trivial-patches --pod=pod - - pod.split: | manifest/Map parts/Map | - parts.do: | id/string contents/ByteArray | - // Only upload if we don't have it in our cache. - key := cache-key-pod-parts - --broker-config=server-config - --organization-id=organization-id - --part-id=id - cli_.cache.get-file-path key: | store/FileStore | - broker-connection_.pod-registry-upload-pod-part contents --part-id=id - --organization-id=organization-id - store.save contents - key := cache-key-pod-manifest - --broker-config=server-config - --organization-id=organization-id - --pod-id=pod.id - cli_.cache.get-file-path key: | store/FileStore | - encoded := ubjson.encode manifest - broker-connection_.pod-registry-upload-pod-manifest encoded --pod-id=pod.id - --organization-id=organization-id - store.save encoded - - description-ids := broker-connection_.pod-registry-descriptions - --fleet-id=fleet-id - --organization-id=organization-id - --names=[pod.name] - --create-if-absent - - description-id := (description-ids[0] as PodRegistryDescription).id - - broker-connection_.pod-registry-add - --pod-description-id=description-id - --pod-id=pod.id - - tag-errors := [] - tags.do: | tag/string | - force := force-tags or (tag == "latest") - exception := catch --unwind=(: not is-existing-tag-error_ it): - broker-connection_.pod-registry-tag-set - --pod-description-id=description-id - --pod-id=pod.id - --tag=tag - --force=force - if exception: - tag-errors.add "Tag '$tag' already exists for pod $pod.name." - - registered-pods := broker-connection_.pod-registry-pods --fleet-id=fleet-id --pod-ids=[pod.id] - pod-entry/PodRegistryEntry := registered-pods[0] - - sorted-uploaded-tags := pod-entry.tags.sort - return UploadResult - --fleet-id=fleet-id - --id=pod.id - --name=pod.name - --revision=pod-entry.revision - --tags=sorted-uploaded-tags - --tag-errors=tag-errors - patch-uploader__/FirmwarePatchUploader? := null patch-uploader_ -> FirmwarePatchUploader: @@ -248,189 +147,20 @@ class Broker: --cli=cli_ return patch-uploader__ - is-cached --pod-id/Uuid -> bool: - manifest-key := cache-key-pod-manifest - --broker-config=server-config - --organization-id=organization-id - --pod-id=pod-id - return cli_.cache.contains manifest-key - - download --pod-id/Uuid -> Pod: - manifest-key := cache-key-pod-manifest - --broker-config=server-config - --organization-id=organization-id - --pod-id=pod-id - encoded-manifest := cli_.cache.get manifest-key: | store/FileStore | - bytes := broker-connection_.pod-registry-download-pod-manifest - --pod-id=pod-id - --organization-id=organization-id - store.save bytes - manifest := ubjson.decode encoded-manifest - return Pod.from-manifest - manifest - --tmp-directory=tmp-directory_ - --download=: | part-id/string | - key := cache-key-pod-parts - --broker-config=server-config - --organization-id=organization-id - --part-id=part-id - cli_.cache.get key: | store/FileStore | - bytes := broker-connection_.pod-registry-download-pod-part - part-id - --organization-id=organization-id - store.save bytes - - list-pods --names/List -> Map: - descriptions := ? - if names.is-empty: - descriptions = broker-connection_.pod-registry-descriptions --fleet-id=fleet-id - else: - descriptions = broker-connection_.pod-registry-descriptions + pod-store__/BrokerPodStore? := null + + /** Pod registry view, lazily constructed on first access. */ + pod-store -> BrokerPodStore: + if not pod-store__: + pod-store__ = BrokerPodStore --fleet-id=fleet-id --organization-id=organization-id - --names=names - --no-create-if-absent - result := {:} - descriptions.do: | description/PodRegistryDescription | - pods := broker-connection_.pod-registry-pods --pod-description-id=description.id - result[description] = pods - return result - - delete --description-names/List: - descriptions := broker-connection_.pod-registry-descriptions - --fleet-id=fleet-id - --organization-id=organization-id - --names=description-names - --no-create-if-absent - unknown-pod-descriptions := [] - description-names.do: | name/string | - was-found := descriptions.any: | description/PodRegistryDescription | - description.name == name - if not was-found: unknown-pod-descriptions.add name - if not unknown-pod-descriptions.is-empty: - if unknown-pod-descriptions.size == 1: - cli_.ui.abort "Unknown pod '$unknown-pod-descriptions[0]'." - else: - quoted := unknown-pod-descriptions.map: "'$it'" - joined := quoted.join ", " - cli_.ui.abort "Unknown pods $joined." - broker-connection_.pod-registry-descriptions-delete - --fleet-id=fleet-id - --description-ids=descriptions.map: it.id - - delete --pod-references/List: - pod-ids := get-pod-ids pod-references - delete --pod-ids=pod-ids - - delete --pod-ids/List: - broker-connection_.pod-registry-delete - --fleet-id=fleet-id - --pod-ids=pod-ids - - add-tags --tags/List --force/bool --references/List: - references = references.map: | reference/PodReference | - reference.is-name-only - ? reference.with --tag="latest" - : reference - - pod-ids := get-pod-ids references - pod-entries := broker-connection_.pod-registry-pods - --fleet-id=fleet-id - --pod-ids=pod-ids - - mapping := {:} - for i := 0; i < pod-ids.size; i++: - mapping[pod-ids[i]] = references[i] - - tag-errors := [] - tags.do: | tag/string | - pod-entries.do: | pod-entry/PodRegistryEntry | - print-on-stderr_ "pod-entry: $pod-entry.to-json" - exception := catch --unwind=(: not is-existing-tag-error_ it): - broker-connection_.pod-registry-tag-set - --pod-description-id=pod-entry.pod-description-id - --pod-id=pod-entry.id - --tag=tag - --force=force - if exception: - ref/PodReference := mapping[pod-entry.id] - tag-errors.add "Tag '$tag' already exists for pod $ref.name." - - if not tag-errors.is-empty: - tag-errors.do: cli_.ui.emit --error it - cli_.ui.abort - - remove-tags --tags/List --references/List: - names := {} - references.do: | reference/PodReference | - assert: reference.is-name-only - names.add reference.name - - descriptions := broker-connection_.pod-registry-descriptions - --fleet-id=fleet-id - --organization-id=organization-id - --names=names.to-list - --no-create-if-absent - - descriptions.do: | description/PodRegistryDescription | - description-id := description.id - tags.do: | tag/string | - broker-connection_.pod-registry-tag-remove - --pod-description-id=description-id - --tag=tag - - get-pod-ids references/List -> List: - references.do: | reference/PodReference | - if not reference.id: - if not reference.name: - throw "Either id or name must be specified: $reference" - if not reference.tag and not reference.revision: - throw "Either tag or revision must be specified: $reference" - - missing-ids := references.filter: | reference/PodReference | - not reference.id - pod-ids-response := broker-connection_.pod-registry-pod-ids - --fleet-id=fleet-id - --references=missing-ids - - has-errors := false - result := references.map: | reference/PodReference | - if reference.id: continue.map reference.id - resolved := pod-ids-response.get reference - if not resolved: - has-errors = true - if reference.tag: - cli_.ui.emit --error "No pod with name '$reference.name' and tag '$reference.tag' in the fleet." - else: - cli_.ui.emit --error "No pod with name '$reference.name' and revision $reference.revision in the fleet." - resolved - if has-errors: cli_.ui.abort - return result - - pod pod-id/Uuid -> PodBroker: - pod-entry := broker-connection_.pod-registry-pods - --fleet-id=fleet-id - --pod-ids=[pod-id] - if not pod-entry.is-empty: - description-id := pod-entry[0].pod-description-id - description := broker-connection_.pod-registry-descriptions --ids=[description-id] - if not description.is-empty: - return PodBroker --id=pod-id --name=description[0].name --revision=pod-entry[0].revision --tags=pod-entry[0].tags - - return PodBroker --id=pod-id --name=null --revision=null --tags=null - - get-pod-id reference/PodReference -> Uuid: - return (get-pod-ids [reference])[0] - - get-pod-id --name/string --tag/string? --revision/int? -> Uuid: - return get-pod-id (PodReference --name=name --tag=tag --revision=revision) - - pod-exists reference/PodReference -> bool: - pod-id := get-pod-id reference - pod-entry := broker-connection_.pod-registry-pods - --fleet-id=fleet-id - --pod-ids=[pod-id] - return not pod-entry.is-empty + --server-config=server-config + --broker-connection=broker-connection_ + --patch-uploader=patch-uploader_ + --cli=cli_ + --tmp-directory=tmp-directory_ + return pod-store__ /** Fetches the device details for the given device ids. @@ -617,37 +347,6 @@ class Broker: --limit=limit --types=types - /** - Fetches the pod information for the given $pod-ids. - - Returns a map from pod id to $PodRegistryEntry. - */ - get-pod-registry-entry-map --pod-ids/List -> Map: - pod-id-entries := broker-connection_.pod-registry-pods - --fleet-id=fleet-id - --pod-ids=pod-ids - pod-entry-map := {:} - pod-id-entries.do: | entry/PodRegistryEntry | - pod-entry-map[entry.id] = entry - return pod-entry-map - - /** - Returns a map from description-id to $PodRegistryDescription. - - The given $pod-registry-entries must be a list of $PodRegistryEntry instances. - */ - get-pod-descriptions --pod-registry-entries/List -> Map: - description-set := {} - description-set.add-all - (pod-registry-entries.map: | entry/PodRegistryEntry | entry.pod-description-id) - description-ids := [] - description-ids.add-all description-set - descriptions := broker-connection_.pod-registry-descriptions --ids=description-ids - description-map := {:} - descriptions.do: | description/PodRegistryDescription | - description-map[description.id] = description - return description-map - notify-created device/Device -> none: identity := { "device_id": "$device.id", diff --git a/src/cli/brokers/broker-pod-store.toit b/src/cli/brokers/broker-pod-store.toit new file mode 100644 index 00000000..dfa5d74e --- /dev/null +++ b/src/cli/brokers/broker-pod-store.toit @@ -0,0 +1,319 @@ +// Copyright (C) 2026 Toit contributors. + +import cli show Cli FileStore +import encoding.ubjson +import uuid show Uuid + +import .broker show BrokerCli +import ..cache +import ..firmware-patches show FirmwarePatchUploader +import ..pod show Pod +import ..pod-registry +import ..pod-store show PodInfo PodStore UploadResult +import ...shared.server-config + +/** +$PodStore implementation backed by a $BrokerCli. + +Stores descriptions/entries/tags in the broker's pod registry tables + and parts/manifests in the broker's pod blob storage. Local cache + keys mirror the broker identity and organization id so multiple + brokers can coexist without collisions. + +Side effect of $upload: trivial firmware patches in the pod's envelope + are pre-uploaded to the broker's firmware bucket via + $FirmwarePatchUploader, so devices can later pull upgrade patches + efficiently. +*/ +class BrokerPodStore implements PodStore: + fleet-id_/Uuid + organization-id_/Uuid + server-config_/ServerConfig + broker-connection_/BrokerCli + patch-uploader_/FirmwarePatchUploader + cli_/Cli + tmp-directory_/string + + constructor + --fleet-id/Uuid + --organization-id/Uuid + --server-config/ServerConfig + --broker-connection/BrokerCli + --patch-uploader/FirmwarePatchUploader + --cli/Cli + --tmp-directory/string: + fleet-id_ = fleet-id + organization-id_ = organization-id + server-config_ = server-config + broker-connection_ = broker-connection + patch-uploader_ = patch-uploader + cli_ = cli + tmp-directory_ = tmp-directory + + static is-existing-tag-error_ error -> bool: + if error is not string: return false + return error.contains "duplicate key value" or error.contains "already exists" + + upload --pod/Pod --tags/List --force-tags/bool -> UploadResult: + patch-uploader_.upload-trivial-patches --pod=pod + + pod.split: | manifest/Map parts/Map | + parts.do: | id/string contents/ByteArray | + // Only upload if we don't have it in our cache. + key := cache-key-pod-parts + --broker-config=server-config_ + --organization-id=organization-id_ + --part-id=id + cli_.cache.get-file-path key: | store/FileStore | + broker-connection_.pod-registry-upload-pod-part contents --part-id=id + --organization-id=organization-id_ + store.save contents + key := cache-key-pod-manifest + --broker-config=server-config_ + --organization-id=organization-id_ + --pod-id=pod.id + cli_.cache.get-file-path key: | store/FileStore | + encoded := ubjson.encode manifest + broker-connection_.pod-registry-upload-pod-manifest encoded --pod-id=pod.id + --organization-id=organization-id_ + store.save encoded + + description-ids := broker-connection_.pod-registry-descriptions + --fleet-id=fleet-id_ + --organization-id=organization-id_ + --names=[pod.name] + --create-if-absent + + description-id := (description-ids[0] as PodRegistryDescription).id + + broker-connection_.pod-registry-add + --pod-description-id=description-id + --pod-id=pod.id + + tag-errors := [] + tags.do: | tag/string | + force := force-tags or (tag == "latest") + exception := catch --unwind=(: not is-existing-tag-error_ it): + broker-connection_.pod-registry-tag-set + --pod-description-id=description-id + --pod-id=pod.id + --tag=tag + --force=force + if exception: + tag-errors.add "Tag '$tag' already exists for pod $pod.name." + + registered-pods := broker-connection_.pod-registry-pods --fleet-id=fleet-id_ --pod-ids=[pod.id] + pod-entry/PodRegistryEntry := registered-pods[0] + + sorted-uploaded-tags := pod-entry.tags.sort + return UploadResult + --fleet-id=fleet-id_ + --id=pod.id + --name=pod.name + --revision=pod-entry.revision + --tags=sorted-uploaded-tags + --tag-errors=tag-errors + + is-cached --pod-id/Uuid -> bool: + manifest-key := cache-key-pod-manifest + --broker-config=server-config_ + --organization-id=organization-id_ + --pod-id=pod-id + return cli_.cache.contains manifest-key + + download --pod-id/Uuid -> Pod: + manifest-key := cache-key-pod-manifest + --broker-config=server-config_ + --organization-id=organization-id_ + --pod-id=pod-id + encoded-manifest := cli_.cache.get manifest-key: | store/FileStore | + bytes := broker-connection_.pod-registry-download-pod-manifest + --pod-id=pod-id + --organization-id=organization-id_ + store.save bytes + manifest := ubjson.decode encoded-manifest + return Pod.from-manifest + manifest + --tmp-directory=tmp-directory_ + --download=: | part-id/string | + key := cache-key-pod-parts + --broker-config=server-config_ + --organization-id=organization-id_ + --part-id=part-id + cli_.cache.get key: | store/FileStore | + bytes := broker-connection_.pod-registry-download-pod-part + part-id + --organization-id=organization-id_ + store.save bytes + + list-pods --names/List -> Map: + descriptions := ? + if names.is-empty: + descriptions = broker-connection_.pod-registry-descriptions --fleet-id=fleet-id_ + else: + descriptions = broker-connection_.pod-registry-descriptions + --fleet-id=fleet-id_ + --organization-id=organization-id_ + --names=names + --no-create-if-absent + result := {:} + descriptions.do: | description/PodRegistryDescription | + pods := broker-connection_.pod-registry-pods --pod-description-id=description.id + result[description] = pods + return result + + delete --description-names/List -> none: + descriptions := broker-connection_.pod-registry-descriptions + --fleet-id=fleet-id_ + --organization-id=organization-id_ + --names=description-names + --no-create-if-absent + unknown-pod-descriptions := [] + description-names.do: | name/string | + was-found := descriptions.any: | description/PodRegistryDescription | + description.name == name + if not was-found: unknown-pod-descriptions.add name + if not unknown-pod-descriptions.is-empty: + if unknown-pod-descriptions.size == 1: + cli_.ui.abort "Unknown pod '$unknown-pod-descriptions[0]'." + else: + quoted := unknown-pod-descriptions.map: "'$it'" + joined := quoted.join ", " + cli_.ui.abort "Unknown pods $joined." + broker-connection_.pod-registry-descriptions-delete + --fleet-id=fleet-id_ + --description-ids=descriptions.map: it.id + + delete --pod-references/List -> none: + pod-ids := get-pod-ids pod-references + delete --pod-ids=pod-ids + + delete --pod-ids/List -> none: + broker-connection_.pod-registry-delete + --fleet-id=fleet-id_ + --pod-ids=pod-ids + + add-tags --tags/List --force/bool --references/List -> none: + references = references.map: | reference/PodReference | + reference.is-name-only + ? reference.with --tag="latest" + : reference + + pod-ids := get-pod-ids references + pod-entries := broker-connection_.pod-registry-pods + --fleet-id=fleet-id_ + --pod-ids=pod-ids + + mapping := {:} + for i := 0; i < pod-ids.size; i++: + mapping[pod-ids[i]] = references[i] + + tag-errors := [] + tags.do: | tag/string | + pod-entries.do: | pod-entry/PodRegistryEntry | + exception := catch --unwind=(: not is-existing-tag-error_ it): + broker-connection_.pod-registry-tag-set + --pod-description-id=pod-entry.pod-description-id + --pod-id=pod-entry.id + --tag=tag + --force=force + if exception: + ref/PodReference := mapping[pod-entry.id] + tag-errors.add "Tag '$tag' already exists for pod $ref.name." + + if not tag-errors.is-empty: + tag-errors.do: cli_.ui.emit --error it + cli_.ui.abort + + remove-tags --tags/List --references/List -> none: + names := {} + references.do: | reference/PodReference | + assert: reference.is-name-only + names.add reference.name + + descriptions := broker-connection_.pod-registry-descriptions + --fleet-id=fleet-id_ + --organization-id=organization-id_ + --names=names.to-list + --no-create-if-absent + + descriptions.do: | description/PodRegistryDescription | + description-id := description.id + tags.do: | tag/string | + broker-connection_.pod-registry-tag-remove + --pod-description-id=description-id + --tag=tag + + get-pod-ids references/List -> List: + references.do: | reference/PodReference | + if not reference.id: + if not reference.name: + throw "Either id or name must be specified: $reference" + if not reference.tag and not reference.revision: + throw "Either tag or revision must be specified: $reference" + + missing-ids := references.filter: | reference/PodReference | + not reference.id + pod-ids-response := broker-connection_.pod-registry-pod-ids + --fleet-id=fleet-id_ + --references=missing-ids + + has-errors := false + result := references.map: | reference/PodReference | + if reference.id: continue.map reference.id + resolved := pod-ids-response.get reference + if not resolved: + has-errors = true + if reference.tag: + cli_.ui.emit --error "No pod with name '$reference.name' and tag '$reference.tag' in the fleet." + else: + cli_.ui.emit --error "No pod with name '$reference.name' and revision $reference.revision in the fleet." + resolved + if has-errors: cli_.ui.abort + return result + + pod pod-id/Uuid -> PodInfo: + pod-entry := broker-connection_.pod-registry-pods + --fleet-id=fleet-id_ + --pod-ids=[pod-id] + if not pod-entry.is-empty: + description-id := pod-entry[0].pod-description-id + description := broker-connection_.pod-registry-descriptions --ids=[description-id] + if not description.is-empty: + return PodInfo --id=pod-id --name=description[0].name --revision=pod-entry[0].revision --tags=pod-entry[0].tags + + return PodInfo --id=pod-id --name=null --revision=null --tags=null + + get-pod-id reference/PodReference -> Uuid: + return (get-pod-ids [reference])[0] + + get-pod-id --name/string --tag/string? --revision/int? -> Uuid: + return get-pod-id (PodReference --name=name --tag=tag --revision=revision) + + pod-exists reference/PodReference -> bool: + pod-id := get-pod-id reference + pod-entry := broker-connection_.pod-registry-pods + --fleet-id=fleet-id_ + --pod-ids=[pod-id] + return not pod-entry.is-empty + + get-pod-registry-entry-map --pod-ids/List -> Map: + pod-id-entries := broker-connection_.pod-registry-pods + --fleet-id=fleet-id_ + --pod-ids=pod-ids + pod-entry-map := {:} + pod-id-entries.do: | entry/PodRegistryEntry | + pod-entry-map[entry.id] = entry + return pod-entry-map + + get-pod-descriptions --pod-registry-entries/List -> Map: + description-set := {} + description-set.add-all + (pod-registry-entries.map: | entry/PodRegistryEntry | entry.pod-description-id) + description-ids := [] + description-ids.add-all description-set + descriptions := broker-connection_.pod-registry-descriptions --ids=description-ids + description-map := {:} + descriptions.do: | description/PodRegistryDescription | + description-map[description.id] = description + return description-map diff --git a/src/cli/fleet.toit b/src/cli/fleet.toit index eed85e17..42c14a1b 100644 --- a/src/cli/fleet.toit +++ b/src/cli/fleet.toit @@ -17,6 +17,7 @@ import .firmware import .pod import .pod-specification import .pod-registry +import .pod-store show PodInfo UploadResult import .utils import .utils.names import .server-config @@ -415,7 +416,7 @@ class Fleet: upload --pod/Pod --tags/List --force-tags/bool -> UploadResult: cli_.ui.emit --info "Uploading pod. This may take a while." - return broker.upload + return broker.pod-store.upload --pod=pod --tags=tags --force-tags=force-tags @@ -426,39 +427,39 @@ class Fleet: pod-id := reference.id if not pod-id: pod-id = get-pod-id reference - if not broker.is-cached --pod-id=pod-id: + if not broker.pod-store.is-cached --pod-id=pod-id: cli_.ui.emit --info "Downloading pod '$reference'." return download --pod-id=pod-id download --pod-id/Uuid -> Pod: - return broker.download --pod-id=pod-id + return broker.pod-store.download --pod-id=pod-id list-pods --names/List -> Map: - return broker.list-pods --names=names + return broker.pod-store.list-pods --names=names delete --description-names/List: - broker.delete --description-names=description-names + broker.pod-store.delete --description-names=description-names delete --pod-references/List: - broker.delete --pod-references=pod-references + broker.pod-store.delete --pod-references=pod-references add-tags --tags/List --force/bool --references/List: - broker.add-tags --tags=tags --force=force --references=references + broker.pod-store.add-tags --tags=tags --force=force --references=references remove-tags --tags/List --references/List: - broker.remove-tags --tags=tags --references=references + broker.pod-store.remove-tags --tags=tags --references=references - pod pod-id/Uuid -> PodBroker: - return broker.pod pod-id + pod pod-id/Uuid -> PodInfo: + return broker.pod-store.pod pod-id get-pod-id reference/PodReference -> Uuid: - return broker.get-pod-id reference + return broker.pod-store.get-pod-id reference get-pod-id --name/string --tag/string? --revision/int? -> Uuid: - return broker.get-pod-id --name=name --tag=tag --revision=revision + return broker.pod-store.get-pod-id --name=name --tag=tag --revision=revision pod-exists reference/PodReference -> bool: - return broker.pod-exists reference + return broker.pod-store.pod-exists reference recovery-urls -> List: return fleet-file_.recovery-urls @@ -904,9 +905,9 @@ class FleetWithDevices extends Fleet: if detailed-device.pod-id-current: pod-ids.add detailed-device.pod-id-current if detailed-device.pod-id-firmware: pod-ids.add detailed-device.pod-id-firmware - broker-pod-entry-map := current-broker.get-pod-registry-entry-map --pod-ids=pod-ids.to-list + broker-pod-entry-map := current-broker.pod-store.get-pod-registry-entry-map --pod-ids=pod-ids.to-list pod-entries[current-broker.server-config.name] = broker-pod-entry-map - broker-description-map := current-broker.get-pod-descriptions + broker-description-map := current-broker.pod-store.get-pod-descriptions --pod-registry-entries=broker-pod-entry-map.values pod-descriptions[current-broker.server-config.name] = broker-description-map diff --git a/src/cli/pod-store.toit b/src/cli/pod-store.toit new file mode 100644 index 00000000..222baddb --- /dev/null +++ b/src/cli/pod-store.toit @@ -0,0 +1,154 @@ +// Copyright (C) 2026 Toit contributors. + +import uuid show Uuid + +import .pod show Pod +import .pod-registry show PodReference + +/** +A store of pods, scoped to a single (fleet, organization) pair. + +Today the only implementation is broker-backed. Future implementations + could store pods on a filesystem (for `git`-managed fleets) or fetch + them from an HTTP-only source (for example a public release page). +*/ +interface PodStore: + /** + Uploads $pod and registers it under its embedded name. + + Applies the given $tags after upload. If $force-tags is true, tags + that already point to a different pod are moved; otherwise such tags + are reported as errors in the result. The "latest" tag is always + moved. + + Implementations may also upload trivial firmware patches as a side + effect, so devices can later download upgrade patches without + needing the original envelope. + */ + upload --pod/Pod --tags/List --force-tags/bool -> UploadResult + + /** + Downloads the pod with the given $pod-id, materialising it on disk. + */ + download --pod-id/Uuid -> Pod + + /** Whether the pod with the given $pod-id is in the local cache. */ + is-cached --pod-id/Uuid -> bool + + /** + Returns the pods of the given $names. + + If $names is empty, returns all pod descriptions in the fleet. + + Returns a map from $PodRegistryDescription to a List of + $PodRegistryEntry instances. + */ + list-pods --names/List -> Map + + /** + Deletes pod descriptions (and all their entries) by name. + + Aborts via the CLI UI if any of the names do not exist. + */ + delete --description-names/List -> none + + /** + Deletes individual pods identified by reference. + + References are resolved to pod ids first; missing references abort. + */ + delete --pod-references/List -> none + + /** Deletes individual pods by id. */ + delete --pod-ids/List -> none + + /** + Sets $tags on the pods identified by $references. + + Name-only references are resolved against the "latest" tag. If + $force is false, attempting to set an existing tag aborts via the + CLI UI. + */ + add-tags --tags/List --force/bool --references/List -> none + + /** + Removes $tags from the pod descriptions named in $references. + + All references must be name-only. + */ + remove-tags --tags/List --references/List -> none + + /** + Resolves a list of $PodReference instances to their pod ids. + + Aborts via the CLI UI if any reference does not resolve. + */ + get-pod-ids references/List -> List + + /** Resolves a single $reference to its pod id. */ + get-pod-id reference/PodReference -> Uuid + + /** Convenience for resolving by name + tag/revision. */ + get-pod-id --name/string --tag/string? --revision/int? -> Uuid + + /** Whether the given $reference resolves to an existing pod. */ + pod-exists reference/PodReference -> bool + + /** + Returns metadata ($PodInfo) for the pod with the given $pod-id. + + If the pod does not exist, returns a $PodInfo with null fields. + */ + pod pod-id/Uuid -> PodInfo + + /** + Fetches pod entries for the given ids. + + Returns a map from pod id to $PodRegistryEntry. + */ + get-pod-registry-entry-map --pod-ids/List -> Map + + /** + Resolves the descriptions referenced by the given pod entries. + + Returns a map from description id to $PodRegistryDescription. + */ + get-pod-descriptions --pod-registry-entries/List -> Map + +/** +Result of $PodStore.upload. +*/ +class UploadResult: + fleet-id/Uuid + id/Uuid + name/string + revision/int + tags/List + tag-errors/List + + constructor --.fleet-id --.id --.name --.revision --.tags --.tag-errors: + + to-json -> Map: + result := { + "fleet-id": "$fleet-id", + "id": "$id", + "name": name, + "revision": revision, + "tags": tags, + } + if not tag-errors.is-empty: + result["tag-errors"] = tag-errors + return result + +/** +Lightweight metadata for a pod resolved by id. + +Fields are null when the pod does not exist in the store. +*/ +class PodInfo: + id/Uuid + name/string? + revision/int? + tags/List? + + constructor --.id --.name --.revision --.tags: From 025abf78cf9fb9d0c592f7188dd0c9612a4070e0 Mon Sep 17 00:00:00 2001 From: Florian Loitsch Date: Sun, 17 May 2026 14:45:20 +0200 Subject: [PATCH 7/7] Remove planning directory. --- .planning/codebase/ARCHITECTURE.md | 146 -------------- .planning/codebase/CONCERNS.md | 238 ---------------------- .planning/codebase/CONVENTIONS.md | 186 ----------------- .planning/codebase/INTEGRATIONS.md | 170 ---------------- .planning/codebase/STACK.md | 108 ---------- .planning/codebase/STRUCTURE.md | 219 -------------------- .planning/codebase/TESTING.md | 311 ----------------------------- 7 files changed, 1378 deletions(-) delete mode 100644 .planning/codebase/ARCHITECTURE.md delete mode 100644 .planning/codebase/CONCERNS.md delete mode 100644 .planning/codebase/CONVENTIONS.md delete mode 100644 .planning/codebase/INTEGRATIONS.md delete mode 100644 .planning/codebase/STACK.md delete mode 100644 .planning/codebase/STRUCTURE.md delete mode 100644 .planning/codebase/TESTING.md diff --git a/.planning/codebase/ARCHITECTURE.md b/.planning/codebase/ARCHITECTURE.md deleted file mode 100644 index 74f8491b..00000000 --- a/.planning/codebase/ARCHITECTURE.md +++ /dev/null @@ -1,146 +0,0 @@ -# Architecture - -**Analysis Date:** 2026-03-15 - -## Pattern Overview - -**Overall:** Layered client-server architecture with a separation between CLI management layer and device-side service layer. The device runs a scheduler-based job execution model that synchronizes state with a broker. - -**Key Characteristics:** -- CLI and service are separate binaries communicating through brokers (HTTP/Supabase) -- Device-side uses event-driven scheduler for responsive job execution -- State synchronization model where goal state is fetched from broker and containers/firmware are updated to match -- Pluggable broker architecture supporting multiple backend implementations -- Job-based execution model with priorities and runlevels for clean shutdown and state management - -## Layers - -**CLI Layer:** -- Purpose: Command-line interface for managing devices, fleets, pods, and configurations -- Location: `src/cli/` -- Contains: Command handlers, user-facing UI, broker clients, authentication -- Depends on: Broker implementations, shared constants and utilities -- Used by: End users via command-line invocation - -**Service Layer:** -- Purpose: Runs on devices to manage containers, firmware updates, and state synchronization -- Location: `src/service/` -- Contains: Job scheduler, container manager, broker connections, device state -- Depends on: Broker implementations, shared utilities, system services -- Used by: Device runtime environment - -**Shared Layer:** -- Purpose: Common utilities and constants used by both CLI and service -- Location: `src/shared/` -- Contains: Version info, server configuration, utilities, JSON diffing logic, patch tools -- Depends on: Third-party packages only -- Used by: Both CLI and service layers - -**Broker Layer:** -- Purpose: Communication abstraction between devices and management servers -- Location: `src/cli/brokers/` (CLI side) and `src/service/brokers/` (service side) -- Contains: HTTP broker implementation, Supabase broker implementation, broker interfaces -- Depends on: HTTP client libraries, encryption libraries -- Used by: CLI and service for device-broker communication - -## Data Flow - -**Device Synchronization Flow (Primary):** - -1. **Initialization**: Service starts on device via `run-artemis()` in `src/service/service.toit` -2. **Scheduler Setup**: `Scheduler` created with `Device`, `ContainerManager`, and `BrokerService` -3. **Synchronization Job**: `SynchronizeJob` connects to broker to fetch goal state -4. **Goal Processing**: Compares current device state with goal state from broker -5. **Updates Applied**: - - Container images downloaded and installed via `ContainerManager` - - Firmware updates applied via `FirmwareUpdateJob` - - Device state persisted to storage -6. **Report Back**: Device state and events reported back to broker -7. **Idle/Wait**: Scheduler waits until next job or state change - -**CLI Command Flow (Secondary):** - -1. User invokes command (e.g., `artemis device update`) -2. CLI command in `src/cli/cmds/` constructs request -3. Request routed through `Artemis` class or `BrokerCli` wrapper -4. API call made through configured broker (HTTP or Supabase) -5. Response returned and formatted for user output - -**State Management:** - -- Device state stored in `Device` class: current applications, firmware version, configuration -- Goal state received from broker as JSON map comparing containers and firmware -- State reconciliation via `json-diff` logic to compute minimal required changes -- State persistence: Job states serialized before deep sleep, restored on wake - -## Key Abstractions - -**Job System:** -- Purpose: Abstraction for schedulable, cancellable work units on device -- Examples: `src/service/jobs.toit` (base), `src/service/containers.toit` (container jobs), `src/service/synchronize.toit` (sync job) -- Pattern: Abstract `Job` class with lifecycle (`start`, `stop`, `schedule`). Subclasses implement scheduling logic. Scheduler tracks and runs jobs. - -**BrokerConnection Interface:** -- Purpose: Protocol abstraction for device-to-broker communication -- Examples: `src/service/brokers/http/http.toit` (HTTP implementation) -- Pattern: Interface defines methods for fetching goals, downloading images/firmware, reporting state and events. Implementations handle transport specifics. - -**ArtemisService Provider:** -- Purpose: Runtime API service for containers to access Artemis capabilities -- Location: `src/service/service.toit` (class `ArtemisServiceProvider`) -- Pattern: Implements Toit service protocol to expose device ID, reboot, container control methods at runtime - -**Pod:** -- Purpose: Container image with metadata and specification -- Location: `src/cli/pod.toit` and `src/cli/pod-specification.toit` -- Pattern: Contains container definition, triggers, environment variables, serialized as JSON - -**Firmware:** -- Purpose: Device firmware image with versioning and update capability -- Location: `src/cli/firmware.toit` and `src/service/firmware-update.toit` -- Pattern: Firmware identified by version string, downloaded in chunks, verified by SHA256 - -## Entry Points - -**CLI Entry Point:** -- Location: `src/cli/artemis.toit` (main function) -- Triggers: Command-line invocation with arguments -- Responsibilities: Sets up certificate roots, parses root command structure, routes to subcommands - -**Service Entry Point:** -- Location: `src/service/service.toit` (function `run-artemis`) -- Triggers: Device boots with Artemis service configured -- Responsibilities: Initializes scheduler, broker, container manager; runs main event loop - -**Artemis Server CLI (Device Management):** -- Location: `src/cli/artemis_servers/artemis-server.toit` -- Triggers: CLI needs to communicate with Artemis server API -- Responsibilities: Authenticates user, creates devices, lists SDK/service versions, downloads service images - -**Synchronizer Job (Device-side):** -- Location: `src/service/synchronize.toit` -- Triggers: Scheduled by the scheduler based on synchronization intervals -- Responsibilities: Connects to broker, fetches goal state, applies container/firmware updates, reports state - -## Error Handling - -**Strategy:** Try-catch blocks with graceful degradation. Critical errors trigger reboot or state reset. Network errors retry with backoff. - -**Patterns:** -- Broker connection failures: Retried with exponential backoff via synchronization job states (state machine with DISCONNECTED → CONNECTING → CONNECTED states) -- Container download failures: Logged with tags, container marked failed, synchronization continues -- Firmware update failures: Error reported back to broker, device remains in current state -- CLI errors: User-friendly messages via `cli.ui.abort()` or `cli.ui.error()` -- Service errors: Logged with context tags (device ID, version, job name) - -## Cross-Cutting Concerns - -**Logging:** Uses Toit `log` package. Loggers created with context via `.with-name` to track component (e.g., "scheduler", "containers", "synchronize"). Tags added to log entries for structured context. - -**Validation:** Pod specifications validated against schema in `src/cli/pod-specification.toit`. Device names and references validated in CLI command handlers. Configuration validated when loaded. - -**Authentication:** User authentication handled by `ArtemisServerCli` class. Credentials cached in local config. Tokens refreshed on demand via `ensure-authenticated` method. - ---- - -*Architecture analysis: 2026-03-15* diff --git a/.planning/codebase/CONCERNS.md b/.planning/codebase/CONCERNS.md deleted file mode 100644 index e7d6ea71..00000000 --- a/.planning/codebase/CONCERNS.md +++ /dev/null @@ -1,238 +0,0 @@ -# Codebase Concerns - -**Analysis Date:** 2026-03-15 - -## Tech Debt - -**Artemis API Package Duplication:** -- Issue: Artemis API package has been temporarily copied from the official repository instead of imported as a dependency. Comments in `src/service/service.toit` (line 8-18) and `src/service/containers.toit` (line 10-18) explicitly state the API will be deleted once changes stabilize. -- Files: `src/service/service.toit`, `src/service/containers.toit`, `artemis-pkg-copy/` -- Impact: Maintenance burden when API changes - all changes must be synchronized with two copies. Risk of divergence between local copy and upstream. When deleted, import statements must be updated throughout codebase. -- Fix approach: Monitor when `toit-artemis` package reaches stable API, then switch imports from `artemis-pkg.api` back to `artemis.api` and remove `artemis-pkg-copy/` directory. Add pre-commit hook or CI check to prevent accidental modifications to copied package. - -**Container Image Bundled Detection Heuristic:** -- Issue: Determining if a container image is bundled relies on checking if `image.name != null` (see `src/service/containers.toit` lines 46-50). This is documented as "a bit of a hack" in TODO comment. -- Files: `src/service/containers.toit` (lines 46-50) -- Impact: Brittle logic that could break if API changes. Unclear contract between image name presence and bundled status. No explicit bundled property available. -- Fix approach: Add explicit bundled flag to container image API, or create dedicated method in API to check bundled status rather than inferring from name. - -**Container Lookup by GID Linear Search:** -- Issue: Finding a container by GID requires linear iteration through all jobs (see `src/service/containers.toit` lines 88-93). TODO suggests optimization via secondary map. -- Files: `src/service/containers.toit` (lines 88-93) -- Impact: O(n) lookup on potentially frequent GID lookups. If number of containers grows, performance degrades. -- Fix approach: Maintain secondary `jobs-by-gid_` map alongside existing `jobs_` map. Update map on container add/remove operations. - -**Firmware Part Matching by Index:** -- Issue: Firmware parts are matched between old and new firmware by array index rather than by name/type (see `src/service/firmware-update.toit` line 115 TODO). -- Files: `src/service/firmware-update.toit` (lines 115-116) -- Impact: Risk of corruption if firmware structure changes and parts reorder. Assumes rigid part ordering across firmware versions. -- Fix approach: Implement name/type-based matching for firmware parts. Store part metadata (name/type) and use for matching instead of array indices. - -**Container Image Installation Error Handling Gap:** -- Issue: Container loading in `src/service/containers.toit` (lines 59-63) has ambiguous error handling. If container image not found in flash, job is silently skipped but comments ask "Should we drop such an app from the current state?" -- Files: `src/service/containers.toit` (lines 52-67) -- Impact: Missing container images fail silently, potentially leaving device in inconsistent state. No logging or recovery mechanism. -- Fix approach: Explicitly log missing container images. Consider whether to treat as error vs silent skip. Possibly preserve container metadata for later recovery. - -**Container Required Status Management:** -- Issue: Required container marking logic (lines 69-77) iterates connections but assumes containers exist. No validation that required containers are actually installed. -- Files: `src/service/containers.toit` (lines 69-77) -- Impact: Can mark nonexistent containers as required without error. May cause synchronization to hang waiting for unavailable containers. -- Fix approach: Validate that all required containers exist before marking. Log warnings for missing required containers. Consider fallback strategy. - -**Container Image Reference Counting:** -- Issue: Image cleanup uses manual iteration through all jobs to check if image is still referenced (see `src/service/containers.toit` lines 135-138 TODO). Comment suggests reference counting would be better. -- Files: `src/service/containers.toit` (lines 135-138) -- Impact: O(n) cleanup on container uninstall. Scales poorly with number of containers. Easy to miss edge cases where image is still in use. -- Fix approach: Implement reference counting system where each job increments/decrements ref count on its image. Only uninstall when ref count reaches zero. - -**Synchronization Error Loop Prevention:** -- Issue: In `src/service/synchronize.toit` (lines 487-490), there's a comment about coding errors in `synchronize-step_` that could cause tight error loops with no fallback mechanism. Example log pattern shown suggests network lookup failures cascading. -- Files: `src/service/synchronize.toit` (lines 487-490) -- Impact: Coding errors in synchronization step handling could cause device to spin in error loop without recovery. Difficult to diagnose from device logs. -- Fix approach: Add explicit error classification in synchronize-step_ that distinguishes between transient network errors and coding errors. Implement exponential backoff with maximum retries. Consider safe-mode trigger for persistent errors. - -## Known Bugs - -**Unreachable Code in Connection Handling:** -- Issue: In `src/service/brokers/http/connection.toit` line 41, method `send-request` has `unreachable` statement after a call that never returns normally. The pattern catches result but then claims unreachable. -- Files: `src/service/brokers/http/connection.toit` (lines 38-41) -- Impact: Code is correct but confusing. If control flow changes, compiler won't catch the error because `unreachable` suppresses warnings. -- Fix approach: Refactor to use explicit return or exception. Remove misleading `unreachable` statement. - -**Assertion Failures on Checkpoint Misalignment:** -- Issue: Multiple assertions check checkpoint assumptions (see `src/service/firmware-update.toit` lines 112, 145). If checkpoints become misaligned due to corruption, assertions fail and crash instead of recovering gracefully. -- Files: `src/service/firmware-update.toit` (lines 112, 145) -- Impact: Firmware update can crash midway if checkpoint metadata corrupts. Device may be left in unrecoverable state. -- Fix approach: Replace assertions with proper error handling. Detect checkpoint corruption early and clear checkpoint to restart from beginning. - -## Security Considerations - -**TLS Session Caching in RTC Memory:** -- Risk: HTTP TLS session cache stored in RTC (RAM) memory that survives deep sleep but loses data on power loss (see `src/service/brokers/http/connection.toit` lines 89-92). -- Files: `src/service/brokers/http/connection.toit` (lines 89-92) -- Current mitigation: Sessions cleared on power loss, preventing replayed sessions from being persistent. TLS protocol provides forward secrecy. -- Recommendations: Document that session data is ephemeral. Consider adding integrity check to detect corrupted cached sessions. Periodically rotate session cache even if valid. - -**Firmware Checkpoint Validation:** -- Risk: Firmware update checkpoints contain old and new firmware checksums but don't verify checkpoint itself (see `src/service/firmware-update.toit` lines 23-25, 252-258). -- Files: `src/service/firmware-update.toit` -- Current mitigation: Checksums validate firmware content, device storage provides basic integrity. -- Recommendations: Add HMAC or signature to checkpoint structure to detect tampering. Validate checkpoint integrity before using it. - -**Network Retry Logic Credential Exposure:** -- Risk: HTTP connection retry logic (lines 56-73 in `src/service/brokers/http/connection.toit`) retries 3 times on 502/520/546 errors. Could theoretically retry with same credentials if broker is compromised. -- Files: `src/service/brokers/http/connection.toit` (lines 56-73) -- Current mitigation: Device headers configured per broker. Client-side secret not exposed. -- Recommendations: Add rate limiting to prevent excessive retry storms. Log all retry attempts for audit trail. - -## Performance Bottlenecks - -**Network Quarantine State Machine Complexity:** -- Problem: Network connection quarantine logic (see `src/service/network.toit` lines 28-148) involves timestamp checks and duration calculations on every connection attempt. Multiple temporary timers if network fails repeatedly. -- Files: `src/service/network.toit` (lines 28-148) -- Cause: Quarantine deadline stored as absolute monotonic microsecond timestamp, must compare against current time each attempt. No batch cleanup of expired quarantines. -- Improvement path: Implement connection quarantine using deadline queue or timer heap. Batch cleanup of expired quarantines on scheduler tick. - -**Scheduler Signal Monitor Custom Implementation:** -- Problem: Scheduler uses custom monitor-based signal mechanism (see `src/service/scheduler.toit` lines 128-137) instead of standard primitives. May not be optimally implemented. -- Files: `src/service/scheduler.toit` (lines 128-137, also TODO comment on line 128) -- Cause: Appears to be custom implementation for non-standard wait semantics. Comment suggests could use standard monitor but doesn't. -- Improvement path: Profile scheduler signal performance. Consider switching to standard Toit monitor primitives if available. Benchmark before/after. - -**Linear Iteration for GID Lookup:** -- Problem: Finding containers by GID requires linear scan (already noted in tech debt section). Called potentially during container RPC handlers. -- Files: `src/service/containers.toit` (lines 88-93) -- Cause: No secondary index by GID. -- Improvement path: Add `jobs-by-gid_` secondary map. Keep synchronized with primary jobs map. - -**Synchronization State Reporting Overhead:** -- Problem: Every synchronization step compares device state with goal state (see `src/service/synchronize.toit` line 504). Full state comparison for each step could be expensive with large state. -- Files: `src/service/synchronize.toit` (lines 486-546) -- Cause: `report-state-if-changed` function likely deep-compares entire state map. -- Improvement path: Implement incremental state tracking. Only report state deltas instead of full state. Cache last reported state. - -## Fragile Areas - -**Synchronization Step Error Handling:** -- Files: `src/service/synchronize.toit` (lines 486-546) -- Why fragile: Synchronization loop is complex with many state transitions. Comments explicitly acknowledge risk of coding errors causing tight error loops (lines 487-490). Log examples show network failures cascading into repeated errors. -- Safe modification: Add comprehensive logging at each state transition. Test error paths explicitly (network failures, timeouts, invalid responses). Consider simplifying state machine into smaller methods. -- Test coverage: Needs end-to-end tests simulating network failures at each step. Mock broker should test both success and failure paths. - -**Firmware Update Checkpoint System:** -- Files: `src/service/firmware-update.toit` -- Why fragile: Checkpoint tracks progress across firmware update but file corruption or bad timing could corrupt checkpoint state. Part matching by array index assumes firmware structure stability. Multiple writes and flushes with potential failure points. -- Safe modification: Add defensive checksum validation before using checkpoint. Handle missing/corrupt checkpoints gracefully by clearing and restarting. Add comprehensive logging of checkpoint lifecycle. -- Test coverage: Test checkpoint corruption scenarios. Test firmware updates interrupted at each checkpoint. Test part reordering in firmware structure. - -**Container Manager Image Lifecycle:** -- Files: `src/service/containers.toit` (lines 33-150) -- Why fragile: Image installation, uninstallation, and bundled status tracking have implicit assumptions about image names and availability. Manual iteration for reference counting. Silent failures on missing images. -- Safe modification: Add validation at each image operation. Explicit logging of image state changes. Consider immutable image metadata structures. Add consistency checks on startup. -- Test coverage: Test missing container images at load time. Test concurrent install/uninstall of same image. Test bundled image protection. - -**Network Manager Connection Quarantine:** -- Files: `src/service/network.toit` (lines 27-150) -- Why fragile: Quarantine deadline logic based on monotonic time comparisons. Multiple ways quarantine could persist incorrectly (time skew, stored value overflow). Iterates connections multiple times in sort. -- Safe modification: Add safeguards against time inconsistencies. Use saturating arithmetic for deadline calculations. Test quarantine expiration explicitly. -- Test coverage: Test time-based quarantine expiration. Test multiple connection failures and recovery. Test quarantine with network transitions. - -**Recovery URL Selection:** -- Files: `src/service/synchronize.toit` (lines 640-641, 652-671) -- Why fragile: Recovery URL picked randomly (line 641). If recovery service is temporarily down, no fallback to try others. Query result cached but minimal validation of response format. -- Safe modification: Validate recovery service response structure before using. Implement retry logic for recovery queries. Log all recovery attempts. -- Test coverage: Test recovery service unavailability. Test malformed recovery response. Test fallback to primary broker. - -## Scaling Limits - -**Container Lookup Performance:** -- Current capacity: O(n) lookup where n = number of containers. Practical limit ~100-1000 containers before noticeable latency. -- Limit: When containers >1000, GID lookups become observable bottleneck during container RPC calls. Synchronization could stall. -- Scaling path: Implement secondary GID index as noted in tech debt. Consider sharding containers if scale exceeds 10,000. - -**State Reporting Frequency:** -- Current capacity: Full state comparison on each synchronization step. With state size <10KB works fine. -- Limit: With large fleets (>10K devices) reporting full state repeatedly, cloud could see excessive traffic. State size growing with container configs could exceed network buffers. -- Scaling path: Implement state delta reporting. Compress state representation. Batch multiple state reports. - -**Firmware Update Bandwidth:** -- Current capacity: Binary patching works efficiently for moderate firmware updates (<100MB). Checkpoint system prevents total loss but adds I/O overhead. -- Limit: Very large firmware images (>500MB) could exhaust device storage for intermediate files. Checkpoint system adds latency to firmware writes. -- Scaling path: Stream firmware in smaller chunks. Implement progressive patching. Consider delta-sync for incremental updates. - -**Quarantine List Memory:** -- Current capacity: Network quarantine list likely <100 entries. Linear search acceptable. -- Limit: With complex network switching topology, quarantine list could grow large. Linear iteration becomes observable. -- Scaling path: Use time-indexed data structure (timer heap). Batch cleanup of expired entries. - -## Dependencies at Risk - -**Toit Language Evolution:** -- Risk: Codebase extensively uses Toit language features including custom monitors, task cancellation, and service providers. These could change in future versions. -- Impact: Major version upgrades of Toit SDK could require substantial refactoring, particularly scheduler and service provider code. -- Migration plan: Monitor Toit SDK changelog. Maintain compatibility shim layer for language features if possible. Plan major refactors around SDK upgrade cycles. - -**HTTP/TLS Library Stability:** -- Risk: HTTP client implementation in broker connection depends on `http` package. TLS session caching relies on undocumented RTC memory buckets. -- Impact: HTTP library changes could affect connection retry logic. TLS session format changes could break cached sessions. -- Migration plan: Monitor HTTP package updates. Abstract HTTP client creation into service provider. Test TLS session migration path explicitly. - -## Missing Critical Features - -**Firmware Rollback Recovery:** -- Problem: Firmware update can fail leaving device in incomplete state. Device has rollback capability but no automated recovery triggers it. Manual intervention required. -- Blocks: Can't reliably deploy faulty firmware updates. Devices can get stuck in validation-pending state. -- Recommendation: Implement automatic rollback trigger after N failed sync attempts post-update. Add metadata to track rollback history. - -**Container Migration Tool:** -- Problem: No tooling to migrate containers between devices or fleets. Container data is ephemeral. -- Blocks: Can't easily redistribute load or backup container state. Disaster recovery requires manual redeployment. -- Recommendation: Build container export/import tool. Document data migration patterns. - -**Network Failover Metrics:** -- Problem: No visibility into why network connections fail or why one connection preferred over another. -- Blocks: Hard to diagnose network configuration issues. Can't optimize connection priority based on actual performance. -- Recommendation: Track and report per-connection success metrics. Log network selection decisions. - -## Test Coverage Gaps - -**Firmware Update Edge Cases:** -- What's not tested: Checkpoint corruption, firmware download interruption at each part boundary, part reordering in firmware structure, concurrent firmware updates -- Files: `src/service/firmware-update.toit` -- Risk: Firmware updates could fail silently or corrupt device. No recovery from corruption scenarios. -- Priority: High - -**Synchronization Error Scenarios:** -- What's not tested: Network errors at each step of synchronization, broker unavailability during image download, timeout handling during state reporting, recovery server fallback -- Files: `src/service/synchronize.toit` -- Risk: Device could get stuck in error loops or miss updates during network issues. -- Priority: High - -**Container Lifecycle Management:** -- What's not tested: Missing container images at boot, concurrent install/uninstall of same image, bundled image protection, container reference counting -- Files: `src/service/containers.toit` -- Risk: Container state inconsistencies, orphaned images, incorrect cleanup. -- Priority: Medium - -**Network Quarantine System:** -- What's not tested: Quarantine deadline expiration, multiple connection failures, connection switching during quarantine, quarantine list memory growth -- Files: `src/service/network.toit` -- Risk: Connections could remain quarantined indefinitely or not quarantine properly. -- Priority: Medium - -**Watchdog Timer Integration:** -- What's not tested: Watchdog creation timeout (line 228), watchdog feeding during long operations, watchdog without service availability -- Files: `src/service/synchronize.toit` (lines 223-237) -- Risk: Watchdog could timeout due to bugs in watchdog integration rather than actual hangs. -- Priority: Medium - -**Task Cancellation Handling:** -- What's not tested: Synchronization cancellation during different states, container startup cancellation, proper cleanup on task.cancel -- Files: `src/service/synchronize.toit` (line 434) -- Risk: Incomplete cleanup on cancellation could leave locks held or connections open. -- Priority: Low - ---- - -*Concerns audit: 2026-03-15* diff --git a/.planning/codebase/CONVENTIONS.md b/.planning/codebase/CONVENTIONS.md deleted file mode 100644 index a78406f3..00000000 --- a/.planning/codebase/CONVENTIONS.md +++ /dev/null @@ -1,186 +0,0 @@ -# Coding Conventions - -**Analysis Date:** 2026-03-15 - -## Naming Patterns - -**Files:** -- Kebab-case for file names: `json-diff.toit`, `pod-registry.toit`, `server-config.toit` -- Test files use suffix `-test.toit` or `-test-slow.toit`: `channel-test.toit`, `cmd-pod-delete-test.toit` -- No file extension variations; all source files are `.toit` - -**Functions:** -- Kebab-case for function names: `read-all`, `test-open`, `create-pods`, `ensure-authenticated` -- Helper/private functions use underscore suffix: `compute-cache-key_`, `install-root-certificates_`, `generate-envelope-path_` -- Test functions use `test-` prefix: `test-open`, `test-send`, `test-simple`, `test-neutering` -- Command handler functions use verb prefix: `sign-up`, `sign-in`, `create-auth-commands`, `ensure-available-artemis-service` - -**Variables:** -- Kebab-case for local variables and parameters: `tmp-dir`, `spec-ids`, `fleet-root`, `organization-id` -- Private instance fields use underscore suffix: `tmp-dir_`, `cache-key_`, `test-ui_`, `ders-already-installed_` -- Constants use SCREAMING-KEBAB-CASE: `DEFAULT-CAPACITY`, `MAGIC-NAME_`, `TEST-ORGANIZATION-UUID` -- UUID variables use `-id` or `-uuid` suffix: `organization-id`, `device-id`, `TEST-ORGANIZATION-UUID` - -**Types:** -- PascalCase for class names: `TestExit`, `TestPrinter`, `TestHumanPrinter`, `TestJsonPrinter`, `TestUi`, `TestCli` -- PascalCase for interface names: `Authenticatable`, `BrokerCli`, `ServerConfig` -- Private class fields use underscore suffix: `test-ui_`, `json_`, `quiet_`, `name_` - -## Code Style - -**Formatting:** -- No explicit formatter detected (no .prettierrc file) -- Consistent indentation of 2 spaces observed throughout codebase -- Line continuations use natural indentation -- Method/function definitions on single line with parameters indented on next lines - -**Linting:** -- No explicit linter configuration files detected (no eslint or similar) -- Code follows conventional Toit patterns with consistent style - -## Import Organization - -**Order:** -1. Standard library imports (system, core) -2. External package imports (cli, encoding, crypto, http, etc.) -3. Relative imports from parent packages (`..` imports) -4. Relative imports from sibling packages (`.` imports) -5. Export declarations - -**Examples from codebase:** -```toit -// File: src/cli/cli.toit -import certificate-roots -import cli show * -import core as core -import host.pipe show stderr -import io - -import .cmds.auth -import .cmds.config -import .cmds.device - -import ..shared.version -``` - -```toit -// File: src/cli/brokers/broker.toit -import cli show Cli -import host.file -import encoding.json -import net -import uuid show Uuid - -import ..auth -import ..config -import .supabase -import .http.base -``` - -**Path Aliases:** -- Direct imports of modules by name without aliases typically -- Aliasing used when name conflicts or for clarity: `import encoding.json as json-encoding` -- Re-exports using `show *` when module provides public API - -## Error Handling - -**Patterns:** -- Throw string literals with error descriptions: `throw "Unknown broker type"` -- Use assert for invariant checks: `assert: root-cmd.check; true` -- Try-finally blocks for cleanup operations: - ```toit - try: - block.call tmp-dir - finally: - directory.rmdir --recursive tmp-dir - ``` -- Try blocks with exception unwinding for control flow: - ```toit - exception = catch --unwind=(: not expect-exit-1 or (not allow-exception and it is not TestExit)): - artemis-pkg.main args --cli=run-cli - ``` -- Null checks using `if not var_:` pattern - -## Logging - -**Framework:** `log` standard library module used for structured logging - -**Patterns:** -- Logging is sparse in source code, mainly used in service/device code -- No universal logging pattern enforced across codebase -- Print statements used for CLI output via `cli.ui.emit` or `core.print` -- Test output via stdout/stderr captured in TestUi class - -## Comments - -**When to Comment:** -- Comments document non-obvious behavior and design decisions -- Interfaces and public functions have documentation comments -- Complex logic has inline comments explaining intent -- TODOs marked with `// TODO(author):` pattern - -**JSDoc/TSDoc:** -- Not applicable to Toit language -- Block comments using `/**` and `*/` for documentation: - ```toit - /** - Responsible for allowing the Artemis CLI to talk to Artemis services on devices. - */ - interface BrokerCli implements Authenticatable: - ``` -- Parameter documentation in comments: - ```toit - /** - The block is called with a $DeviceDetailed as argument: - The block must return a new goal state which replaces the actual goal state. - */ - update-goal --device-id/Uuid [block] -> none - ``` - -## Function Design - -**Size:** Functions are typically 5-50 lines, with test functions often longer due to test setup and assertions - -**Parameters:** -- Named parameters using `--parameter-name/type` syntax -- Block parameters with `[block]` or `[block/block-type]` syntax -- Required parameters without defaults -- Optional parameters with `?` type modifier: `--email/string?` - -**Return Values:** -- Explicit return types in function signatures: `-> string`, `-> List`, `-> none`, `-> bool` -- Functions return values implicitly (last expression) -- Null returns for operations that complete without producing values - -**Example patterns:** -```toit -// Function with named parameters and block -update-goal --device-id/Uuid [block] -> none - -// Function with optional parameters -constructor --quiet/bool=true --json/bool=false - -// Function with list parameters -update-goals --device-ids/List --goals/List -> none - -// Helper function returning computed value -cache-key -> string: - if not cache-key_: - cache-key_ = base64-lib.encode --url-mode (sha1.sha1 compute-cache-key_) - return cache-key_ -``` - -## Module Design - -**Exports:** -- Explicit export statements: `export Device` -- Modules export types, interfaces, and top-level functions -- Private implementation details use underscore suffix convention - -**Barrel Files:** -- Not commonly used; imports are specific and direct -- Relative imports reference exact modules needed - ---- - -*Convention analysis: 2026-03-15* diff --git a/.planning/codebase/INTEGRATIONS.md b/.planning/codebase/INTEGRATIONS.md deleted file mode 100644 index 0dc4eb35..00000000 --- a/.planning/codebase/INTEGRATIONS.md +++ /dev/null @@ -1,170 +0,0 @@ -# External Integrations - -**Analysis Date:** 2026-03-15 - -## APIs & External Services - -**Supabase (Primary Backend):** -- Service: Supabase Backend-as-a-Service (PostgREST API, Auth, Storage) - - SDK/Client: toit-supabase ^0.3.1 (`github.com/toitware/toit-supabase`) - - Authentication: JWT tokens with anon key - - Environment variables: `SUPABASE_URL`, `SUPABASE_ANON_KEY` - - Configuration: `ServerConfigSupabase` class in `src/shared/server-config.toit` - - Usage locations: - - `src/cli/brokers/supabase/supabase.toit` - CLI broker implementation - - `src/cli/artemis_servers/supabase/supabase.toit` - Artemis server API client - - `src/cli/utils/supabase.toit` - Utility functions - -**HTTP Brokers (Alternative Backend):** -- Service: Custom Toit HTTP broker protocol - - SDK/Client: pkg-http ^2.11.0 (`github.com/toitlang/pkg-http`) - - Configuration: `ServerConfigHttp` class in `src/shared/server-config.toit` - - Usage locations: - - `src/cli/brokers/http/base.toit` - HTTP broker implementation - - `src/cli/artemis_servers/http/base.toit` - HTTP server client - - `src/service/brokers/http` - Device-side HTTP broker - -**NTP Time Service:** -- Service: Network Time Protocol for system time synchronization - - SDK/Client: pkg-ntp ^1.1.0 (`github.com/toitlang/pkg-ntp`) - - Purpose: Ensuring accurate time on devices - - Usage: Time-critical device operations - -## Data Storage - -**Databases:** -- Provider: Supabase (PostgreSQL 15) - - Connection: Supabase REST API over HTTP/HTTPS - - Client: toit-supabase ^0.3.1 - - Schemas exposed: - - `public` - Main public schema - - `storage` - File storage metadata - - `graphql_public` - GraphQL API schema - - `toit_artemis` - Artemis-specific schema - - Configuration file: `supabase_artemis/supabase/config.toml` - - Migrations: Located in `supabase_artemis/supabase/migrations/` - - Seed data: `supabase_artemis/supabase/seed.sql` - -**File Storage:** -- Supabase Storage buckets - - Access: Through Supabase REST API - - Size limit: 50 MiB per file - - Support for public and private buckets - - Referenced in Edge Function `b/index.ts` for firmware and image uploads/downloads - -**Caching:** -- Configuration caching: Local filesystem in `~/.cache/artemis/` (device-specific) -- Server connection caching uses cache keys based on host/port/path hashing - -## Authentication & Identity - -**Auth Provider:** -- Supabase Auth (Email/Password and OAuth providers) - - Implementation: Supabase authentication service - - Methods supported: - - Email/password signup and sign-in (`sign-up`, `sign-in` with credentials) - - OAuth provider sign-in (Google, GitHub, etc.) - via Supabase OAuth providers - - JWT token-based authentication with 1-hour expiry (configurable) - - Session management: Browser-based with redirect URL handling - - OAuth Redirect URL: `https://toit.io/auth` for production - -**Authorization:** -- Row-Level Security (RLS) policies in PostgreSQL -- Organization-based access control -- Device ownership verification through organization membership -- Role-based access: User, Member, and organization-specific roles - -## Monitoring & Observability - -**Error Tracking:** -- Not detected as a dedicated service integration -- Error handling through application logging - -**Logs:** -- Console logging via `import log` in Toit code -- Supabase Edge Function logs available through Supabase dashboard -- Local development logs directed to console - -## CI/CD & Deployment - -**Hosting:** -- Supabase Cloud (primary) - API and database hosting -- Self-hosted option supported for local development and testing -- Public broker instance: `supabase.co` domain (e.g., `ezxwpyeoypvnnldpdotx.supabase.co`) -- Private Artemis instance: `artemis-api.toit.io` - -**Deployment:** -- Edge Functions deployed via Supabase CLI: `supabase functions deploy` -- Database migrations via Supabase CLI: `supabase db push` -- Service images managed through `tools/service_image_uploader/uploader.toit` - -**Build/Test Infrastructure:** -- Local Supabase instances: `make start-supabase` / `make start-supabase-no-config` -- Test targets: - - Unit tests: `make test` - - Serial tests: `make test-serial` - - Supabase integration tests: `make test-supabase` - -## Environment Configuration - -**Required env vars:** -- `SUPABASE_URL` - Supabase project URL (from config) -- `SUPABASE_ANON_KEY` - Supabase anonymous key for public access -- `ARTEMIS_CONFIG` - Path to local artemis configuration (default: `~/.config/artemis-dev/config`) -- `TOIT_PKG_AUTO_SYNC` - Whether to auto-sync packages (CMake option, ON by default) -- `DEFAULT_SDK_VERSION` - SDK version for compilation -- `ARTEMIS_GIT_VERSION` - Version string for builds (auto-computed from git) - -**Test Environment Vars:** -- `ARTEMIS_HOST` - Artemis server host (production: `artemis-api.toit.io`) -- `ARTEMIS_ANON` - Anonymous JWT token for Artemis server -- `ARTEMIS_TEST_HOST` - Test Supabase instance host -- `ARTEMIS_TEST_ANON` - Test Supabase instance anonymous key - -**Secrets location:** -- `.env` files (not committed) - Development secrets -- Supabase project settings - Production secrets -- Makefile JWT tokens (test credentials only, not for production) - -## Webhooks & Callbacks - -**Incoming:** -- Supabase Edge Function endpoint: `/functions/v1/b` - - POST endpoint receiving binary-encoded commands - - Commands handled: - - Device goal updates (`COMMAND_UPDATE_GOAL_`) - - Device state reporting (`COMMAND_REPORT_STATE_`) - - Event reporting (`COMMAND_REPORT_EVENT_`) - - Firmware/image downloads (`COMMAND_DOWNLOAD_`, `COMMAND_DOWNLOAD_PRIVATE_`) - - Pod registry operations (10+ commands for pod management) - - Authentication: JWT Bearer token in Authorization header or anon key - -**Outgoing:** -- Device -> Broker API calls for: - - State synchronization - - Event reporting - - Goal fetching - - Firmware/image downloading -- No webhook-style callbacks to external systems detected - -## Data Formats - -**Communication Protocols:** -- Binary protocol for device-broker communication (command-based) - - Command byte followed by JSON or binary payload - - Handled in Edge Function: `supabase_artemis/supabase/functions/b/index.ts` - - HTTP request body contains binary command + payload - -**REST API:** -- PostgREST API for database table operations -- supabase.rpc() for stored procedure calls -- Supabase Storage API for file operations - -**Serialization:** -- JSON for REST API payloads -- Binary/ArrayBuffer for firmware and image data -- Base64 encoding for certificate storage and transmission - ---- - -*Integration audit: 2026-03-15* diff --git a/.planning/codebase/STACK.md b/.planning/codebase/STACK.md deleted file mode 100644 index c7ae8ff4..00000000 --- a/.planning/codebase/STACK.md +++ /dev/null @@ -1,108 +0,0 @@ -# Technology Stack - -**Analysis Date:** 2026-03-15 - -## Languages - -**Primary:** -- Toit v2.0.0-alpha.190 (development SDK version) - Core application logic, CLI, service, brokers, and tests -- TypeScript - Supabase Edge Functions for broker API endpoints - -**Secondary:** -- SQL - Database schemas and migrations for Supabase PostgreSQL -- CMake - Build system configuration -- Bash - Build scripts and development tooling - -## Runtime - -**Environment:** -- Toit Runtime (custom managed runtime for IoT/embedded systems) -- Deno - Runtime for Supabase Edge Functions (TypeScript) -- PostgreSQL 15 - Database backend - -**Package Manager:** -- Toit Package Manager (toit pkg) -- Lockfile: `package.lock` (present in root) - -## Frameworks - -**Core:** -- Toit Artemis Framework ^0.1.1 - Device and broker management system -- Toit CLI Framework ^2.6.0 - Command-line interface building -- Toit Supabase ^0.3.1 - Supabase client and authentication - -**Build/Dev:** -- CMake 3.23+ - Build orchestration -- Ninja - Build executor (used by CMake) -- Supabase CLI - Local development database and edge function management - -**Protocols & Networking:** -- Toit HTTP ^2.11.0 - HTTP client and server -- Toit NTP ^1.1.0 - Network Time Protocol for time synchronization -- TLS/HTTPS - Secure communication with root certificate support - -## Key Dependencies - -**Critical:** -- toit-supabase ^0.3.1 - Supabase client library with authentication and database access -- toit-artemis ^0.1.1 - Core Artemis device management library -- pkg-http ^2.11.0 - HTTP transport layer for broker communication -- toit-cert-roots ^1.11.0 - Root certificates for TLS/HTTPS connections - -**Infrastructure:** -- pkg-cli ^2.6.0 - CLI framework for command-line tools -- pkg-fs ^2.3.1 - Filesystem access -- pkg-host ^1.16.2 - Host system integration -- toit-watchdog ^1.4.1 - Watchdog timer functionality -- toit-partition-table-esp32 ^1.4.0 - ESP32 partition table management -- toit-semver ^1.1.0 - Semantic versioning utilities -- pkg-ar ^1.4.1 - Archive handling -- artemis-pkg-copy (local path) - Local artemis package copy - -**Development/Tooling:** -- snapshot (local path: tools/snapshot) - Snapshot building utility - -## Configuration - -**Environment:** -- Toit configuration stored in `~/.config/artemis-dev/config` (configurable via `ARTEMIS_CONFIG`) -- Broker credentials and configuration stored in local config files -- Support for both HTTP and Supabase broker configurations - -**Build:** -- `CMakeLists.txt` - Main build configuration -- `package.yaml` - Root package manifest with dependencies -- `Makefile` - Development workflow automation -- `supabase_artemis/supabase/config.toml` - Local Supabase configuration -- `public/supabase_broker/supabase/config.toml` - Broker Supabase configuration - -**Build Configuration Files:** -- PostgreSQL version pinned to 15 in `supabase_artemis/supabase/config.toml` -- API schemas: `public`, `storage`, `graphql_public`, `toit_artemis` -- Storage limit: 50MiB per file -- JWT expiry: 3600 seconds (1 hour) - -## Platform Requirements - -**Development:** -- Toit executable (toit CLI) -- CMake 3.23+ -- Ninja build system -- Supabase CLI (for local database development) -- Docker (for Supabase local development) -- GNU Make -- Git - -**Supported Hardware:** -- ESP32 (primary embedded target) -- ESP32-QEMU (for testing) -- Host systems (x86_64, ARM) - -**Production:** -- Supabase hosting infrastructure (cloud or self-hosted) -- PostgreSQL 15+ database -- Deno runtime (for Edge Functions) - ---- - -*Stack analysis: 2026-03-15* diff --git a/.planning/codebase/STRUCTURE.md b/.planning/codebase/STRUCTURE.md deleted file mode 100644 index 2aff4e65..00000000 --- a/.planning/codebase/STRUCTURE.md +++ /dev/null @@ -1,219 +0,0 @@ -# Codebase Structure - -**Analysis Date:** 2026-03-15 - -## Directory Layout - -``` -artemis/ -├── src/ # Main source code (Toit) -│ ├── cli/ # CLI tool -│ │ ├── artemis.toit # Main entry point -│ │ ├── cli.toit # CLI root command setup -│ │ ├── cmds/ # Command implementations -│ │ ├── artemis_servers/ # Artemis server API client -│ │ ├── brokers/ # Broker client implementations -│ │ ├── utils/ # CLI utilities -│ │ └── *.toit # Core CLI modules (config, device, fleet, pod, etc.) -│ ├── service/ # Device-side runtime service -│ │ ├── service.toit # Main service entry point -│ │ ├── scheduler.toit # Job scheduler -│ │ ├── containers.toit # Container management -│ │ ├── brokers/ # Broker connection implementations -│ │ ├── jobs.toit # Base job abstraction -│ │ ├── synchronize.toit # State synchronization job -│ │ ├── firmware-update.toit # Firmware update logic -│ │ ├── device.toit # Device state representation -│ │ ├── storage.toit # Persistent storage -│ │ └── *.toit # Supporting modules -│ └── shared/ # Shared code (CLI and service) -│ ├── version.toit # Version constants -│ ├── constants.toit # Global constants (commands) -│ ├── server-config.toit # Broker configuration -│ ├── json-diff.toit # JSON difference computation -│ └── utils/ # Shared utilities -├── tests/ # Test files -│ ├── *-test.toit # Individual test files -│ ├── gold/ # Expected output files for tests -│ └── spec_extends_tests/ # Test extensions -├── supabase_artemis/ # Supabase edge functions and migrations -│ └── supabase/ -│ ├── functions/ # Edge function code -│ ├── migrations/ # Database migrations -│ └── snippets/ # Code snippets -├── public/ # Public documentation and schemas -│ ├── docs/ # Documentation -│ ├── examples/ # Example configurations -│ └── schemas/ # JSON schemas for pod specs -├── tools/ # Utility tools -│ ├── http_servers/ # Test HTTP servers -│ ├── service_image_uploader/ # Upload service images -│ ├── snapshot/ # Snapshot tool -│ ├── lan_ip/ # LAN IP discovery -│ └── windows_installer/ # Windows installer -├── auth/ # Authentication utilities -├── benchmarks/ # Performance benchmarks -├── recovery/ # Recovery tools -├── artemis-pkg-copy/ # Temporary copy of artemis package -├── build/ # CMake build directory (generated) -├── .github/ # GitHub workflows and configuration -└── .planning/ # GSD planning documents -``` - -## Directory Purposes - -**src/cli:** -- Purpose: Command-line interface for managing devices, fleets, pods, and organizations -- Contains: Entry point, command handlers, broker clients, configuration management -- Key files: `artemis.toit` (CLI orchestrator), `cli.toit` (command structure), `cmds/` (individual commands) - -**src/service:** -- Purpose: Device-side runtime that manages containers and synchronization -- Contains: Scheduler, job implementations, container/firmware management, broker connections -- Key files: `service.toit` (entry point), `scheduler.toit` (job orchestration), `containers.toit` (app management), `synchronize.toit` (state sync) - -**src/shared:** -- Purpose: Code shared between CLI and service layers -- Contains: Version strings, command constants, server configuration, utilities -- Key files: `version.toit` (versioning), `constants.toit` (command codes), `server-config.toit` (broker config) - -**src/cli/brokers:** -- Purpose: CLI-side broker implementations for communicating with servers -- Contains: HTTP broker, Supabase broker, request/response handling -- Key files: `broker.toit` (interface), `http/base.toit` (HTTP implementation), `supabase/supabase.toit` (Supabase implementation) - -**src/service/brokers:** -- Purpose: Device-side broker implementations for communicating with management servers -- Contains: Connection handling, goal state fetching, state reporting -- Key files: `broker.toit` (interface), `http/http.toit` (HTTP implementation) - -**tests:** -- Purpose: Test suite for CLI commands and core functionality -- Contains: Command output tests, synchronization tests, JSON diff tests -- Key files: `*-test.toit` (individual tests), `gold/` (expected outputs for verification) - -**supabase_artemis:** -- Purpose: Supabase backend infrastructure (database and edge functions) -- Contains: Database schema migrations, serverless functions for API endpoints -- Key files: `migrations/` (database schema), `functions/` (API endpoints) - -## Key File Locations - -**Entry Points:** -- `src/cli/artemis.toit`: CLI main entry point and version handling -- `src/service/service.toit`: Device service entry point, exports `run-artemis` function -- `src/cli/cli.toit`: CLI command structure and routing - -**Configuration:** -- `src/shared/server-config.toit`: Broker connection configuration (HTTP/Supabase) -- `src/cli/config.toit`: User configuration management (profiles, brokers, cache) -- `package.yaml`: Toit package dependencies - -**Core Logic:** -- `src/service/scheduler.toit`: Job scheduler that drives device operation -- `src/service/containers.toit`: Container lifecycle management -- `src/service/synchronize.toit`: State synchronization with broker -- `src/cli/artemis.toit`: Device manager from CLI perspective -- `src/cli/fleet.toit`: Fleet management operations - -**Testing:** -- `tests/cmd-fleet-status-test.toit`: Fleet command test -- `tests/synchronizer.toit`: Synchronization logic test -- `tests/gold/`: Expected command output files - -## Naming Conventions - -**Files:** -- Toit source files: `lowercase-with-hyphens.toit` -- Executable or tool files: `lowercase-with-hyphens` (no extension) -- Test files: `*-test.toit` or `*_test.toit` (ending in -test or _test) -- Generated files: `*.generated.toit` or `version.toit.in` (template) - -**Directories:** -- Module directories: `lowercase-with-hyphens/` (e.g., `artemis_servers`, `brokers`) -- Command implementations: Inside `cmds/` with command name (e.g., `device.toit`, `fleet.toit`) -- Test support: `gold/` for expected outputs, `spec_extends_tests/` for test utilities - -**Classes:** -- PascalCase for classes: `ContainerManager`, `SynchronizeJob`, `ArtemisServerCli` -- Abstract base classes: `Job`, `TaskJob`, `BrokerConnection`, `BrokerService` - -**Functions/Methods:** -- snake-case with hyphens: `run-artemis`, `connect-network_`, `ensure-authenticated` -- Private methods: trailing underscore `_` before method name (e.g., `connected-artemis-server_`) -- Getter methods: simple names (e.g., `runlevel`, `is-running`) - -**Constants:** -- ALL-CAPS with hyphens: `RUNLEVEL-NORMAL`, `STATE-SYNCHRONIZED`, `COMMAND-UPDATE-GOALS_` -- Maps of constants: `ARTEMIS-COMMAND-TO-STRING`, `BROKER-COMMAND-TO-STRING` - -**Variables:** -- snake-case with hyphens: `max-offline-time`, `job-states`, `images_` -- Field names: lowercase: `name`, `id`, `tasks_` - -## Where to Add New Code - -**New CLI Command:** -1. Create handler in `src/cli/cmds/` (e.g., `src/cli/cmds/new-command.toit`) -2. Implement `create-new-command-commands() -> List` function -3. Import and call from `src/cli/cli.toit` in main function -4. Add command tests to `tests/` with corresponding gold output in `tests/gold/` - -**New Service Job:** -1. Create in `src/service/` (e.g., `src/service/new-job.toit`) -2. Extend `Job` or `TaskJob` abstract class from `src/service/jobs.toit` -3. Implement required methods: `is-running`, `schedule`, `start`, `stop` -4. Import and add to scheduler via `scheduler.add-job` in `src/service/service.toit` -5. Store/restore state via `scheduler-state` property if needed - -**New Broker Implementation:** -- CLI side: Create in `src/cli/brokers/` (e.g., `src/cli/brokers/custom/custom.toit`) -- Service side: Create in `src/service/brokers/` (e.g., `src/service/brokers/custom/custom.toit`) -- Implement `BrokerConnection` interface -- Add factory method in broker constructor in respective layer -- Update configuration to support new broker type - -**Shared Utilities:** -- General utilities: `src/shared/utils/utils.toit` or `src/shared/utils/specific-util.toit` -- Constants: Add to `src/shared/constants.toit` -- Protocols: Define interfaces in appropriate module or new dedicated file - -**Tests:** -- New test file: `tests/my-feature-test.toit` -- Expected output: `tests/gold/cmd-my-feature-test/` (directory with output files) -- Import test utilities from existing test files as reference - -## Special Directories - -**build/** -- Purpose: CMake build output directory (generated at build time) -- Generated: Yes -- Committed: No (in .gitignore) -- Contains: Compiled binaries, build artifacts, test executables - -**tests/gold/** -- Purpose: Expected output files for command tests (golden files) -- Generated: No (manually curated) -- Committed: Yes -- Usage: Tests compare actual output against files in this directory - -**supabase_artemis/supabase/** -- Purpose: Supabase infrastructure as code -- Generated: No -- Committed: Yes -- Structure: `migrations/` (numbered SQL files), `functions/` (edge functions) - -**.packages/** -- Purpose: Cached package dependencies (managed by Toit package manager) -- Generated: Yes (via `toit pkg` commands) -- Committed: No (in .gitignore) - -**artemis-pkg-copy/** -- Purpose: Temporary copy of the Artemis package API from public repository -- Generated: No (static copy) -- Committed: Yes -- Status: Marked for deletion when public package API stabilizes - ---- - -*Structure analysis: 2026-03-15* diff --git a/.planning/codebase/TESTING.md b/.planning/codebase/TESTING.md deleted file mode 100644 index 80d278bd..00000000 --- a/.planning/codebase/TESTING.md +++ /dev/null @@ -1,311 +0,0 @@ -# Testing Patterns - -**Analysis Date:** 2026-03-15 - -## Test Framework - -**Runner:** -- CMake-based test runner using ctest -- Toit language runtime executes tests via `toit run` -- Config: `/home/flo/work/artemis/tests/CMakeLists.txt` - -**Assertion Library:** -- `expect` standard library module for assertions -- Provides functions like `expect-equals`, `expect-null`, `expect-throw`, `expect-bytes-equal`, `expect-identical` - -**Run Commands:** -```bash -make test # Run all tests -make test-serial # Run serial tests only -make test-supabase # Run Supabase-specific tests -``` - -CMake test execution: -```bash -cd build && ninja check # Run all tests -cd build && ninja check_serial # Serial tests -cd build && ninja check_supabase # Supabase tests -``` - -## Test File Organization - -**Location:** -- Tests located in `/home/flo/work/artemis/tests/` directory -- Co-located with main source but in separate `tests` directory -- Test files live at same hierarchy level as source modules - -**Naming:** -- Pattern: `*-test.toit` for regular tests -- Pattern: `*-test-slow.toit` for slow/long-running tests -- Pattern: `serial-*` for tests that must run sequentially -- Examples: `channel-test.toit`, `cmd-pod-delete-test.toit`, `supabase-artemis-broker-policies-test.toit` - -**Structure:** -``` -tests/ -├── *-test.toit # Standard unit/integration tests -├── *-test-slow.toit # Slow tests with longer timeouts -├── serial-*.toit # Serial tests (resource locks) -├── gold/ # Gold files for CLI output comparison -├── utils.toit # Test utilities and fixtures -├── CMakeLists.txt # Test configuration -└── .packages/ # Vendored dependencies -``` - -## Test Structure - -**Suite Organization:** - -Tests use a `main` entry point that spawns task(s) and may install/uninstall service providers: - -```toit -// File: tests/channel-test.toit -main: - provider := TestServiceProvider - provider.install - spawn:: test - provider.uninstall --wait - -test: - test-open "small" 32 * 1024 - test-open "medium" 64 * 1024 - 3.repeat: test-simple "fisk" - test-neutering "hest" - test-full "fisk" -``` - -**Patterns:** - -1. **Setup/Teardown:** - - `main` function handles initialization - - `spawn::` for concurrent test tasks - - Service provider `install`/`uninstall` for resource management - - Try-finally blocks for cleanup: - ```toit - try: - // test code - list.do: channel.send it - finally: - channel.close - ``` - -2. **Assertion Pattern:** - ```toit - expect-equals expected actual - expect-null value - expect-bytes-equal expected-bytes actual-bytes - expect-throw "ERROR_MESSAGE": function-call - expect-identical obj1 obj2 - ``` - -3. **CLI Test Pattern (from utils.toit):** - ```toit - run-gold test-name description args --ignore-spacing=false --expect-exit-1=false - ``` - Compares CLI output against gold files stored in `gold/` directory. - -4. **Test Helpers:** - ```toit - with-tmp-directory [block] // Create temp directory for test - with-tmp-config-cli [block] // Create test CLI with config - with-fleet [block] // Fleet testing context - with-server [block] // Server testing context - ``` - -## Mocking - -**Framework:** Manual mocking using test doubles and custom implementations - -**Patterns:** - -1. **Test Doubles:** - ```toit - // From tests/utils.toit - class TestExit: - - class TestPrinter extends cli-pkg.Printer: - print_ str/string: - test-ui_.stdout += "$str\n" - - class TestUi extends cli-pkg.Ui: - stdout/string := "" - stderr/string := "" - quiet_/bool - ``` - -2. **Service Provider Mocking:** - ```toit - provider := TestServiceProvider - provider.install - spawn:: test - provider.uninstall --wait - ``` - -3. **Capturing Output:** - ```toit - ui := TestUi --quiet=quiet --json=json - run-cli := cli.with --ui=ui - // Run code that uses ui - output := ui.stdout - ``` - -**What to Mock:** -- CLI output and printing (use TestUi) -- Service providers for isolated component testing -- File system operations (use temporary directories) -- External HTTP/Supabase servers (pre-configured with test fixtures) - -**What NOT to Mock:** -- Core language features -- Standard library functions -- Data structures (ByteArray, List, Map) -- File I/O when actual files needed for integration tests - -## Fixtures and Factories - -**Test Data:** - -Constants and fixture creation in `tests/utils.toit`: - -```toit -/** test@example.com is an admin of the $TEST-ORGANIZATION-UUID. */ -TEST-EXAMPLE-COM-EMAIL ::= "test@example.com" -TEST-EXAMPLE-COM-PASSWORD ::= "password" -TEST-EXAMPLE-COM-UUID ::= Uuid.parse "f76629c5-a070-4bbc-9918-64beaea48848" -TEST-EXAMPLE-COM-NAME ::= "Test User" - -TEST-ORGANIZATION-NAME ::= "Test Organization" -TEST-ORGANIZATION-UUID ::= Uuid.parse "4b6d9e35-cae9-44c0-8da0-6b0e485987e2" - -TEST-DEVICE-UUID ::= Uuid.parse "eb45c662-356c-4bea-ad8c-ede37688fddf" -TEST-POD-UUID ::= Uuid.parse "0e29c450-f802-49cc-b695-c5add71fdac3" -``` - -**Factory Functions:** - -```toit -// Create test CLI with temporary config -with-tmp-config-cli [block]: - with-tmp-directory: | directory | - config-path := "$directory/config" - app-name := "artemis-test" - config := cli-pkg.Config --app-name=app-name --path=config-path --data={:} - cli := cli-pkg.Cli app-name --config=config - block.call cli - -// Create pods for testing -create-pods name/string fleet/TestFleet --count/int -> List: - spec := """ - { "$schema": "https://toit.io/...", "name": "$name", ... } - """ - spec-path := "$fleet.fleet-dir/$(name).json" - write-blob-to-file spec-path spec - count.repeat: - fleet.run ["pod", "upload", spec-path] - return [description-id, spec-ids] -``` - -**Location:** -- Test utilities and fixtures in `tests/utils.toit` -- Test data constants defined at module level -- Fleet testing utilities in TestFleet class - -## Coverage - -**Requirements:** Not detected; no coverage enforcement found in configuration - -**View Coverage:** Not applicable; Toit test framework does not expose coverage metrics - -## Test Types - -**Unit Tests:** -- Scope: Individual functions and small modules -- Approach: Direct function calls with assertions -- Example: `test-open`, `test-send` in `channel-test.toit` -- Pattern: Parameterized helpers that run multiple scenarios - -**Integration Tests:** -- Scope: Multiple components interacting (CLI + server + broker) -- Approach: Full command execution with temporary servers -- Example: `cmd-pod-delete-test.toit`, broker policy tests -- Pattern: Use `with-fleet`, `with-server` context managers -- Configuration via `// ARTEMIS_TEST_FLAGS:` comments in test file - -**E2E Tests:** -- Framework: CMake-based with test flags for different server configurations -- Patterns: - - `// ARTEMIS_TEST_FLAGS: ARTEMIS` - requires Artemis server - - `// ARTEMIS_TEST_FLAGS: BROKER` - requires broker - - Multiple test variants per file with different flags -- Execution: Tests run with different server/broker combinations via CMake - -**Resource Locks:** -- Tests that need exclusive resources use CMake resource locks -- Lock names: `artemis_server`, `broker`, `artemis_broker`, `serial` -- Supabase tests automatically get locks when they use supabase flags - -## Common Patterns - -**Async Testing:** - -Concurrency via `spawn::` for parallel task execution: - -```toit -main: - provider.install - spawn:: test // Spawn test as concurrent task - provider.uninstall --wait -``` - -**Error Testing:** - -```toit -// Test that error is thrown with specific message -expect-throw "OUT_OF_RANGE: 209 > 200": - channel.acknowledge 209 - -// Test that specific exception type is thrown -expect-throw "ALREADY_IN_USE": - // code that throws -``` - -**Parameterized Tests:** - -Test helper functions called with different parameters: - -```toit -test-open "small" 32 * 1024 -test-open "medium" 64 * 1024 -test-open "large" 512 * 1024 - -test-neutering "hest" -[1, 2, 5, 127, 128, 129, 512, 1024, 3000].do: - test-neutering topic it -``` - -**Gold File Tests:** - -CLI output comparison against golden files: - -```toit -run-gold "BAA-delete-pod-revision" - "Delete a pod by revision" - [ - "pod", "delete", "$pod1-name#2" - ] - -// Compares output to gold/gold-dir-name/BAA-delete-pod-revision.txt -// Updates gold files if UPDATE_GOLD=1 environment variable set -``` - -**Test Timeouts:** - -Set via CMake based on test pattern: -- Default: `200` seconds -- Slow tests (`*-test-slow.toit`): `300` seconds -- Serial tests (`serial-*`): `1000` seconds -- QEMU tests (`qemu-*`): `300` seconds - ---- - -*Testing analysis: 2026-03-15*