Skip to content

TODO-mallet.md

Latest commit

 

History

History
267 lines (232 loc) · 10.2 KB

TODO-mallet.md

File metadata and controls

267 lines (232 loc) · 10.2 KB
Raw

TODO: Mallet Implementation

Implement Mallet - a Podman-based Tekton executor as part of the chisel monorepo.

Updated Plan: ~/.config/claude/history/research/2026-02/PLAN-mallet-implementation-v2.md Original Research: ~/.config/claude/history/research/2026-02/RESEARCH-podman-migration.md

Overview

Transform chisel into a monorepo supporting two backends:

  • Chisel - Dagger backend (existing, ~8,135 LOC)
  • Mallet - Podman backend (new, ~2,850 LOC to add)

Current Code Reusability: ~80% of codebase (4,810 LOC) is backend-agnostic Must Reimplement: ~20% of codebase (527 LOC) is Dagger-specific

Implementation Phases

Phase 1: Backend Abstraction Layer ✅

Timeline: Week 1 Status: COMPLETE (PR #14 merged 2026-02-05) Actual LOC: +811/-164

  • Create pkg/backend/ package
    • Define Backend interface in backend.go
    • Define StepRequest, StepResult, SidecarRequest, WorkspaceMount types
    • Add comprehensive godoc comments
  • Create pkg/orchestrator/ package (~600 LOC moved)
    • Move scheduler, matrix, when, retry, timeout, steptemplate, volumes, results
    • Export all functions for backend use
    • Update package declarations and imports
  • Create pkg/backend/dagger/ package
    • Refactor executor.go to implement Backend interface
    • Move sidecar.go
    • All interface methods implemented
  • Update Chisel CLI
    • Update imports to use new backend structure
  • Verified: All 60+ tests pass, all examples work, zero regressions

Phase 2: Mallet CLI Stub ✅

Timeline: Week 2 Status: COMPLETE (2026-02-06) Actual LOC: +548 (4 commits)

  • Create cmd/mallet/ directory
    • main.go, root.go, run.go, workspace.go
    • Uses Podman backend, shares parser/types/ui with chisel
  • Create stub pkg/backend/podman/ package
    • TDD: Tests written first (8 tests)
    • All interface methods return ErrNotImplemented
  • Create Makefile with build targets
    • build, build-chisel, build-mallet, test, lint, clean, install, check
  • Update .goreleaser.yaml for multi-binary builds
    • Separate builds and archives for chisel and mallet
  • Update root README.md to mention both tools
  • Create cmd/mallet/README.md
  • Verified: Both binaries compile, mallet fails gracefully with "not implemented"
    • chisel: 29MB, mallet: 15MB
    • Dry-run works for both
    • mallet execution: "podman backend not yet implemented"

Phase 3: Podman Core Implementation ✅

Timeline: Weeks 3-4 Status: COMPLETE (PR #16 merged 2026-02-06) Actual LOC: +1,333/-69

  • Add Podman dependencies (pure HTTP API, no CGO)
  • Implement pkg/backend/podman/client.go (~115 LOC)
    • Connection to Podman socket via HTTP
    • Auto-detect socket path (user vs system)
    • Service availability checks
    • Clear error messages if service not running
  • Implement pkg/backend/podman/container.go (~479 LOC)
    • Container creation via REST API
    • Container lifecycle: create → start → wait → logs → cleanup
    • Log streaming with multiplexed stdout/stderr parsing
    • Timeout handling with context cancellation
  • Implement pkg/backend/podman/executor.go (~108 LOC)
    • Build ContainerSpec from StepRequest
    • Execute and capture output
    • Return StepResult
    • Timeout support
  • Integration tests with real Podman (skip when unavailable)
  • Verified: Can execute single-step tasks, hello-pipelinerun works (~700ms)

Phase 4: Full Orchestration Integration ✅

Timeline: Week 5 Status: COMPLETE (PR #17, 2026-02-06) Actual LOC: ~270 LOC

  • Implement pkg/backend/podman/results.go (~50 LOC)
    • ReadResultFromPath and CollectResults helpers
    • Mount temp directory to /tekton/results
    • Collect results after container execution
  • Integration with orchestrator (~165 LOC changes)
    • Use BuildDAG and ExecuteParallel for parallel execution
    • Expand matrix tasks before building DAG
    • Evaluate when conditions for conditional execution
    • Pass results between dependent tasks
    • Execute finally tasks after main pipeline
  • Add parameter substitution (~60 LOC)
    • $(params.name) with string, array, object support
    • $(tasks.taskname.results.resultname) for result passing
    • $(workspaces.name.path) and $(context.*) variables
  • Verified: Full pipeline execution with result passing works

Phase 5: Sidecars & Advanced Features ✅

Timeline: Week 6 Status: COMPLETE (PR #18, 2026-02-06) Actual LOC: ~540 LOC

  • Implement pkg/backend/podman/sidecar.go (~285 LOC)
    • Pod operations (CreatePod, StartPod, StopPod, RemovePod)
    • CreateContainerInPod for running containers in pods
    • StartSidecar and StopSidecar implementation
    • Make container command optional (use image default)
  • Implement pkg/backend/podman/cleanup.go (~150 LOC)
    • ResourceTracker for tracking containers, pods, volumes
    • Thread-safe concurrent access
    • CleanupAll with fresh context for cleanup
    • RemoveVolume function
  • Test sidecars, cleanup
  • Verified: Sidecar lifecycle works, cleanup works

Phase 6: Polish, Documentation & Release ⏳

Timeline: Week 7 Status: In progress (2026-02-06) Estimated LOC: ~200 LOC

  • Bug fixes from example testing
    • Apply stepTemplate defaults to steps
    • Handle task-level volumes with emptyDir support
    • Variable substitution in workingDir
    • Override entrypoint for scripts (handle images with custom entrypoints)
    • Create workingDir if it doesn't exist
  • Example testing (13/13 passing)
    • hello-pipelinerun: PASS
    • params-pipelinerun: PASS (array param expansion as designed)
    • results-pipelinerun: PASS
    • parallel-pipelinerun: PASS
    • matrix-pipelinerun: PASS
    • when-pipelinerun: PASS
    • retry-pipelinerun: PASS
    • timeout-pipelinerun: PASS
    • steptemplate-pipelinerun: PASS
    • volumes-pipelinerun: PASS
    • verbose-pipelinerun: PASS
    • workspace-override-pipelinerun: PASS
    • sidecar-pipelinerun: PASS (pod-based execution with shared network)
  • Pod-based task execution (like Tekton)
    • Each task runs in its own Podman pod
    • Sidecars and steps share network namespace
    • Host aliases for sidecar hostname resolution (sidecars accessible by name)
  • Performance optimization
    • Connection pooling
    • Parallel container cleanup
    • Volume reuse strategies
  • Documentation
    • Update root README.md (overview, when to use each tool, installation)
    • Create cmd/mallet/README.md (Podman setup, troubleshooting)
    • Create docs/ARCHITECTURE.md (backend abstraction design)
    • Create docs/MIGRATION.md (skipped - README covers tool selection)
  • CI/CD updates
    • Update .github/workflows/ci.yaml (build/test both, mallet dry-run tests)
    • Update .github/workflows/release.yaml (verify builds, pinned actions)
    • .goreleaser.yaml already configured for multi-binary builds
  • Example updates
    • Add comments to all YAML files
    • Create examples/README.md
  • Final testing
    • All examples with both tools
    • Cross-platform testing
    • Load testing, memory leak testing
  • Verify: Mallet v0.1.0 ready for release

Success Criteria

Functional Requirements

  • Chisel continues working with zero regressions
  • All 20+ example PipelineRuns execute successfully with Mallet
  • Output matches Chisel (except execution timing)
  • No resource leaks (containers, pods, volumes cleaned up)
  • Graceful error handling and Ctrl+C cleanup

Performance Requirements

  • Startup time < 2 seconds (first run)
  • Execution time within 30% of Chisel for same workload
  • Memory usage < 500MB for typical pipeline

Code Quality Requirements

  • All packages have >70% test coverage
  • All linters pass (golangci-lint run)
  • All tests pass (go test ./...)
  • Code reuse ~80% between tools

User Experience Requirements

  • Clear installation instructions
  • Complete --help documentation
  • Actionable error messages
  • Examples run out of the box
  • Binary sizes: Chisel ~21MB, Mallet ~15MB

Quick Reference

Build:

make build          # Build both
make build-chisel   # Build chisel only
make build-mallet   # Build mallet only

Test:

./chisel run examples/simple/hello-pipelinerun.yaml
./mallet run examples/simple/hello-pipelinerun.yaml

Architecture:

pkg/
  types/           # Shared: Data structures
  parser/          # Shared: YAML parsing
  ui/              # Shared: Logging
  orchestrator/    # Shared: Orchestration logic
  backend/
    backend.go     # Interface
    dagger/        # Chisel implementation
    podman/        # Mallet implementation

cmd/
  chisel/          # Dagger backend CLI
  mallet/          # Podman backend CLI

Dependencies

New (Mallet Only)

  • github.com/containers/podman/v5/pkg/bindings (~3MB)
  • github.com/containers/common/libnetwork/types
  • github.com/containers/storage
  • github.com/opencontainers/runtime-spec/specs-go

Shared (Already Present)

  • dagger.io/dagger (Chisel only)
  • github.com/spf13/cobra
  • gopkg.in/yaml.v3
  • github.com/go-git/go-git/v5 (Git resolver)
  • github.com/google/go-containerregistry (Bundles resolver)

Timeline

Phase Duration LOC Deliverable
1. Backend Abstraction Week 1 ~800 Chisel refactored
2. Mallet CLI Stub Week 2 ~500 Mallet compiles
3. Podman Core Weeks 3-4 ~900 Basic execution
4. Orchestration Week 5 ~150 Full pipelines
5. Sidecars & Cleanup Week 6 ~300 Feature parity
6. Polish & Release Week 7 ~200 v0.1.0
Total 7 weeks ~2,850 Production ready

Notes

  • Updated Plan (Feb 5, 2026): ~/.config/claude/history/research/2026-02/PLAN-mallet-implementation-v2.md
  • Original Research: ~/.config/claude/history/research/2026-02/RESEARCH-podman-migration.md
  • Current Codebase: 8,135 LOC total (80% reusable, 20% Dagger-specific)
  • Estimated Completion: 2026-03-26 (7 weeks from start)