Merge pull request #358 from opsmill/chore/dev-docs-alignment

docs(dev): author dev/ guides, guidelines, knowledge, and seed ADRs

Author: BeArchiTek

AGENTS.md

@@ -15,7 +15,7 @@ This file is the portable router: repo-wide facts every agent needs up front. De
 |-----------|-------------|
 | Python | >=3.10, <3.14 |
 | ansible-core | >=2.17.7rc1 (Python 3.10+) |
-| infrahub-sdk | >=1.5, <2.0 |
+| infrahub-sdk | >=1.19.0, <2.0 |
 | Linter/Formatter | Ruff (pinned in pyproject.toml) |
 | Tests | pytest, ansible-test sanity (Docker-based) |
 | Docs | Docusaurus + Jinja2 generation |

dev/README.md

@@ -1,36 +1,67 @@
-# dev/
+# Developer Documentation
 
-Developer knowledge base and AI agent resources for the `opsmill.infrahub` Ansible collection.
+Internal documentation for `opsmill.infrahub` collection contributors. For
+user-facing docs, see the [Docusaurus site](https://docs.infrahub.app/ansible/)
+generated from `docs/`.
 
-## Structure
+## Quick Navigation
 
-### knowledge/
+| I want to...                       | Go to                  |
+|------------------------------------|------------------------|
+| Understand how the collection works | `knowledge/`          |
+| Follow coding and writing standards | `guidelines/`         |
+| Do a specific task step by step     | `guides/`             |
+| Learn why we made a decision        | `adr/`                |
+| Read the project constitution       | `constitution.md`     |
+| Use agent commands                  | `../.agents/commands/` |
 
-Deep reference material extracted from the codebase.
+## Directory Guide
 
-- [architecture.md](knowledge/architecture.md) — Collection structure, plugin types, data flow, key abstractions
-- [plugin-patterns.md](knowledge/plugin-patterns.md) — Ansible-specific conventions: boilerplate, docstrings, arg specs, conditional imports, state management
-- [infrahub-sdk-usage.md](knowledge/infrahub-sdk-usage.md) — InfrahubclientWrapper, InfrahubModule, processor classes, sync-only pattern
+- **constitution.md**: Pointer to the binding project principles in
+  `.specify/memory/constitution.md`. The authoritative reference.
+- **knowledge/**: Descriptive reference. How the system works.
+- **guidelines/**: Prescriptive rules. How code and docs should be written.
+- **guides/**: Step-by-step procedures for specific tasks.
+- **adr/**: Architecture Decision Records. Why we chose what we chose.
+
+## Current Knowledge
 
-### guidelines/
+- [architecture.md](knowledge/architecture.md) — Collection layout, plugin types, data flow, key abstractions
+- [plugin-patterns.md](knowledge/plugin-patterns.md) — Ansible conventions: boilerplate, docstrings, arg specs, conditional imports, state management
+- [infrahub-sdk-usage.md](knowledge/infrahub-sdk-usage.md) — InfrahubclientWrapper, InfrahubModule, processor classes, sync-only pattern
+- [inventory-and-lookup.md](knowledge/inventory-and-lookup.md) — Dynamic inventory and GraphQL lookup plugins, end to end
+- [processors-and-arg-spec.md](knowledge/processors-and-arg-spec.md) — `INFRAHUB_ARG_SPEC`, the `deepcopy` extension pattern, and the processor class hierarchy
 
-Standards and conventions for contributing.
+## Current Guidelines
 
-- [python.md](guidelines/python.md) — Ruff config, line length 120, rule selection, format settings
-- [testing.md](guidelines/testing.md) — Docker-based test execution, unit tests with mocks, sanity tests
+- [python.md](guidelines/python.md) — Ruff config, line length 120, type hints, dependencies
+- [testing.md](guidelines/testing.md) — Docker-based test execution, mocking, sanity/unit/integration
 - [documentation.md](guidelines/documentation.md) — Doc generation pipeline, Jinja2 templates, Docusaurus, Vale
+- [module-docstrings.md](guidelines/module-docstrings.md) — DOCUMENTATION/EXAMPLES/RETURN rules and doc fragments
+- [markdown.md](guidelines/markdown.md) — Markdown conventions, file naming, markdownlint/Vale/yamllint
 - [git-workflow.md](guidelines/git-workflow.md) — Branch model (develop/stable), PR conventions, CI, versioning
 
-### guides/
+## Current Guides
 
-Step-by-step how-tos for common tasks.
-
-- [creating-a-module.md](guides/creating-a-module.md) — Add a new module + action plugin + tests + docs
+- [creating-a-module.md](guides/creating-a-module.md) — Add a new module + action/module_utils + tests + docs
 - [running-tests.md](guides/running-tests.md) — Invoke tasks, Docker Compose, pytest, troubleshooting
+- [adding-a-doc-fragment.md](guides/adding-a-doc-fragment.md) — When and how to add a shared doc fragment
+- [debugging-sanity-failures.md](guides/debugging-sanity-failures.md) — Reproduce and fix `ansible-test sanity` failures
+- [releasing-the-collection.md](guides/releasing-the-collection.md) — Cut a release: develop→stable, automated version bump, build, and Galaxy publish
+
+## Current ADRs
+
+- [0001-two-plugin-patterns.md](adr/0001-two-plugin-patterns.md) — Module-utils vs action plugin patterns
+- [0002-sdk-abstraction-wrapper.md](adr/0002-sdk-abstraction-wrapper.md) — Wrap the SDK behind `InfrahubclientWrapper`
+- [0003-sync-only-sdk.md](adr/0003-sync-only-sdk.md) — Synchronous-only SDK usage
+- [0004-docker-based-testing.md](adr/0004-docker-based-testing.md) — Run all test modes in Docker via a multi-stage image
+- [0005-doc-generation-pipeline.md](adr/0005-doc-generation-pipeline.md) — Generate plugin reference MDX from docstrings
 
-### commands/
+## Agent Commands
 
-Claude Code slash commands (available via `/add-module`, `/fix-bug`).
+Agent commands live at the repository root under
+[`../.agents/commands/`](../.agents/commands/):
 
-- [add-module.md](commands/add-module.md) — Scaffold a new Ansible module
-- [fix-bug.md](commands/fix-bug.md) — Guided bug investigation and fix workflow
+- [add-module](../.agents/commands/add-module.md) — Scaffold a new Ansible module
+- [fix-bug](../.agents/commands/fix-bug.md) — Guided bug investigation and fix workflow
+- `speckit.*` — Spec-kit workflow commands (specify, plan, tasks, implement, …)

dev/adr/0001-two-plugin-patterns.md

@@ -0,0 +1,54 @@
+# ADR-0001: Two Plugin Patterns (Module-Utils vs Action Plugin)
+
+**Status**: Accepted
+**Date**: 2026-02-25
+**Source**: `.specify/memory/constitution.md` (Principle II) — backfilled from existing code
+
+## Context
+
+Every Ansible operation in this collection is exposed as a module under
+`plugins/modules/`, but the modules differ sharply in what they do. Some
+(`node`, `branch`) perform stateful CRUD that must be idempotent, support
+`check_mode`, and emit `--diff` output. Others (`query_graphql`,
+`artifact_fetch`, `artifact_generate`, `object_file_fetch`, `schema`) are
+read-mostly operations against the Infrahub API that do not map cleanly onto
+present/absent state. Forcing both into a single execution model would either
+saddle read-only queries with unused idempotency machinery or strip stateful
+modules of the state tracking they need.
+
+## Decision
+
+Support two distinct implementation patterns, chosen per module:
+
+- **Module-utils pattern** — for stateful CRUD. The module stub in
+  `plugins/modules/` builds an `AnsibleModule` from `deepcopy(INFRAHUB_ARG_SPEC)`
+  and delegates to an `InfrahubModule` subclass in `plugins/module_utils/`
+  (`NodeModule`, `BranchModule`). These subclasses implement `run()`, support
+  `state` (present/absent), `check_mode`, and `--diff` via the base class's
+  `_ensure_object_exists` / `_ensure_object_absent` and diff helpers.
+- **Action plugin pattern** — for read-mostly operations. The module stub
+  carries documentation and the arg spec; an `ActionModule` in `plugins/action/`
+  runs the real logic controller-side, instantiating `InfrahubclientWrapper`
+  directly and returning a result dict.
+
+**Decision criteria**: use the module-utils pattern for anything requiring
+idempotency and state management; use the action plugin pattern for read-only
+queries or operations that cannot be made idempotent.
+
+## Consequences
+
+- Contributors must classify a new module up front. The procedure is documented
+  in [../guides/creating-a-module.md](../guides/creating-a-module.md) (Step 2a
+  vs 2b), and the patterns are detailed in
+  [../knowledge/plugin-patterns.md](../knowledge/plugin-patterns.md).
+- Both paths share the same SDK wrapper and credential handling, so divergence
+  is contained to the execution model, not the API surface.
+- `node` and `branch` are the only module-utils modules today; the remaining
+  modules use action plugins.
+
+## Alternatives Considered
+
+- **Single module-utils path for everything**: rejected — read-only queries gain
+  no benefit from state/diff machinery and would need awkward no-op semantics.
+- **Single action-plugin path for everything**: rejected — action plugins have no
+  built-in idempotency or `--diff` support, which `node`/`branch` require.

dev/adr/0002-sdk-abstraction-wrapper.md

@@ -0,0 +1,54 @@
+# ADR-0002: SDK Abstraction Behind InfrahubclientWrapper
+
+**Status**: Accepted
+**Date**: 2026-02-25
+**Source**: `.specify/memory/constitution.md` (Principle IV) — backfilled from existing code
+
+## Context
+
+This collection talks to Infrahub exclusively through the `infrahub-sdk`
+package. Without a discipline around how the SDK is used, every plugin —
+modules, action plugins, the inventory source, and the lookup plugin — would
+independently construct `InfrahubClientSync`, assemble its `Config`, translate
+SDK exceptions into Ansible errors, and reimplement node/branch/GraphQL calls.
+That duplication drifts as the SDK evolves and scatters error handling across
+the codebase.
+
+## Decision
+
+Route all Infrahub API access through a single wrapper, `InfrahubclientWrapper`,
+defined in `plugins/module_utils/infrahub_utils.py`.
+
+- The wrapper builds the SDK `Config` (address, `api_token`, `default_branch`,
+  `timeout`, `tls_insecure`) and owns the `InfrahubClientSync` instance. Callers
+  never instantiate the SDK client directly.
+- It exposes intent-named methods — `fetch_single_node`, `fetch_nodes`,
+  `create_node`, `save_node`, `delete_node`, `fetch_single_schema`,
+  `fetch_branch`, `create_branch`, `delete_branch`, `execute_graphql`,
+  `fetch_single_artifact`, `generate_artifact`.
+- Higher-level orchestration sits on top: the `InfrahubModule` base class for
+  stateful modules, and the `InfrahubBaseProcessor` /
+  `InfrahubNodesProcessor` / `InfrahubQueryProcessor` classes for the inventory
+  and lookup plugins.
+- SDK exceptions are mapped to Ansible errors in one place via
+  `handle_infrahub_exceptions_decorator` (`plugins/module_utils/exception.py`),
+  not at each call site.
+
+## Consequences
+
+- A single seam absorbs SDK API changes; call sites stay stable.
+- New plugins inherit consistent credential handling, branch awareness, and
+  error mapping for free. Usage is documented in
+  [../knowledge/infrahub-sdk-usage.md](../knowledge/infrahub-sdk-usage.md) and
+  the overall data flow in [../knowledge/architecture.md](../knowledge/architecture.md).
+- The wrapper is shared core code (~1500 lines); changes to it affect every
+  plugin and warrant extra review.
+
+## Alternatives Considered
+
+- **Direct `InfrahubClientSync` use in each plugin**: rejected — duplicates
+  config assembly and error handling, and couples every plugin to the SDK's
+  surface.
+- **A thin pass-through that only constructs the client**: rejected — leaves
+  exception mapping and node/branch/query orchestration to each caller, the very
+  duplication this decision avoids.

dev/adr/0003-sync-only-sdk.md

@@ -0,0 +1,44 @@
+# ADR-0003: Synchronous-Only SDK Usage
+
+**Status**: Accepted
+**Date**: 2026-02-25
+**Source**: `.specify/memory/constitution.md` (Principle IV) — backfilled from existing code
+
+## Context
+
+The `infrahub-sdk` ships both an asynchronous client (`InfrahubClient`) and a
+synchronous one (`InfrahubClientSync`). Ansible modules, action plugins, and
+inventory/lookup plugins all execute in a synchronous context — Ansible invokes
+them as blocking calls and expects a result dict (or populated inventory) when
+they return. Introducing `async`/`await` would mean managing an event loop
+inside each plugin and offers no benefit, since the collection issues API calls
+serially within a single task.
+
+## Decision
+
+Use `InfrahubClientSync` exclusively. The async `InfrahubClient` is never used.
+
+- `InfrahubclientWrapper` constructs and holds an `InfrahubClientSync`; all
+  wrapper methods block until completion.
+- The `infrahub-sdk` dependency is pinned `>=1.19.0,<2.0` with the `[all]` extras,
+  which include synchronous support.
+- No plugin defines `async def` entry points or runs an event loop.
+
+## Consequences
+
+- Plugins stay simple — straight-line synchronous code with no loop management.
+- Aligns with Ansible's execution model; results are available on return.
+- Throughput is bounded by serial API calls; this is acceptable for the
+  collection's workloads and is the lever tracked separately for inventory
+  fetch optimization.
+- Contributors must select the sync variant of any new SDK call. The sync-only
+  rule is reiterated in
+  [../knowledge/infrahub-sdk-usage.md](../knowledge/infrahub-sdk-usage.md).
+
+## Alternatives Considered
+
+- **Async `InfrahubClient` with an event loop per plugin**: rejected — adds
+  loop-management complexity to every plugin for no gain in a synchronous
+  Ansible context.
+- **Mixed sync/async depending on plugin**: rejected — two client lifecycles and
+  two error-handling paths for no clear benefit.

dev/adr/0004-docker-based-testing.md

@@ -0,0 +1,57 @@
+# ADR-0004: Docker-Based Test Execution
+
+**Status**: Accepted
+**Date**: 2026-02-25
+**Source**: `.specify/memory/constitution.md` — backfilled from existing code
+
+## Context
+
+The collection's test suite spans three `ansible-test` modes — sanity, unit,
+and integration — each with its own tooling expectations: a pinned Python
+version, a built-and-installed copy of the collection on a custom
+`ANSIBLE_COLLECTIONS_PATH`, and (for integration) a live Infrahub to talk to.
+Running these directly on a contributor's machine is fragile: `ansible-test`
+is sensitive to the Python version, the collection must be built and placed at
+exactly `ansible_collections/opsmill/infrahub`, and host state leaks between
+runs. CI and local runs would drift unless the environment were pinned in one
+place.
+
+## Decision
+
+Run every test mode inside Docker, built from a single multi-stage `Dockerfile`
+and orchestrated by `docker-compose.yml` and the Invoke tasks in `tasks/tests.py`.
+
+- The `Dockerfile` defines a `base` stage (Python `${PYTHON_VER}`, `uv`, the
+  virtualenv on `PATH`) and three derived targets: `sanity`, `unittests`, and
+  `integration`. Each target builds the collection with `ansible-galaxy
+  collection build`, installs it, switches to the collection path, and runs the
+  matching `ansible-test` command.
+- `docker-compose.yml` maps one service per target (`sanity`, `unit`,
+  `integration`), sharing build args and threading through
+  `ANSIBLE_SANITY_ARGS` / `ANSIBLE_UNIT_ARGS` / `ANSIBLE_INTEGRATION_ARGS` so
+  extra `ansible-test` flags (e.g. `-vvv`) can be passed from the environment.
+- `invoke tests-sanity` / `tests-unit` / `tests-integration` each run
+  `docker compose up --build --force-recreate --exit-code-from <service>
+  <service>` with `PYTHON_VER` taken from the Invoke config, so the task's exit
+  code reflects the test result.
+
+## Consequences
+
+- Tests are reproducible: the same image runs locally and in CI, so "works on
+  my machine" failures around Python version or collection layout disappear.
+- Contributors do not manage a local `ansible_collections` tree or build step by
+  hand — the image does it on every run.
+- The Python version is a single build arg (`PYTHON_VER`), making it cheap to
+  test against multiple interpreters.
+- Running tests requires Docker; the procedure and troubleshooting live in
+  [../guides/running-tests.md](../guides/running-tests.md) and the conventions in
+  [../guidelines/testing.md](../guidelines/testing.md).
+
+## Alternatives Considered
+
+- **Run `ansible-test` directly on the host**: rejected — fragile against
+  Python-version and collection-path drift, and pollutes host state between
+  runs.
+- **`tox` / `nox` virtualenv matrix without containers**: rejected — still
+  depends on host-level interpreters and does not isolate the integration
+  target's Infrahub dependency the way a container network does.

dev/adr/0005-doc-generation-pipeline.md

@@ -0,0 +1,55 @@
+# ADR-0005: Generated Plugin Reference Documentation
+
+**Status**: Accepted
+**Date**: 2026-02-25
+**Source**: `.specify/memory/constitution.md` — backfilled from existing code
+
+## Context
+
+Each plugin already carries its full interface in standard Ansible docstrings —
+`DOCUMENTATION`, `EXAMPLES`, and `RETURN` — which `ansible-test sanity`
+validates. The user-facing reference site (the Docusaurus build under `docs/`,
+published to <https://docs.infrahub.app/ansible/>) needs the same information as
+MDX pages. Maintaining a second, hand-written copy of every parameter and return
+value would guarantee drift: the docstrings are the source `ansible-test`
+enforces, so any prose duplicate of them would silently fall out of date.
+
+## Decision
+
+Generate the plugin reference MDX from the docstrings; never hand-edit the
+output. The `invoke generate-doc` task (`tasks/docs.py`) is the single pipeline.
+
+- `generate_docs` discovers plugin files under `plugins/{modules,inventory,
+  lookup}/`, parses `DOCUMENTATION` / `EXAMPLES` / `RETURN` out of each, and
+  renders them through the Jinja2 templates in `docs/_templates/`
+  (`plugin.mdx.j2` and the readme template).
+- Output is written to `docs/docs/references/plugins/<plugin>_<type>.mdx` and the
+  landing page `docs/docs/readme.mdx`, stamped with the collection version from
+  `galaxy.yml` and the `requires_ansible` value.
+- A `mdx_safe` Jinja2 filter escapes JSX-reserved braces in prose (so a
+  docstring `{var}` is not parsed as a JSX expression) while leaving Markdown
+  code spans untouched.
+- `invoke docusaurus` then builds the static site (`npm run build`) from the
+  generated MDX.
+
+## Consequences
+
+- The docstrings are the single source of truth; the reference site cannot drift
+  from the validated interface.
+- Files under `docs/docs/references/plugins/` and `docs/docs/readme.mdx` are
+  generated artifacts and must never be edited directly — fix the docstring in
+  `plugins/modules/*.py` or the template in `docs/_templates/` and regenerate.
+- Changing a docstring obliges the author to run `invoke generate-doc`; CI also
+  regenerates and commits the docs on release
+  (`workflow-changelog-and-docs.yml`).
+- The rules and pipeline detail live in
+  [../guidelines/documentation.md](../guidelines/documentation.md) and
+  [../guidelines/module-docstrings.md](../guidelines/module-docstrings.md).
+
+## Alternatives Considered
+
+- **Hand-written reference pages**: rejected — duplicates the docstrings and
+  drifts from the interface `ansible-test sanity` validates.
+- **`antsibull-docs` to build a standalone Ansible doc site**: rejected — the
+  project standardised on a single Docusaurus site, so a bespoke Jinja2 → MDX
+  step keeps all docs (reference and narrative) in one toolchain.

dev/constitution.md

@@ -0,0 +1,6 @@
+# Constitution
+
+The binding project principles live in
+[`.specify/memory/constitution.md`](../.specify/memory/constitution.md) — the
+single source of truth maintained through the spec-kit workflow. This pointer
+exists so `dev/` links resolve; do not duplicate the body here, to avoid drift.

dev/guidelines/documentation.md

@@ -147,7 +147,7 @@ CI runs Vale automatically on documentation changes (`.github/workflows/workflow
 
 ## Markdown Linting
 
-Configuration in `.markdownlint.yml`. Ensures consistent markdown formatting.
+Configuration in `.markdownlint.yaml`. Ensures consistent markdown formatting.
 
 ## What to Document