docs: add validation framework requirements and detailed design by hdamker · Pull Request #447 · camaraproject/ReleaseManagement

hdamker · 2026-03-17T16:21:07Z

What type of PR is this?

documentation

What this PR does / why we need it:

Requirements and detailed design for the CAMARA Validation Framework, split into two documents:

Requirements (CAMARA-Validation-Framework-Requirements.md) — what the framework must do and what constraints apply:

Use cases and validation contexts (profiles, rule model, execution contexts)
MVP scope definition
Check inventory and rule architecture
Bundling and external ref resolution (Commonalities consumption, cache sync)
Artifact and findings surfacing
Workflow contract (reusable workflow inputs/outputs)
Rollout strategy (central config, stage model)
Release automation integration points

Detailed Design (CAMARA-Validation-Framework-Detailed-Design.md) — architecture decisions, processing flows, and implementation detail for developers:

Rule metadata model (YAML structure, condition evaluation, Spectral pass-through)
Bundling pipeline ($ref interaction, dependency categories, version matrix)
Artifact surfacing detail (naming, temporary branches, wip handling)
Caller workflow design (token resolution, GitHub App, triggers, ref resolution, templates)
Rollout implementation (central config alternatives, rulesets, caller update strategy)
Release automation implementation (handoff models, file restriction checks, context model)

The detailed design document is expected to migrate to the tooling repository alongside the implementation code.

Which issue(s) this PR fixes:

Related to #435

Special notes for reviewers:

The requirements document is the primary review target — it defines what the framework must do. The detailed design document provides supporting context for reviewers who want to understand architectural choices.

Presented at WG meeting 2026-03-17.

Changelog input

release-note
n/a (supporting document)

Additional documentation

docs
Added documentation/SupportingDocuments/CAMARA-Validation-Framework-Requirements.md
Added documentation/SupportingDocuments/CAMARA-Validation-Framework-Detailed-Design.md

Initial draft of the CAMARA Validation Framework requirements document covering use cases, validation contexts, rule architecture, bundling, caller workflow design, and rollout strategy. Work in progress — Sessions 5-6 (integration points, consolidation) still pending.

…on 5) Add sections 11 (Release Automation Integration) and 12 (Operational Views) to the validation framework requirements document. Update cross-references in sections 2.2, 2.3, 3, and 7.7. Resolve deferred bundling/snapshot interaction topic.

Split the 1,178-line combined document into a focused requirements document (522 lines) and a companion detailed design document (634 lines). The requirements document is now reviewable by WG members without implementation detail. The detailed design document preserves all architecture decisions, processing flows, YAML structures, and workflow templates for implementers.

Add sections 8 and 9 to the detailed design document, completing the workflow architecture needed for implementation: - Section 8: job architecture, checkout strategy, context builder, engine orchestration, composite action boundaries, common findings model - Section 9: post-filter processing, profile application, output formatting, truncation strategy, line number mapping, error handling, bundled spec artifact handoff (Model B) Also updates: - Section 6.2: central config schema with fork_owners allowlist - Section 7.1: settle on artifact handoff model (Model B) - Section 7.4: clarify mode input for release automation invocation - Requirements section 9.2: add mode input to workflow inputs table

hdamker · 2026-03-18T19:29:19Z

Added sections 8 and 9 to the detailed design document (Session 7):

Section 8 — End-to-end processing flow: single-job workflow architecture, checkout strategy, context builder with branch/trigger/profile derivation, engine orchestration (yamllint → bundling → Spectral/gherkin/Python), common findings model
Section 9 — Output pipeline: post-filter processing, profile-based blocking, output formatting (summary, annotations, PR comment, commit status), truncation, line number mapping, error handling, bundled spec artifact handoff to release automation

Also updated: central config schema (section 6.2) with fork_owners allowlist, settled on artifact handoff model for bundled specs (section 7.1), added mode input to workflow contract (section 7.4 + requirements 9.2).

tanjadegroot

Looks good to me - a few suggestions for consideration

/LGTM

tanjadegroot · 2026-03-18T22:30:01Z

documentation/SupportingDocuments/CAMARA-Validation-Framework-Requirements.md

+
+### 1.3 Release Automation
+
+UC-08: Use validation as a strict gate before creating a release snapshot.


If possible, this should include the check on the presence of all required release artefacts.

Gate should fail if any not present.

If present, content review should be done before any release PR is created.

Yes, but that's on detail level which rules are implemented and which severity they have. The "pre-snapshot" validation will use the "strict" profile, which means that every warning will block.

What do you mean with "content review"? That might go beyond what automated checks can do.

added User Story existence to the automated checks in 77b728a
Content check stays manual.

tanjadegroot · 2026-03-18T23:12:06Z

documentation/SupportingDocuments/CAMARA-Validation-Framework-Detailed-Design.md

+
+**Spectral (existing + new):**
+- OpenAPI version enforcement (3.0.3)
+- Naming conventions (paths: kebab-case, schemas: PascalCase, operationId: camelCase)


additional rules:

ENUMs: UPPER_SNAIL_CASE

examples: UPPER_SNAIL_CASE

parameters: kebab-case

headers: kebab-case

tags: Upper Camel Case (with spaces) applied to Api Name

Subject of the Spectral rule set, I changed the line to examples, not the complete list in 31bc028

@tanjadegroot There is now an update of section 2.1 with a summary of the rules per engine, and an appendix with all identified naming conventions which can be checked (## Appendix A: Naming Conventions).

documentation/SupportingDocuments/CAMARA-Validation-Framework-Detailed-Design.md

…Detailed-Design.md Co-authored-by: Tanja de Groot <87864067+tanjadegroot@users.noreply.github.com>

- Add CAMARA-Validation-Framework-Design-Guide-Audit.md: comprehensive audit of 106 machine-checkable rules from both design guides, cross- referenced against existing Spectral rules, OWASP rules (tooling#95), Linting-rules.md, and api_review_validator_v0_6.py - Update section 2.2 with complete check areas by engine - Add Appendix A: naming conventions (full list from design guide) - Clean internal tracking identifiers from document

Audit of 65 machine-checkable rules from API-Testing-Guidelines.md, cross-referenced against gherkin-lintrc (25 rules) and api_review_validator_v0_6.py test alignment checks.

Cross-reference identified discrepancies, gaps, and disabled rules against prior Commonalities and tooling issues/PRs to document known rationale and decisions.

camara-language-spelling was never intended for Spectral — marked "No/No" from day one. Spelling checks are better suited to AI-based review tools. Move from discrepancies to other items section.

hdamker · 2026-03-19T09:20:16Z

Two working documents added as input for the validation framework check inventory:

CAMARA-Validation-Framework-Design-Guide-Audit.md — Audit of 106 machine-checkable rules extracted from CAMARA-API-Design-Guide.md and CAMARA-API-Event-Subscription-and-Notification-Guide.md, cross-referenced against existing Spectral rules, the OWASP rules from tooling#95, Linting-rules.md, and api_review_validator_v0_6.py. Includes coverage mapping, gap identification, version sensitivity analysis (r3.4 vs r4.x), and a summary of prior Commonalities discussions related to the findings.
CAMARA-Validation-Framework-Testing-Guidelines-Audit.md — Audit of 65 machine-checkable rules from API-Testing-Guidelines.md, cross-referenced against the gherkin-lintrc configuration and the v0_6 validator test alignment checks.

Both documents are working documents to support the design review — feedback on completeness and accuracy is welcome.

The check areas summary in the detailed design (section 2.2) and a naming conventions appendix (Appendix A) have been updated based on the audit findings.

Section 2.2: add gherkin-lint engine category listing test definition file checks (structural rules, step ordering, tagging, formatting). Section 8.4: add gherkin-lint column to findings normalization source mapping table.

Conditional check for stable public APIs per API Readiness Checklist. File existence is automated; content adequacy remains manual.

rartych · 2026-03-19T11:06:30Z

documentation/SupportingDocuments/CAMARA-Validation-Framework-Design-Guide-Audit.md

+| DG-025 | info.version MUST follow semver/wip/alpha/rc | v0_6: V6-004; linting-rules.md: camara-info-version-format (tbd) | spectral | both | **DISCREPANCY**: Linting-rules.md marks "tbd", not in .spectral.yaml |
+| DG-026 | info.license.name MUST be "Apache 2.0" | v0_6: V6-005 | spectral | both | |
+| DG-027 | info.license.url MUST be Apache URL | v0_6: V6-006; core-oas: license-url | spectral | both | |
+| DG-028 | x-camara-commonalities MUST specify version | v0_6: V6-007 | spectral | both | |


DG-028 This will be changed by camaraproject/Commonalities#599 (ready to merge)

rartych · 2026-03-19T11:44:56Z

documentation/SupportingDocuments/CAMARA-Validation-Framework-Design-Guide-Audit.md

+
+### Cross-cutting finding
+
+The incremental rollout approach was established in [PR#74](https://github.com/camaraproject/Commonalities/pull/74) (Oct 2023), where rules were documented before implementation. Several rules from that era remain documented but not yet implemented. No existing issue specifically tracks the gap between documented rules in Linting-rules.md and implemented rules in .spectral.yml.


That's true.
Most of missing rules requires TypeScript function implementation, some are even hard to implement within spectral.
BTW, camara-schema-type-check rule is implemented partially, I have prepared function that looks into more complex schema definitions that seems to work. Probably new rule should be added.
It happens that type: object is missing - it works with even Swagger - as properties: implicitly indacates it is an object - so Swagger doesn't complain.

rartych · 2026-03-19T12:20:55Z

documentation/SupportingDocuments/CAMARA-Validation-Framework-Detailed-Design.md

+
+#### Release automation handoff (Model B)
+
+When release automation invokes validation with `mode: pre-snapshot`, it consumes the bundled spec artifacts to populate the snapshot branch. The handoff uses the **artifact model** (section 7.1, Model B):


The handoff uses the artifact model (section 7.1, Model B):

It is not clear in 7.1 what is Model B ?

tanjadegroot · 2026-03-20T16:34:43Z

question on the "machine-checkable rules", do we envisage to add (e.g. as a rule number/id or just an "*" os "A" or something in parentheses) into the original guideline documents such that it is clear that these will be checked automatically ?

tanjadegroot

comment batch on this file - looks great !!

tanjadegroot · 2026-03-20T18:13:48Z

documentation/SupportingDocuments/CAMARA-Validation-Framework-Design-Guide-Audit.md

+|----|------|----------|-----------|---------|-------|
+| DG-083 | info.version MUST be wip on main branch | v0_6: V6-054 | python | both | Requires branch context |
+| DG-084 | info.version MUST NOT be wip on release branches | v0_6: V6-056 | python | both | Requires branch context |
+| DG-085 | Server URL version MUST match info.version | v0_6: V6-057 | python | both | Cross-field validation |


this is not correct: server URL version MUST be

"v" + major version number from info.version if >= 1

"v" + major version number (0) + "." + minor version number from info.version (CAMARA exception)

tanjadegroot · 2026-03-20T18:18:51Z

documentation/SupportingDocuments/CAMARA-Validation-Framework-Design-Guide-Audit.md

+| Event type | org.camaraproject.\<api\>.\<ver\>.\<event\> | — | Event Guide 3.1 | **Gap** |
+| Error codes | SCREAMING_SNAKE_CASE | — | 3.2 | **Gap** (r4.x-only explicit) |
+| API-specific error codes | API_NAME.SPECIFIC_CODE | — | 3.2.1 | **Gap** |
+


the list is missing the examples: "EXAMPLE_" + UPPER SNAKE_CASE

tanjadegroot · 2026-03-20T18:19:31Z

documentation/SupportingDocuments/CAMARA-Validation-Framework-Design-Guide-Audit.md

+
+---
+
+## Naming Conventions (complete list)


it seems we use both SCREAMING_SNAKE_CASE and UPPER_SNAKE_CASE, which seem to be the same.

We need to pick one for consistency.

tanjadegroot · 2026-03-20T18:27:52Z

documentation/SupportingDocuments/CAMARA-Validation-Framework-Design-Guide-Audit.md

+
+---
+
+## Section 5: API Definition


I have seen some teams using a "- title: xxx" section in e.g. tags, paths, etc.
OAS.3.0.3 says that, part from its use in info.title (the API title), also a property "title" may exist taken directly from the JSON Schema definition and follows the same specifications.

See https://spec.openapis.org/oas/v3.0.3.html#properties

Should we disallow its use beyond the "info.title" or just ignore it ?

tanjadegroot · 2026-03-20T18:37:36Z

documentation/SupportingDocuments/CAMARA-Validation-Framework-Design-Guide-Audit.md

+|----|------|----------|-----------|---------|-------|
+| DG-086 | Event type MUST follow org.camaraproject.\<api\>.\<ver\>.\<event\> | gap (v0_6 partial) | python | both | |
+| DG-087 | specversion MUST be "1.0" enum | gap | spectral | both | |
+| DG-088 | Subscription API filename MUST append "subscriptions" | gap | python | both | Filesystem check |


tanjadegroot · 2026-03-20T18:41:33Z

documentation/SupportingDocuments/CAMARA-Validation-Framework-Design-Guide-Audit.md

+
+| ID | Rule | Coverage | v1 Engine | Version | Notes |
+|----|------|----------|-----------|---------|-------|
+| DG-067 | HTTPS must always be used (no HTTP) | owasp: api8:2023-no-scheme-http + no-server-http (error) | spectral | both | |


exception is DG-090 ?

tanjadegroot · 2026-03-20T18:44:46Z

documentation/SupportingDocuments/CAMARA-Validation-Framework-Design-Guide-Audit.md

+| DG-092 | sinkCredential MUST NOT appear in responses | gap | python | both | |
+| DG-093 | notificationsBearerAuth scheme for callbacks | v0_6: V6-037/V6-044–046/V6-048 | python | both | |
+| DG-094 | Notification content-type: application/cloudevents+json | gap | spectral | both | |
+| DG-095 | Event version independent of API version | gap | python | both | Complex versioning logic |


Do we add a rule that the event version structure must be "v"+integer ? so no v0.y or v0.y.z allowed for event versions ?

tanjadegroot · 2026-03-20T18:49:12Z

documentation/SupportingDocuments/CAMARA-Validation-Framework-Design-Guide-Audit.md

+| DG-101 | Test files exist for each API | v0_6: V6-072 | python | both | |
+| DG-102 | Test file version alignment | v0_6: V6-073 | python | both | |
+| DG-103 | Filename uses kebab-case | v0_6: V6-058 | python | both | |
+| DG-104 | Filename matches api-name in server URL | v0_6: V6-059 | python | both | |


or filename matches api-name + "-" + operationId
and all operationIds shall be covered by test files (for release-candidate latest)

tanjadegroot · 2026-03-20T19:07:45Z

documentation/SupportingDocuments/CAMARA-Validation-Framework-Design-Guide-Audit.md

+
+| ID | Rule | Coverage | v1 Engine | Version | Notes |
+|----|------|----------|-----------|---------|-------|
+| DG-105 | Avoid telco-specific terminology | spectral: camara-language-avoid-telco (hint) | spectral | both | |


one API is introducing "EMEI". Do we consider this as Telco language ? (I lean towards yes).

If so, should we add it to the check and also define a CAMARA common schema "DeviceIdentifier" with its format regex ?

hdamker added 3 commits March 17, 2026 17:18

hdamker changed the title ~~docs: add validation framework requirements and design~~ docs: add validation framework requirements and detailed design Mar 17, 2026

hdamker requested review from a team and Kevsy March 17, 2026 22:25

hdamker marked this pull request as ready for review March 17, 2026 22:26

hdamker requested review from rartych, soadeyemo and tanjadegroot as code owners March 17, 2026 22:26

This was referenced Mar 18, 2026

Implement CAMARA Validation Framework v1 #448

Open

Implement PR Validation Framework (C1 Workflow) #353

Closed

tanjadegroot previously approved these changes Mar 18, 2026

View reviewed changes

Update documentation/SupportingDocuments/CAMARA-Validation-Framework-…

5ff2df3

…Detailed-Design.md Co-authored-by: Tanja de Groot <87864067+tanjadegroot@users.noreply.github.com>

hdamker dismissed tanjadegroot’s stale review via 5ff2df3 March 19, 2026 07:14

hdamker added 5 commits March 19, 2026 08:28

docs: expand naming conventions examples in check areas summary

31bc028

docs: add testing guidelines audit

21d8949

Audit of 65 machine-checkable rules from API-Testing-Guidelines.md, cross-referenced against gherkin-lintrc (25 rules) and api_review_validator_v0_6.py test alignment checks.

docs: add prior discussions section to design guide audit

b2a1480

Cross-reference identified discrepancies, gaps, and disabled rules against prior Commonalities and tooling issues/PRs to document known rationale and decisions.

docs: reclassify spelling rule as deliberate exclusion

8d90eb0

camara-language-spelling was never intended for Spectral — marked "No/No" from day one. Spelling checks are better suited to AI-based review tools. Move from discrepancies to other items section.

hdamker added 2 commits March 19, 2026 10:26

docs: add user story file existence check to Python engine

77b728a

Conditional check for stable public APIs per API Readiness Checklist. File existence is automated; content adequacy remains manual.

rartych reviewed Mar 19, 2026

View reviewed changes

hdamker mentioned this pull request Mar 20, 2026

Release Issue Not Automatically Created for SessionInsights Repo #451

Open

tanjadegroot reviewed Mar 20, 2026

View reviewed changes


		### 1.3 Release Automation

		UC-08: Use validation as a strict gate before creating a release snapshot.


		### Cross-cutting finding

		The incremental rollout approach was established in [PR#74](https://github.com/camaraproject/Commonalities/pull/74) (Oct 2023), where rules were documented before implementation. Several rules from that era remain documented but not yet implemented. No existing issue specifically tracks the gap between documented rules in Linting-rules.md and implemented rules in .spectral.yml.


		#### Release automation handoff (Model B)

		When release automation invokes validation with `mode: pre-snapshot`, it consumes the bundled spec artifacts to populate the snapshot branch. The handoff uses the artifact model (section 7.1, Model B):

	\| DG-088 \| Subscription API filename MUST append "subscriptions" \| gap \| python \| both \| Filesystem check \|
	\| DG-088 \| Subscription API filename MUST append "-subscriptions" \| gap \| python \| both \| Filesystem check \|

Conversation

hdamker commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for reviewers:

Changelog input

Additional documentation

Uh oh!

hdamker commented Mar 18, 2026

Uh oh!

tanjadegroot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hdamker Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hdamker Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hdamker Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hdamker commented Mar 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rartych Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tanjadegroot commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tanjadegroot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hdamker commented Mar 17, 2026 •

edited

Loading

hdamker Mar 19, 2026 •

edited

Loading

hdamker Mar 19, 2026 •

edited

Loading

hdamker Mar 19, 2026 •

edited

Loading

rartych Mar 19, 2026 •

edited

Loading

tanjadegroot commented Mar 20, 2026 •

edited

Loading