diff --git a/documentation/SupportingDocuments/CAMARA-Validation-Framework-Design-Guide-Audit.md b/documentation/SupportingDocuments/CAMARA-Validation-Framework-Design-Guide-Audit.md new file mode 100644 index 0000000..76a3e69 --- /dev/null +++ b/documentation/SupportingDocuments/CAMARA-Validation-Framework-Design-Guide-Audit.md @@ -0,0 +1,473 @@ +# Commonalities Design Guide Audit + +**Date**: 2026-03-19 +**Scope**: Machine-checkable rules from CAMARA-API-Design-Guide.md and CAMARA-API-Event-Subscription-and-Notification-Guide.md +**Versions compared**: r3.4 (Commonalities 0.5) vs r4.1 (Commonalities 0.6) + +## Methodology + +1. Walked both design guides section by section, extracting every machine-checkable requirement +2. Cross-referenced against: Linting-rules.md, current .spectral.yaml (17 CAMARA rules + core OAS), tooling#95 OWASP rules, api_review_validator_v0_6.py (80 checks) +3. Compared r3.4 tag against current r4.1 for version sensitivity + +## Legend + +**Coverage column**: +- `spectral: ` — rule exists in current .spectral.yaml +- `owasp: ` — rule exists in tooling#95 (not merged) +- `linting-rules.md: ` — listed in Linting-rules.md but NOT in .spectral.yaml +- `v0_6: V6-NNN` — covered by api_review_validator_v0_6.py +- `core-oas: ` — spectral:oas built-in rule +- `gap` — no current implementation + +**v1 Engine**: `spectral` / `python` / `manual` / `obsolete` + +**Version**: `both` / `r4.x-only` / `changed` / `r3.4-only` + +--- + +## Section 2: Data Types + +### 2.2 Data Type Constraints + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-001 | String: maxLength or enum MUST be used | owasp: api4:2023-string-limit (warn) | spectral | r4.x-only | New MUST in r4.1. Target severity: error | +| DG-002 | String: format/pattern/enum/const SHOULD be used | owasp: api4:2023-string-restricted (warn) | spectral | both | Advisory | +| DG-003 | date-time description MUST state RFC 3339 with timezone | gap | spectral | both | Regex on description field | +| DG-004 | duration description MUST state RFC 3339 | gap | spectral | both | Regex on description field | +| DG-005 | Array: maxItems MUST be specified | owasp: api4:2023-array-limit (warn) | spectral | r4.x-only | New MUST in r4.1. Target severity: error | +| DG-006 | Integer: format (int32/int64) MUST be specified | owasp: api4:2023-integer-format (warn) | spectral | r4.x-only | Was informal in r3.4. Target: error | +| DG-007 | Integer: minimum and maximum MUST be specified | owasp: api4:2023-integer-limit-legacy (warn) | spectral | r4.x-only | Was informal in r3.4. Target: error | +| DG-008 | Object: required properties MUST be listed | gap | spectral | both | Check for `required` array on object schemas | +| DG-009 | type attribute MUST be present on all properties | spectral: camara-schema-type-check (error) | spectral | both | | + +### 2.2.1 Discriminator + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-010 | oneOf/anyOf SHOULD include discriminator | spectral: camara-discriminator-use (hint) | spectral | changed | Deprecated in r4.1 (was warn in r3.4) | + +--- + +## Section 3: Errors and Responses + +### 3.1 Business-level Outcomes (new in r4.1) + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-011 | contextCode values SHOULD follow API_NAME.SPECIFIC_CODE in SCREAMING_SNAKE_CASE | gap | python | r4.x-only | New section. Check if contextCode field is used | + +### 3.2 Error Responses + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-012 | Error response MUST have status, code, message fields | v0_6: V6-033 | python | both | ErrorInfo schema structure. Becomes spectral with bundling (ref to CAMARA_common.yaml) | +| DG-013 | Error code MUST NOT be numeric | gap | spectral | both | Regex on error code enum values | +| DG-014 | Error code MUST be SCREAMING_SNAKE_CASE | gap | spectral | r4.x-only | New explicit requirement in r4.1 | +| DG-015 | API-specific error codes MUST follow API_NAME.SPECIFIC_CODE | gap | spectral | both | Pattern: `^[A-Z][A-Z0-9_]*\.[A-Z][A-Z0-9_]*$`. Clarified in r4.1 | +| DG-016 | All APIs MUST document 401 response | owasp: api8:2023-define-error-responses-401 (error) | spectral | both | | +| DG-017 | All APIs MUST document 403 response | gap | spectral | both | OWASP only covers 401, not 403 | +| DG-018 | CONFLICT error code is DEPRECATED | gap | python | r4.x-only | Warn if API uses CONFLICT | +| DG-019 | Error response content-type MUST be application/json | v0_6: V6-028 | spectral | both | | +| DG-020 | Error responses MUST reference ErrorInfo or allOf containing ErrorInfo | v0_6: V6-029 | python | both | Needs $ref resolution | + +--- + +## Section 5: API Definition + +### 5.1 Reserved Words + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-021 | No reserved words in paths, params, operationIds, components, security schemes | spectral: camara-reserved-words (warn) | spectral | both | | +| DG-022 | Resource names MUST NOT contain HTTP method names | linting-rules.md: camara-resource-reserved-words (warn) | spectral | both | **DISCREPANCY**: listed in Linting-rules.md but NOT in .spectral.yaml | + +### 5.2 OpenAPI Version + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-023 | OpenAPI version MUST be 3.0.3 | spectral: camara-oas-version (error) | spectral | both | | + +### 5.3 Info Object + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-024 | info.title MUST NOT include "API" | v0_6: V6-003; linting-rules.md: camara-info-title (tbd) | spectral | both | **DISCREPANCY**: Linting-rules.md marks "tbd", not in .spectral.yaml | +| DG-025 | info.version MUST follow semver/wip/alpha/rc | v0_6: V6-004; linting-rules.md: camara-info-version-format (tbd) | spectral | both | **DISCREPANCY**: Linting-rules.md marks "tbd", not in .spectral.yaml | +| DG-026 | info.license.name MUST be "Apache 2.0" | v0_6: V6-005 | spectral | both | | +| DG-027 | info.license.url MUST be Apache URL | v0_6: V6-006; core-oas: license-url | spectral | both | | +| DG-028 | x-camara-commonalities MUST specify version | v0_6: V6-007 | spectral | both | | +| DG-029 | info.description MUST contain "Authorization and authentication" section | v0_6: V6-010/V6-011 | python | both | Normalized text matching | +| DG-030 | info.description MUST contain "Additional CAMARA error responses" section | v0_6: V6-012/V6-013 | python | r4.x-only | New in v0.6 | +| DG-031 | info.termsOfService MUST be absent | v0_6: V6-008 | spectral | both | | +| DG-032 | info.contact MUST be absent | gap | spectral | both | core-oas has contact-properties disabled | + +### 5.4 External Docs + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-033 | externalDocs MUST be present | v0_6: V6-014 | spectral | both | | +| DG-034 | externalDocs.description MUST match expected text | v0_6: V6-015 | spectral | both | "Product documentation at CAMARA" | +| DG-035 | externalDocs.url MUST be https://github.com/camaraproject/{repo} | v0_6: V6-016/V6-017 | spectral | both | | + +### 5.5 Servers + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-036 | At least one server MUST be defined | core-oas: oas3-api-servers | spectral | both | | +| DG-037 | Server URL MUST follow {apiRoot}/api-name/api-version | v0_6: V6-020/V6-057 | python | both | Cross-field: URL pattern vs info.version | +| DG-038 | All servers MUST have same api-name and api-version | v0_6: partial | python | both | | + +### 5.6 Tags + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-039 | Each operation SHOULD use exactly one tag | core-oas: operation-singular-tag (warn) | spectral | both | | +| DG-040 | Operation tags MUST be defined in global tags | core-oas: operation-tag-defined (warn) | spectral | both | | +| DG-041 | Tag names SHOULD use Title Case (with spaces) | gap | spectral | both | No casing rule for tags exists | + +### 5.7.1 Paths + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-042 | Paths MUST use kebab-case | spectral: camara-parameter-casing-convention (error) | spectral | both | | +| DG-043 | Path parameters MUST NOT be generic {id} | spectral: camara-path-param-id (warn) | spectral | both | | +| DG-044 | Path parameter morphology SHOULD be consistent ({entityId}) | linting-rules.md: camara-path-param-id-morphology (warn) | spectral | both | **DISCREPANCY**: in Linting-rules.md but NOT in .spectral.yaml | +| DG-045 | No trailing slash on paths | core-oas: path-keys-no-trailing-slash | spectral | both | | +| DG-046 | No query string in path | core-oas: path-not-include-query | spectral | both | | + +### 5.7.2 Operations + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-047 | Operations MUST have operationId | core-oas: operation-operationId | spectral | both | | +| DG-048 | operationId MUST use camelCase | spectral: camara-operationid-casing-convention (hint) | spectral | both | **SEVERITY**: .spectral.yaml=hint, Linting-rules.md=error | +| DG-049 | Operations MUST have summary | spectral: camara-operation-summary (warn) | spectral | both | | +| DG-050 | Operations MUST have description | spectral: camara-routes-description (warn) | spectral | both | | +| DG-051 | operationId MUST be unique | core-oas: operation-operationId-unique (error) | spectral | both | | +| DG-052 | Valid HTTP methods only (GET/PUT/POST/PATCH/DELETE/OPTIONS) | spectral: camara-http-methods (error) | spectral | both | | +| DG-053 | Operations MUST have at least one 2xx/3xx response | core-oas: operation-success-response | spectral | both | | + +### 5.7.4 Parameters + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-054 | All parameters MUST have description | spectral: camara-parameters-descriptions (warn) | spectral | both | | +| DG-055 | Property names MUST use lowerCamelCase | linting-rules.md: camara-property-casing-convention (error) | spectral | both | **DISCREPANCY**: in Linting-rules.md but NOT in .spectral.yaml | + +### 5.7.5 Request Bodies + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-056 | GET/DELETE MUST NOT have requestBody | spectral: camara-get-no-request-body (error) | spectral | both | | + +### 5.7.6 Responses + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-057 | All responses MUST have description | spectral: camara-response-descriptions (warn) | spectral | both | | +| DG-058 | Array response items MUST have description | gap | spectral | r4.x-only | New requirement in r4.1 | + +### 5.8.1 Schemas + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-059 | Schema names MUST use PascalCase | spectral: camara-schema-casing-convention (warn) | spectral | both | | +| DG-060 | All schema properties MUST have description | spectral: camara-properties-descriptions (warn) | spectral | both | | + +### 5.8.5 Headers + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-061 | x-correlator header MUST be included | v0_6: V6-032 | spectral | both | Check parameter/header presence | +| DG-062 | x-correlator pattern MUST match expected regex | v0_6: V6-062 | spectral | both | Pattern: `^[a-zA-Z0-9-_:;.\/<>{}]{0,256}$` | + +### 5.8.6 Security Schemes + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-063 | MUST use openIdConnect scheme named 'openId' | v0_6: V6-036/V6-041 | spectral | both | | +| DG-064 | openIdConnectUrl MUST use HTTPS | v0_6: V6-039 | spectral | both | | +| DG-065 | openIdConnectUrl MUST point to .well-known/openid-configuration | v0_6: V6-040 | spectral | both | | +| DG-066 | No oauth2 scheme type (must use openIdConnect) | v0_6: V6-042 | spectral | both | | + +--- + +## Section 6: Security + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-067 | HTTPS must always be used (no HTTP) | owasp: api8:2023-no-scheme-http + no-server-http (error) | spectral | both | | +| DG-068 | Sensitive data (MSISDN/IMSI/phoneNumber) MUST NOT be in path/query params | spectral: camara-security-no-secrets-in-path-or-query-parameters (warn) | spectral | both | | +| DG-069 | Write operations MUST have security scheme | owasp: api2:2023-write-restricted (error) | spectral | r4.x-only | | +| DG-070 | Read operations SHOULD have security scheme | owasp: api2:2023-read-restricted (warn) | spectral | r4.x-only | | +| DG-071 | Short-lived access tokens with refresh | owasp: api2:2023-short-lived-access-tokens (error) | spectral | r4.x-only | | +| DG-072 | No credentials in URL parameters | owasp: api2:2023-no-credentials-in-url (error) | spectral | r4.x-only | | +| DG-073 | No numeric IDs (use UUIDs or random) | owasp: api1:2023-no-numeric-ids (error) | spectral | r4.x-only | | +| DG-074 | Admin endpoints: unique security scheme | owasp: api5:2023-admin-security-unique (error) | spectral | r4.x-only | | +| DG-075 | No unconstrained additionalProperties | owasp: api3:2023-no-additionalProperties (warn) | spectral | r4.x-only | | +| DG-076 | Constrained additionalProperties | owasp: api3:2023-constrained-additionalProperties (warn) | spectral | r4.x-only | | +| DG-077 | 401 response MUST be defined on all operations | owasp: api8:2023-define-error-responses-401 (error) | spectral | both | | +| DG-078 | Error validation response (400/422/4XX) SHOULD be defined | owasp: api8:2023-define-error-validation (warn) | spectral | r4.x-only | | + +### 6.6 Scope Naming + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-079 | Scopes MUST follow api-name:[resource:]action pattern | v0_6: V6-050 | python | both | Context-dependent (subscription vs regular) | +| DG-080 | Subscription POST scopes: api-name:event-type:create | v0_6: V6-051 | python | both | | +| DG-081 | Subscription GET scopes: api-name:read | v0_6: V6-052 | python | both | | +| DG-082 | Subscription DELETE scopes: api-name:delete | v0_6: V6-053 | python | both | | + +--- + +## Section 7: Versioning (context-dependent) + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-083 | info.version MUST be wip on main branch | v0_6: V6-054 | python | both | Requires branch context | +| DG-084 | info.version MUST NOT be wip on release branches | v0_6: V6-056 | python | both | Requires branch context | +| DG-085 | Server URL version MUST match info.version | v0_6: V6-057 | python | both | Cross-field validation | + +--- + +## Naming Conventions (complete list) + +Consolidated from design guide sections. This is the full set for the detailed design appendix. + +| Element | Convention | Spectral Name | DG Section | Coverage | +|---------|-----------|--------------|------------|----------| +| Paths (URLs) | kebab-case | camara-parameter-casing-convention | 5.7.1 | Implemented (error) | +| Path parameters | lowerCamelCase (entityId form) | camara-path-param-id, camara-path-param-id-morphology | 5.7.1 | Partial (morphology not in .spectral.yaml) | +| Schemas | PascalCase | camara-schema-casing-convention | 5.8.1 | Implemented (warn) | +| operationId | camelCase | camara-operationid-casing-convention | 5.7.2 | Implemented (hint; should be error per Linting-rules.md) | +| Properties | lowerCamelCase | camara-property-casing-convention | 5.7.4 | **Not in .spectral.yaml** (Linting-rules.md: error) | +| Enum values | SCREAMING_SNAKE_CASE | camara-enum-casing-convention | 3.2 | **Not in .spectral.yaml** (Linting-rules.md: info, tbd) | +| Tags | Title Case (with spaces) | — | 5.7.3 | **Gap** — no rule exists | +| Headers | kebab-case | — | 5.8.5 | **Gap** — implied by x-correlator convention | +| Scope names | kebab-case with : separators | — | 6.6 | v0_6 only (V6-050) | +| API name | kebab-case | — | 1.2 | v0_6 only (V6-058/V6-059) | +| Event type | org.camaraproject.\.\.\ | — | Event Guide 3.1 | **Gap** | +| Error codes | SCREAMING_SNAKE_CASE | — | 3.2 | **Gap** (r4.x-only explicit) | +| API-specific error codes | API_NAME.SPECIFIC_CODE | — | 3.2.1 | **Gap** | +| Examples (named) | SCREAMING_SNAKE_CASE | — | OAS 3.0.3 | **Gap** | + +--- + +## Event Subscription and Notification Guide + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-086 | Event type MUST follow org.camaraproject.\.\.\ | gap (v0_6 partial) | python | both | | +| DG-087 | specversion MUST be "1.0" enum | gap | spectral | both | | +| DG-088 | Subscription API filename MUST append "-subscriptions" | gap | python | both | Filesystem check | +| DG-089 | Explicit subscriptions: 4 operations (POST/GET/GET{id}/DELETE) | v0_6: V6-065/V6-066 | python | both | | +| DG-090 | protocol attribute MUST be "HTTP" | gap | spectral | both | Enum constraint | +| DG-091 | sink MUST use HTTPS URI | gap | spectral | both | | +| DG-092 | sinkCredential MUST NOT appear in responses | gap | python | both | | +| DG-093 | notificationsBearerAuth scheme for callbacks | v0_6: V6-037/V6-044–046/V6-048 | python | both | | +| DG-094 | Notification content-type: application/cloudevents+json | gap | spectral | both | | +| DG-095 | Event version independent of API version | gap | python | both | Complex versioning logic | +| DG-096 | Implicit subscriptions: callbacks defined in operations | v0_6: V6-067 | python | both | | + +--- + +## Cross-File Checks + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-097 | Common schemas identical across API files | v0_6: V6-068 | python/obsolete | both | Becomes partially obsolete with bundling | +| DG-098 | License identical across API files | v0_6: V6-069 | python | both | | +| DG-099 | x-camara-commonalities identical across API files | v0_6: V6-070 | python | both | | +| DG-100 | Test directory exists | v0_6: V6-071 | python | both | | +| DG-101 | Test files exist for each API | v0_6: V6-072 | python | both | See testing guidelines audit TG-004, TG-052, TG-066 for operation-specific file naming and completeness | +| DG-102 | Test file version alignment | v0_6: V6-073 | python | both | | +| DG-103 | Filename uses kebab-case | v0_6: V6-058 | python | both | | +| DG-104 | Filename matches api-name in server URL | v0_6: V6-059 | python | both | | + +--- + +## Telco Language and Spelling + +| ID | Rule | Coverage | v1 Engine | Version | Notes | +|----|------|----------|-----------|---------|-------| +| DG-105 | Avoid telco-specific terminology | spectral: camara-language-avoid-telco (hint) | spectral | both | | +| DG-106 | Spelling check on descriptions | linting-rules.md: camara-language-spelling (No/warn) | spectral | both | Listed but explicitly not enabled | + +--- + +## Gap Summary + +### Gaps requiring new Spectral rules (r4.x ruleset) + +| ID | Rule | Priority | +|----|------|----------| +| DG-003 | date-time description MUST state RFC 3339 | Low (advisory) | +| DG-004 | duration description MUST state RFC 3339 | Low (advisory) | +| DG-008 | Object: required properties MUST be listed | Medium | +| DG-013 | Error code MUST NOT be numeric | Medium | +| DG-014 | Error code MUST be SCREAMING_SNAKE_CASE | Medium (r4.x-only) | +| DG-015 | API-specific error codes: API_NAME.SPECIFIC_CODE | Medium | +| DG-017 | All APIs MUST document 403 response | Medium | +| DG-032 | info.contact MUST be absent | Low | +| DG-041 | Tag names SHOULD use Title Case | Low | +| DG-058 | Array items MUST have description | Medium (r4.x-only) | +| DG-087 | specversion MUST be "1.0" | Low (subscription APIs only) | +| DG-090 | protocol MUST be "HTTP" | Low (subscription APIs only) | +| DG-091 | sink MUST use HTTPS | Low (subscription APIs only) | +| DG-094 | Notification content-type: cloudevents+json | Low (subscription APIs only) | + +### Gaps requiring Python checks + +| ID | Rule | Priority | +|----|------|----------| +| DG-011 | contextCode SCREAMING_SNAKE_CASE format | Low (r4.x-only) | +| DG-018 | CONFLICT error code deprecated warning | Low (r4.x-only) | +| DG-086 | Event type format validation | Medium | +| DG-088 | Subscription API filename convention | Medium | +| DG-092 | sinkCredential not in responses | Medium | +| DG-095 | Event version independence from API version | Low | + +### Rules already covered by v0_6 but not by Spectral (need Python in v1) + +Most v0_6 checks that need Python (V6-010, V6-012, V6-029, V6-050–053, V6-054–057, V6-068–080) will continue to need Python in v1 due to cross-field, cross-file, or context-dependent logic. + +--- + +## Linting-rules.md Discrepancies + +Rules listed in Linting-rules.md but **NOT implemented** in current .spectral.yaml: + +| Linting-rules.md Rule | Severity | Status | Notes | +|------------------------|----------|--------|-------| +| camara-resource-reserved-words | warn | Missing | Resource names must not contain HTTP method names | +| camara-path-param-id-morphology | warn | Missing | Consistent {entityId} morphology | +| camara-property-casing-convention | error | Missing | Property names lowerCamelCase | +| camara-enum-casing-convention | info | Missing (tbd) | Enum values SCREAMING_SNAKE_CASE | +| camara-info-title | warn | Missing (tbd) | Title must not contain "API" | +| camara-info-version-format | warn | Missing (tbd) | Version format x.y.z/wip/alpha/rc | + +Note: `camara-language-spelling` is listed in Linting-rules.md as "No/No" — this is not a discrepancy but a deliberate exclusion. Spelling and description quality checks are better suited to AI-based review tools than pattern-based linting. + +Rules in .spectral.yaml but **NOT listed** in Linting-rules.md: +- None found — all current .spectral.yaml rules are listed + +## Severity Alignment Issues + +| Rule | .spectral.yaml | Linting-rules.md | Design Guide Intent | +|------|----------------|-------------------|---------------------| +| camara-operationid-casing-convention | hint | error | MUST (error appropriate) | +| camara-schema-casing-convention | warn | warn | MUST in DG text (error may be more appropriate) | +| camara-path-param-id | warn | warn | Aligned | +| camara-operation-summary | warn | warn | Aligned | + +--- + +## Version-Sensitive Rules Summary + +### New in r4.x (not applicable to r3.4 repos) + +| ID | Rule | OWASP? | +|----|------|--------| +| DG-001 | String maxLength/enum MUST | Yes | +| DG-005 | Array maxItems MUST | Yes | +| DG-006 | Integer format MUST | Yes | +| DG-007 | Integer min/max MUST | Yes | +| DG-011 | contextCode naming | No | +| DG-014 | Error code SCREAMING_SNAKE_CASE | No | +| DG-018 | CONFLICT deprecated | No | +| DG-030 | info.description error responses section | No | +| DG-058 | Array items description | No | +| DG-069–078 | All OWASP rules | Yes | + +### Changed between versions + +| ID | Rule | r3.4 | r4.x | +|----|------|------|------| +| DG-010 | Discriminator on oneOf/anyOf | warn (recommended) | hint (deprecated) | +| DG-048 | operationId casing | hint | error (per Linting-rules.md) | + +### Implications for per-version Spectral rulesets + +- The r3.4 Spectral ruleset is effectively the current .spectral.yaml (no OWASP, no new MUSTs) +- The r4.x ruleset adds: OWASP rules (from tooling#95) + new CAMARA rules for DG-014, DG-058, and the missing Linting-rules.md rules +- OWASP api4 rules have current (warn) and target (error) severities — target is for 2027 releases + +--- + +## Prior Discussions + +Research of Commonalities and tooling issues/PRs (open and closed) for rationale behind identified discrepancies, gaps, and disabled rules. + +### Linting-rules.md discrepancies + +| Rule | Prior Discussion | Rationale | +|------|-----------------|-----------| +| camara-resource-reserved-words | [Commonalities#74](https://github.com/camaraproject/Commonalities/pull/74) (topic 13) | Documented as planned in initial linting ruleset PR (Oct 2023), confirmed as "to be developed in a later iteration." Never implemented. Gap from incremental rollout. | +| camara-path-param-id-morphology | [Commonalities#16](https://github.com/camaraproject/Commonalities/issues/16) (naming discussion) | Listed in Linting-rules.md but not in summary table. Same incremental rollout gap. No dedicated discussion about the missing implementation. | +| camara-property-casing-convention | [Commonalities#74](https://github.com/camaraproject/Commonalities/pull/74) (topic 6) | Documented with an explicit question: "Should it be lowerCamelCase in DG?" — indicating uncertainty about the design guide requirement. In [PR#466](https://github.com/camaraproject/Commonalities/pull/466) the summary table was updated to `Yes/Yes`, but the rule was still never added to .spectral.yml. | +| camara-enum-casing-convention | [Commonalities#74](https://github.com/camaraproject/Commonalities/pull/74) | Explicitly marked "tbd" since Dec 2023. Linting-rules.md notes "No clear requirement" in the design guide. Deliberate deferral — r4.1 has since added explicit SCREAMING_SNAKE_CASE for error codes (section 3.2), which partially resolves this for error enums but not for general enums. | +| camara-info-title | [Commonalities#74](https://github.com/camaraproject/Commonalities/pull/74) (topic 3); [#201](https://github.com/camaraproject/Commonalities/issues/201) / [#214](https://github.com/camaraproject/Commonalities/pull/214) | Raised as "Do we need rules for info-title; info-version?" in PR#74 review. #214 added DG guidelines (title without "API") but no linting rule followed. "tbd" since Dec 2023, unchanged. | +| camara-info-version-format | [Commonalities#74](https://github.com/camaraproject/Commonalities/pull/74) (topic 4) | Same as info-title. Acknowledged as potential new rule. Complexity noted: format allows wip, x.y.z-alpha.n, x.y.z-rc.n. "tbd" since Dec 2023, unchanged. | +| ~~camara-language-spelling~~ | — | Not a discrepancy — see "Other items" below | + +### Severity alignment issues + +| Rule | Prior Discussion | Rationale | +|------|-----------------|-----------| +| camara-operationid-casing-convention (hint vs error) | [Commonalities#76](https://github.com/camaraproject/Commonalities/issues/76) (severity discussion) | Internal inconsistency within Linting-rules.md itself: section 3 says `error`, section 4 summary says `Hint`. Implementation matches `hint`. Likely a documentation error in section 3 that was never caught. | +| camara-schema-casing-convention (warn vs MUST) | [Commonalities#76](https://github.com/camaraproject/Commonalities/issues/76) | General pattern: linting rules use lower severities than DG language suggests. MegaLinter only blocks on errors, so `warn` means "flagged but not blocking." Conservative choice during rollout. | + +### OWASP disabled rules + +All OWASP enablement decisions were individually reviewed in the master tracking issue [Commonalities#539](https://github.com/camaraproject/Commonalities/issues/539) with sub-issues [#548](https://github.com/camaraproject/Commonalities/issues/548) (api2), [#549](https://github.com/camaraproject/Commonalities/issues/549) (api3), [#551](https://github.com/camaraproject/Commonalities/issues/551) (api4), [#552](https://github.com/camaraproject/Commonalities/issues/552) (api8). Documentation landed via [PR#582](https://github.com/camaraproject/Commonalities/pull/582). + +| Category | Disabled Rules | Rationale | +|----------|---------------|-----------| +| Authentication (api2) | auth-insecure-schemes, jwt-best-practices, no-http-basic, no-api-keys-in-url | CAMARA uses only OpenID Connect (defined by ICM). These check for auth schemes never present in CAMARA specs — would never trigger. | +| Rate limiting (api4) | rate-limit, rate-limit-retry-after, rate-limit-responses-429 | Rate limiting is implementation/API Gateway specific. No IETF standard for rate limit headers yet. Not mandated by CAMARA. | +| CORS (api8) | define-cors-origin | CORS is implementation/API Gateway specific, not in API specs. | +| Error 500 (api8) | define-error-responses-500 | CAMARA guidelines do NOT recommend defining 500 responses — including them could leak implementation details. | +| Inventory (api9) | inventory-access, inventory-environment | Require vendor extensions (x-internal) and environment terms (staging, production) that CAMARA does not use. | +| Integer limit (api4) | integer-limit | For OAS 3.1 only — CAMARA uses OAS 3.0.x. The equivalent `integer-limit-legacy` IS enabled. | + +### OWASP severity downgrades (api4 rules at warn) + +Discussed in [Commonalities#539](https://github.com/camaraproject/Commonalities/issues/539) and [PR#582](https://github.com/camaraproject/Commonalities/pull/582). Rationale: these rules introduce new requirements that existing CAMARA APIs do not yet satisfy (many APIs lack maxLength on strings, format on integers, maxItems on arrays). Making them errors immediately would break all existing API linting runs. The `warn` severity provides a transition window. + +- The "Target CAMARA severity" column in Linting-rules.md explicitly targets `error` for 2027 meta-releases +- [tooling#95](https://github.com/camaraproject/tooling/pull/95) implements both current (`.spectral-owasp.yaml`) and target (`.spectral-owasp-target.yaml`) configurations +- Commonalities' own templates were fixed in [PR#590](https://github.com/camaraproject/Commonalities/pull/590) (CAMARA_common.yaml) and [PR#591](https://github.com/camaraproject/Commonalities/pull/591) (event-subscription-template.yaml) +- Active debate: [Commonalities#596](https://github.com/camaraproject/Commonalities/issues/596) challenges whether api4 constraints should apply to output properties at all (arguing OWASP api4 is about input validation only). Unresolved. + +### Other items + +| Topic | Prior Discussion | Finding | +|-------|-----------------|---------| +| camara-language-spelling | None | Marked "No/No" from day one (Dec 2023). Not a discrepancy — never intended for Spectral. Spelling and description quality checks are better handled by AI-based review tools, not pattern-based linting. [#545](https://github.com/camaraproject/Commonalities/issues/545) proposes spelling checks for markdown docs, not API YAML descriptions. | +| Tag naming convention (Title Case) | [Commonalities#80](https://github.com/camaraproject/Commonalities/issues/80) | #80 discussed operation tag usage but not Title Case enforcement. No discussion found about linting tag casing. | +| 403 response required | None | OWASP covers 401 only. No discussion found about requiring 403 via linting. | +| Linting-rules.md as source of truth | [Commonalities#482](https://github.com/camaraproject/Commonalities/issues/482) (open) | Acknowledges dual location of linting config. Proposes removing Commonalities copy, keeping tooling as single source. | +| Rule alignment umbrella | [Commonalities#532](https://github.com/camaraproject/Commonalities/issues/532) (open, Spring26) | "Review and correct artifacts to be conforming with linting rules" — focuses on template compliance, not the tbd rules. | + +### Cross-cutting finding + +The incremental rollout approach was established in [PR#74](https://github.com/camaraproject/Commonalities/pull/74) (Oct 2023), where rules were documented before implementation. Several rules from that era remain documented but not yet implemented. No existing issue specifically tracks the gap between documented rules in Linting-rules.md and implemented rules in .spectral.yml. + +--- + +## Statistics + +| Category | Count | +|----------|-------| +| Total rules extracted | 106 | +| Covered by existing Spectral | 26 | +| Covered by OWASP (tooling#95) | 17 | +| Covered by v0_6 validator only | 28 | +| Gaps (no implementation) | 20 | +| Multiple coverage | 15 (overlap) | +| r4.x-only rules | 19 | +| Changed between versions | 2 | +| Linting-rules.md discrepancies | 6 | +| Severity alignment issues | 2 | diff --git a/documentation/SupportingDocuments/CAMARA-Validation-Framework-Detailed-Design.md b/documentation/SupportingDocuments/CAMARA-Validation-Framework-Detailed-Design.md new file mode 100644 index 0000000..1eaafc1 --- /dev/null +++ b/documentation/SupportingDocuments/CAMARA-Validation-Framework-Detailed-Design.md @@ -0,0 +1,1338 @@ +# Validation Framework — Detailed Design + +**Status**: Work in progress +**Last updated**: 2026-03-18 + +> This document supplements the [Validation Framework Requirements](CAMARA-Validation-Framework-Requirements.md) with design and implementation detail for developers and architects. It is expected to migrate to the `tooling` repository alongside the implementation. + +--- + +## 1. Rule Metadata Model + +### 1.1 YAML Structure and Examples + +Rule metadata is expressed in YAML. Fields that are not constrained are omitted (omitted = applies in all contexts for that dimension). The primary use of the metadata is **post-filter and severity mapping**: engines run and produce findings, then the framework applies applicability and conditional level to interpret the results in the current context. + +```yaml +id: "042" # flat sequential ID, stable across engine changes +name: path-kebab-case # human-readable name +engine: spectral # spectral | yamllint | gherkin | python | manual +engine_rule: "camara-parameter-casing-convention" # native engine rule ID (if applicable) +hint: "Use kebab-case for all path segments: /my-resource/{resourceId}" + +applicability: # only list fields that constrain; omitted = no constraint + branch_types: [main, release] + trigger_types: [pr, dispatch] + # ... further conditions as needed + +conditional_level: + default: error # always present + overrides: # only if level varies by context + - condition: + target_release_type: [pre-release-alpha] + level: hint +``` + +Conditional level examples: + +```yaml +# "Test definition must be present" — hint by default, warn for RC/public of stable APIs +conditional_level: + default: hint + overrides: + - condition: + target_api_maturity: [stable] + target_release_type: [pre-release-rc, public-release] + level: warn + +# "Commonalities compliance check" — warn by default, suppressed for draft APIs +conditional_level: + default: warn + overrides: + - condition: + target_api_status: [draft] + level: off +``` + +### 1.2 Condition Evaluation + +``` +applicability match: + for each field in rule.applicability: + if field is array: context value must be IN the array (OR) + if field is range string: context value must satisfy the range expression + if field is boolean: context value must equal the field value + all fields must match (AND) + omitted fields are unconstrained (always match) + +conditional level: + for each override in rule.conditional_level.overrides (in order): + if override.condition matches context (same logic as applicability): + return override.level + return rule.conditional_level.default +``` + +### 1.3 Spectral Pass-Through Principle + +The framework uses Spectral's severity names (`error`, `warn`, `hint`) as its native level values. This gives identity mapping for the primary engine: + +| Spectral | Framework | Notes | +|----------|-----------|-------| +| `error` | `error` | Identity | +| `warn` | `warn` | Identity | +| `hint` | `hint` | Identity | +| `info` | `hint` | Mapped (rarely used) | +| `off` | `off` | Identity (disable rule) | + +Spectral rules already include `message` fields with fix guidance. Therefore, **Spectral rules that do not need context-dependent severity or applicability filtering do not require explicit framework metadata entries**. Their findings pass through with direct severity mapping and native messages. + +Framework metadata is only needed for Spectral rules when: +- The level should change based on context (e.g., error on release branch, hint on feature branch) +- The rule should be suppressed in certain contexts (applicability filtering) +- The fix hint should be overridden or augmented + +This minimizes the metadata surface: only rules with context-dependent behavior need explicit entries. + +The framework consumes Spectral output as structured data (JSON), not terminal text. This enables programmatic post-filtering, severity remapping, and merging with findings from other engines. + +Spectral does not resolve `$ref` references before linting — it validates the document as-is. Checks that depend on the content of referenced schemas (e.g., from CAMARA_common.yaml) require either a pre-bundled input spec or a Python implementation with explicit ref resolution. See section 3.1 (Spectral and `$ref` Interaction) for implications. + +For further Spectral-specific details, see [spectral-integration-notes.md](../reviews/spectral-integration-notes.md). + +### 1.4 Derived Context Fields + +Two per-API context fields are derived from content rather than declared in release-plan.yaml: + +**`target_api_maturity`**: Derived from `apis[].target_api_version` — `initial` if major version is 0 (v0.x.y), `stable` if major version >= 1 (vx.y.z). Determines which asset requirements apply per the API Readiness Checklist (e.g., stable public APIs require enhanced test cases and user stories). + +**`api_pattern`**: Detected from OpenAPI spec content — `request-response`, `implicit-subscription`, or `explicit-subscription`. Detection logic examines paths (subscription endpoints), callbacks, schema names, and content types. Multiple pattern-specific rule sets (REQ, IMP, EXP, EVT categories) depend on this classification. This detection is a cross-cutting capability used by many rules. + +### 1.5 Spectral Migration Potential + +Analysis of the deprecated `api_review_validator_v0_6.py` shows that approximately 40% of its checks are implementable as Spectral rules (single-file OpenAPI pattern matching), and an additional 15% could use Spectral custom JavaScript functions. This includes: + +- Mandatory error response checks (400, 401, 403) +- Server URL format validation +- info.version format validation +- License and security scheme validation +- ErrorInfo and XCorrelator schema presence +- Error response structure validation + +The main blocker for migrating ~20% of checks to Spectral is the dependency on `api_pattern` detection — Spectral cannot natively apply rules conditionally based on detected API type. These checks either need custom JS functions that embed the detection logic, or remain as Python checks that use `api_pattern` from the context. + +The Commonalities audit should evaluate each candidate check against the current design guide version before migration. + +### 1.6 Authoritative Schema References + +The execution context fields and their allowed values are defined by the following schemas, which are the authoritative sources: + +- **release-plan.yaml**: `artifacts/metadata-schemas/schemas/release-plan-schema.yaml` (in ReleaseManagement) +- **release-metadata.yaml**: `artifacts/metadata-schemas/schemas/release-metadata-schema.yaml` (in ReleaseManagement) + +The framework must accept exactly the values defined in these schemas. Any change to the schemas must be reflected in the framework's context model. + +--- + +## 2. Check Inventory Detail + +### 2.1 Inventory Status + +The per-rule inventory is based on a Commonalities audit that examined `CAMARA-API-Design-Guide.md` and `CAMARA-API-Event-Subscription-and-Notification-Guide.md` at both r3.4 and r4.1 versions, cross-referenced against the existing Spectral rules (17 CAMARA custom + core OAS), the OWASP rules from [tooling#95](https://github.com/camaraproject/tooling/pull/95), the `Linting-rules.md` maintained in Commonalities, and the deprecated `api_review_validator_v0_6.py` (80 checks). + +The audit identified 106 machine-checkable rules total: 26 already covered by existing Spectral, 17 by OWASP rules (tooling#95, not yet merged), 28 by the v0_6 validator only, and 20 gaps with no current implementation. Of the 106 rules, 19 are r4.x-only (not applicable to r3.4 repositories), 2 changed between versions, and 7 rules listed in `Linting-rules.md` are not yet implemented in the `.spectral.yaml` configuration. + +Remaining inventory work: +- **Existing rule classification**: Map each current Spectral rule to the framework metadata model (applicability, conditional level, hints). The existing Spectral severity levels are assumed valid for now; detailed severity review is deferred. + +### 2.2 Check Areas by Engine + +**Spectral (existing rules):** +- OpenAPI version enforcement (3.0.3) +- Naming conventions (see Appendix A for full list: paths, schemas, operationId, plus gaps for properties, enums, tags) +- Required descriptions (operations, parameters, responses, properties) +- Reserved words detection (language-specific + HTTP method names in resource paths) +- Security: no secrets in path/query parameters +- HTTP method validity, no request body on GET/DELETE +- Unused components detection +- Discriminator on oneOf/anyOf (deprecated in r4.x, now hint) +- Schema type attribute presence + +**Spectral (new rules needed):** +- info.version format (wip/alpha.n/rc.n/semver) +- info.title must not contain "API" +- info.contact and info.termsOfService must be absent +- externalDocs presence and format +- x-correlator header presence and pattern +- Error code format: not numeric, SCREAMING_SNAKE_CASE (r4.x), API_NAME.SPECIFIC_CODE pattern +- 403 response required on all operations +- Array items must have description (r4.x) +- Tag names: Title Case convention +- Property names: lowerCamelCase (listed in Linting-rules.md, not yet implemented) +- Enum values: SCREAMING_SNAKE_CASE (listed in Linting-rules.md, not yet implemented) +- Subscription API schemas: specversion enum, protocol enum, sink HTTPS, notification content-type + +**OWASP Spectral rules (from [tooling#95](https://github.com/camaraproject/tooling/pull/95), r4.x-only):** +- String limits: maxLength/enum/const (warn, target error) +- Array limits: maxItems (warn, target error) +- Integer limits: format + minimum/maximum (warn, target error) +- String restriction: format/pattern/enum/const (warn) +- Security: no credentials in URL, no HTTP scheme, write-restricted, read-restricted, short-lived access tokens, no numeric IDs, admin security unique +- Error responses: 401 required (error), error validation response (warn) +- Additional properties: constrained or disabled (warn) + +**Python (cross-field, cross-file, and context-dependent):** +- Server URL version consistency with info.version (cross-field) +- Version must be wip on main, must not be wip on release branches (context-dependent) +- release-plan.yaml non-exclusivity check (PR diff analysis) +- release-plan.yaml schema and semantic validation (existing, to be integrated) +- Error response structure: ErrorInfo schema compliance, $ref resolution (cross-schema) +- info.description: authorization and error response template sections (normalized text matching) +- Security scheme validation: openIdConnect named 'openId', notificationsBearerAuth for callbacks +- Scope naming: api-name:[resource:]action pattern, subscription-specific scopes +- Event type format: org.camaraproject.\.\.\ (subscription APIs) +- Subscription API structure: required operations, sinkCredential not in responses +- Test file existence and version alignment (cross-file) +- CHANGELOG format and link tag-locking (file content analysis) +- Common schema consistency across API files (cross-file; partially obsolete with bundling) +- License and x-camara-commonalities consistency across API files (cross-file) +- Filename conventions: kebab-case, matches api-name (filesystem) +- CONFLICT error code deprecated warning (r4.x) +- User story file existence in `documentation/API_documentation/` (conditional: mandatory for stable public APIs per API Readiness Checklist) + +**Gherkin-lint (test definition files):** +- Structural rules: named features and scenarios, unique names, non-empty backgrounds, scenarios with examples +- Step ordering: Given → When → Then, use `And` for repeated keywords +- Tagging: required tags, no restricted tags (@watch, @wip), no duplicates +- Formatting: indentation, no trailing spaces, no multiple empty lines +- Limits: max 50 scenarios per file, max 250 character names + +**Manual + prompt:** +- Data minimization compliance +- Meaningful description quality (beyond presence checks) +- User story adequacy (content quality; file existence is checked automatically for stable public APIs) +- Breaking change justification + +**Obsolete (handled by release automation):** +- API Readiness Checklist file management (files should no longer be in the repository) +- Release tag creation and format +- Version field replacement on release branches (wip → actual version) +- release-metadata.yaml generation +- README update with release information + +--- + +## 3. Bundling Pipeline + +### 3.1 Spectral and `$ref` Interaction + +#### Bundling vs full dereferencing + +The framework uses **bundling** (external ref resolution only), not full dereferencing: + +- **Bundling**: Resolves external `$ref` — pulls content from `code/common/`, `code/modules/`, and other local files into the document. Internal `$ref` (`#/components/schemas/...`, `#/components/responses/...`) are preserved. +- **Full dereferencing**: Resolves all `$ref` including internal ones, producing a flat document with zero `$ref` and massive duplication. The framework must **not** use full dereferencing. + +Preserving internal `$ref` ensures that: + +- Spectral rules checking component structure, `$ref` patterns, and `#/components/` organization continue to work on bundled output +- Bundled output remains readable and structurally equivalent to what reviewers expect +- No Spectral rule changes are needed between copy-paste and bundled models + +Any constraints on where API designers may use external vs internal `$ref` are defined in the [bundling design document](https://github.com/camaraproject/ReleaseManagement/pull/436), not by the validation framework. The framework enforces whatever ref patterns the design document specifies. + +#### Transition period + +During migration from copy-paste to the local copy model, both repository types coexist: + +- **Copy-paste repos**: All schemas inline. Spectral runs directly on source. No bundling needed. +- **`$ref` repos**: Spectral runs on bundled output. All external refs resolved, internal refs preserved. Structurally equivalent to copy-paste. + +No rule changes are needed between the two models — bundling normalizes external refs while preserving the internal structure that Spectral rules depend on. Rule IDs remain stable across the transition (flat namespace from Requirements section 5). + +#### Bundling is MVP scope + +Bundling support for `CAMARA_common.yaml` via `$ref` is **within MVP scope**. Commonalities 0.7.x requires updated common schemas, and the ability to consume them via `$ref` — with the framework handling bundling transparently — is the key additional value of the validation framework v1 for codeowners. This avoids repeating the difficult-to-validate copy-paste pattern. + +In the MVP, some parts may still be manual — providing the correct copy in `code/common/` and ensuring it matches the declared `commonalities_release` version. But the `$ref` option is available for early adopters, and the framework handles bundling when `$ref` is detected. Automated cache synchronization and strict version enforcement are post-MVP enhancements. + +### 3.2 Dependency Categories and File Mapping + +Three categories of shared schema dependencies exist, each with different characteristics: + +**Commonalities** (well-known, hardcoded in tooling): +The Commonalities repository provides shared schemas that are well-known to automation tooling. Currently two files are relevant: +- `CAMARA_common.yaml` — common data types, error responses, headers +- `notification-as-cloud-event.yaml` — CloudEvents notification schema + +These files, their source location in the Commonalities repository, and the mapping from `release-plan.yaml.dependencies.commonalities_release` to the correct version are built into the tooling. No per-repository configuration is needed. + +**ICM** (version compatibility constraint): +Identity and Consent Management schemas are currently contained within Commonalities files — there are no separate ICM files to cache. The `dependencies.identity_consent_management_release` in `release-plan.yaml` is a version compatibility constraint (potentially `>= x.y.z`) rather than a file-caching relationship. The exact nature of this dependency requires further discussion. + +**Sub-project commons** (extensible, declared per repository): +Sub-projects may define common schemas shared across their API repositories (e.g., a device API family sharing common device type definitions). These dependencies must be declarable without requiring changes to the automation tooling. Each sub-project dependency requires: +- Source repository +- Release tag or version +- Array of files to consume + +This extensible model requires a dependency declaration format, either within `release-plan.yaml` or as a separate manifest. The schema design is a follow-up topic for the bundling design document. + +#### File caching strategy + +Which files are cached in `code/common/` — demand-driven (only files actually `$ref`'d) vs declaration-driven (all files from declared dependencies) — is a sync mechanism concern defined in the [bundling design document](https://github.com/camaraproject/ReleaseManagement/pull/436), not a validation framework decision. + +The framework's checks are the same regardless: cached files must match their declared source version, and `$ref` targets must exist. + +### 3.3 Commonalities Version Matrix + +#### Active versions + +The framework must support validation rules that vary by Commonalities version. Active versions at the time of writing: + +- **r3.4** (Commonalities v0.6.x) — Fall25 meta-release, frozen, maintenance releases only +- **r4.x** (Commonalities v0.7.x) — Spring26 meta-release. r4.1 is the release candidate (available now); r4.2 is the upcoming public release and will replace r4.1 + +Within a version line (r4.x), the latest release is always authoritative. When r4.2 is available, r4.1 becomes obsolete — new releases must target r4.2. If a maintenance release r4.3 follows, it replaces r4.2 for validation purposes. + +Future Commonalities major versions (e.g., r5.x for v1.0.0) will add further version lines. The architecture must not assume a fixed number of active versions. + +#### Spectral ruleset selection (pre-selection) + +Each Commonalities major version line gets its own Spectral ruleset (e.g., `.spectral-r3.4.yaml`, `.spectral-r4.yaml`). The framework reads `commonalities_release` from `release-plan.yaml` and selects the matching ruleset before running Spectral. + +This avoids running contradicting rules from different Commonalities versions simultaneously, which would produce confusing Spectral output even if the results were filtered afterwards. The r3.4 ruleset is effectively frozen — only maintenance fixes. New rule development targets the current r4.x ruleset. + +#### Framework rule metadata (post-filter with conditionals) + +Framework rule metadata uses a single ruleset with `commonalities_release` range conditions for version-specific behavior. This is appropriate because: + +- Python checks are framework-controlled and do not produce confusing intermediate output +- Most framework rules apply across versions; only a minority are version-specific +- Duplicating shared rules into per-version files would create drift risk + +The Commonalities audit will identify which rules changed between r3.4 and r4.x. Those rules receive `commonalities_release` range conditions in their metadata. + +### 3.4 Placeholder Handling + +#### Current state + +The current `CAMARA_common.yaml` contains placeholder patterns (e.g., `{{SPECIFIC_CODE}}`) that have no defined resolution rules. These should be removed from Commonalities, with API repositories extending shared schemas via `allOf` instead (per the [bundling design document](https://github.com/camaraproject/ReleaseManagement/pull/436)). + +#### Future direction + +Placeholder replacement with defined values could be introduced together with bundling as part of a broader transformation pipeline. This could include dynamic variables such as `api_version`, `commonalities_release`, `commonalities_version`, effectively replacing the current "wip" and "/main/" substitutions done by the snapshot transformer. In this model, bundling + transformation (including placeholder replacement) would produce the release-ready artifact. + +### 3.5 Rule Architecture Integration + +Bundling integrates into the rule architecture (Requirements section 5) without requiring changes to the context model or rule metadata: + +- **Step assignment**: Each rule runs in either step 1 (pre-bundling validation) or step 3 (full validation). Assignment is an implementation detail — the framework knows which checks belong to which step. +- **No new context fields**: The context model from Requirements section 2.2 is sufficient. Whether external refs existed and were resolved is an implementation concern, not a rule applicability condition. +- **Cache sync is a check, not context**: The cache synchronization validation (section 3.2) produces findings (warning or error depending on profile). It is not a context field consumed by other rules. +- **Spectral ruleset selection**: The `commonalities_release` field (already in the context model) drives Spectral ruleset pre-selection (section 3.3). No additional metadata is needed. + +--- + +## 4. Artifact Surfacing Detail + +### 4.1 Workflow Artifact Naming + +- Bundled specs are uploaded as GitHub workflow artifacts with a naming convention that identifies the API name, branch, and commit SHA +- Bundled files include a header comment: `# For information only - DO NOT EDIT` +- Workflow artifact retention uses the GitHub default (90 days) + +### 4.2 Temporary Branch Model + +Workflow artifacts replace the temporary branch model (`/tmp/bundled/-`) for MVP. Temporary branches may be revisited post-MVP if reviewers need a browsable view of bundled content. + +### 4.3 "wip" Version Handling + +Bundling on `main` leaves `info.version` as-is — it contains `wip` as expected for unreleased code. Version replacement on release branches is handled by release automation (snapshot transformer), not by the validation framework's bundling step. The framework validates version correctness per branch type — this is an existing check (section 2.2), not a new bundling-specific requirement. + +--- + +## 5. Caller Workflow Design + +### 5.1 Token Resolution Strategy + +The framework uses a layered token resolution strategy. The order prioritizes consistent branding (all findings come from the same bot identity) over using whichever token happens to be available: + +1. **Snapshot context**: `camara-release-automation` app token — provided by the calling release workflow. The validation framework does not mint this token; it is passed in by the release automation caller. +2. **Validation default**: Dedicated validation app bot token — the framework mints an installation token for the current repository. This is the primary path for all PR and dispatch contexts, ensuring consistent bot identity on annotations and comments. +3. **Fallback**: `GITHUB_TOKEN` with write access — used when the validation app is not installed (e.g., dispatch in a fork, or repositories not yet onboarded to the app). Write capability is probed at runtime. +4. **Read-only**: Workflow summary and diagnostic artifacts only — when no write token is available. + +In normal operation, both upstream PRs and fork PRs show findings from the validation bot. The `GITHUB_TOKEN` fallback and read-only mode are degraded paths, not the expected default. + +### 5.2 Validation GitHub App + +A dedicated GitHub App handles write surfaces for validation. This is a **separate app** from `camara-release-automation` — it has a narrower permission scope and a different purpose. + +| Aspect | Validation App | camara-release-automation | +|--------|---------------|--------------------------| +| **Purpose** | PR annotations, comments, commit status | Release snapshot creation, branch management | +| **Permissions** | `checks: write`, `pull-requests: write`, `statuses: write` | `contents: write`, `workflows: write`, plus release management | +| **Commits/pushes** | Never | Yes (snapshot branches, tags, release assets) | +| **EasyCLA** | Not needed (no commits) | Required (commits to repos with CLA enforcement) | + +Org-level configuration: +- `vars.VALIDATION_APP_ID` — app ID (org variable) +- `secrets.VALIDATION_APP_PRIVATE_KEY` — app private key (org secret) + +The validation app is introduced from day one (MVP) to establish consistent bot identity and avoid caller workflow changes later. + +#### Relationship to v0 surfacing + +The v0 workflow surfaces findings via MegaLinter's built-in reporters (`GITHUB_COMMENT_REPORTER`, `GITHUB_STATUS_REPORTER`) and custom `actions/github-script` steps for release-plan validation. The v1 framework replaces all of these with its own unified surfacing layer. MegaLinter is no longer used as the orchestration layer. + +### 5.3 Trigger and Concurrency YAML + +#### PR trigger + +```yaml +on: + pull_request: + branches: + - main + - release-snapshot/** + - maintenance/** +``` + +- **`main`**: Standard development PRs. Profile: standard (or strict if release review PR detected) +- **`release-snapshot/**`**: Release review PRs created by release automation on snapshot branches. Profile: strict +- **`maintenance/**`**: Maintenance branch PRs. Profile: standard + +Default event types (`opened`, `synchronize`, `reopened`) are sufficient. The framework validates code content, not PR metadata — `edited` (title/body changes) is not needed. + +#### Dispatch trigger + +```yaml + workflow_dispatch: +``` + +Dispatch runs on whatever branch the user selects in the GitHub UI. The framework derives branch type, release context, and all validation parameters from the checked-out branch content. + +#### Concurrency + +```yaml +concurrency: + group: ${{ github.ref }}-${{ github.workflow }} + cancel-in-progress: true +``` + +Same model as v0: a new push to a PR branch cancels the previous validation run. Dispatch behaves identically — a second dispatch on the same branch cancels the first. + +### 5.4 Permissions Detail + +The caller workflow declares the maximum permission set. The reusable workflow inherits these as a ceiling — it cannot elevate above what the caller declares. + +```yaml +permissions: + checks: write + pull-requests: write + issues: write + contents: read + statuses: write + id-token: write +``` + +| Permission | Purpose | Fork PR behavior | +|------------|---------|-----------------| +| `checks: write` | Check run annotations (findings inline in PR diff) | Restricted by GitHub; validation app token used instead | +| `pull-requests: write` | PR review interactions | Restricted by GitHub; validation app token used instead | +| `issues: write` | PR comments (PRs use the Issues API for comments) | Restricted by GitHub; validation app token used instead | +| `contents: read` | Repository checkout | Available (read-only) | +| `statuses: write` | Commit status (per-check context in checks list) | Restricted by GitHub; validation app token used instead | +| `id-token: write` | OIDC token for tooling ref resolution (section 5.6) | May not be granted for fork PRs — see section 5.6 | + +For fork PRs, `GITHUB_TOKEN` write permissions are restricted by GitHub regardless of what the caller declares. The validation app token (section 5.1) bypasses this restriction because it is minted from the app's own credentials, independent of `GITHUB_TOKEN`. + +### 5.5 Input Design Detail + +The reusable workflow does not accept inputs that duplicate information derivable from the checked-out branch. This prevents contradictions where an input says one thing but branch content says another. + +**Example of the problem avoided**: If the workflow accepted a `release_type` input, a user could dispatch on `main` with `release_type: public-release` while `release-plan.yaml` on `main` says `target_release_type: pre-release-alpha`. The framework would need reconciliation logic, and the user would get confusing results. + +**Forbidden inputs**: `branch_type`, `release_type`, `api_status`, `commonalities_version`, `configurations` — all derivable from branch content or from the central configuration file (Requirements section 10). + +**Consequence for the caller workflow**: No per-repo inputs exist. All per-repo configuration (linting config subfolder, enabled features, rollout stage) lives in the central config file read by the reusable workflow. The caller workflow is identical across all repositories, with no `with:` block needed in standard operation. This makes it protectable via CODEOWNERS or rulesets — nobody ever needs to edit it. + +### 5.6 Ref Resolution + +#### OIDC-based ref resolution (primary) + +The reusable workflow resolves its own tooling repository and commit SHA via OIDC claims (`job_workflow_sha`), following the pattern established in tooling#121. This ensures all internal checkouts (linting config, shared actions at runtime) use the same tooling version that the caller specified. + +The caller workflow declares `id-token: write` to enable OIDC token generation. + +#### Hardcoded version fallback + +If OIDC token generation fails (e.g., fork PRs where `id-token: write` may not be granted), the reusable workflow falls back to its own hardcoded version tag (e.g., `v1`). This is the same pattern as v0's hardcoded `ref: v0`. + +This fallback is acceptable because: +- **Fork PRs**: Contributors do not need pinned-SHA ref resolution — the release version tag (`v1`) is correct for production validation +- **Feature branch testing**: Done by admins and rule developers who have write access, so OIDC works +- **Release automation**: Always triggered by codeowners with write access, so OIDC works + +The fallback means fork PR validation always uses the published version of the tooling, not a feature branch. This is the expected behavior — only admins test unreleased tooling versions. + +#### Break-glass override + +`tooling_ref_override` — a 40-character SHA input to the reusable workflow. Takes precedence over both OIDC and the hardcoded fallback. Documented as pilot/break-glass only. Same mechanism as release automation. + +#### Resolution order + +1. `tooling_ref_override` input (if set) — explicit SHA, highest priority +2. OIDC `job_workflow_sha` claim (if `id-token` available) — exact commit SHA +3. Hardcoded version tag in the reusable workflow (e.g., `v1`) — always available + +### 5.7 Caller Workflow Template + +The caller workflow is identical across all repositories: + +```yaml +name: CAMARA Validation + +on: + pull_request: + branches: + - main + - release-snapshot/** + - maintenance/** + workflow_dispatch: + +concurrency: + group: ${{ github.ref }}-${{ github.workflow }} + cancel-in-progress: true + +permissions: + checks: write + pull-requests: write + issues: write + contents: read + statuses: write + id-token: write + +jobs: + validation: + uses: camaraproject/tooling/.github/workflows/validation.yml@v1 + secrets: inherit +``` + +No `with:` block in standard operation. The caller is a thin pass-through that provides triggers, permissions, and concurrency. All validation logic, configuration, and surfacing are handled by the reusable workflow. + +### 5.8 Version Tagging and Secrets + +#### Version tagging + +The reusable workflow uses a floating version tag (`v1`) analogous to v0's `v0` tag. The tag is moved forward as the framework evolves within the v1 major version. Breaking changes (new required permissions, changed caller contract) require a new major version tag. + +#### Secrets + +The caller passes `secrets: inherit`. The reusable workflow uses: +- `GITHUB_TOKEN` — inherited, for checkout and fallback write surfaces +- Org secrets for validation app token minting (`VALIDATION_APP_PRIVATE_KEY`) — accessed via `secrets` context +- Org variables for app identity (`VALIDATION_APP_ID`) — accessed via `vars` context + +--- + +## 6. Rollout Implementation + +### 6.1 Why Separate Callers + +The v0 reusable workflow has a fundamentally different structure (MegaLinter-based, single job, different permissions and output model). A single caller with version switching would require complex conditional logic. + +Separate callers allow independent lifecycle: v0 can be removed per-repo after v1 is proven, without coordinating a simultaneous switch. GitHub rulesets can reference the v1 check name independently of v0. + +### 6.2 Central Config Alternatives Analysis + +**Rationale for config file over alternatives:** + +- **Org variable with repo list** (rejected): JSON arrays in org variables become unwieldy at 60+ repos and hit variable size limits. Not PR-reviewable. +- **Per-repo variable** (rejected): Requires touching each repository to enable. Violates UC-13 — central administration without per-repo configuration changes. +- **Caller version tag** (rejected): Would require editing the caller workflow per-repo, undermining the identical-caller-across-all-repos design (section 5.7). +- **Tooling config file** (chosen): Version-controlled, PR-reviewable, scalable. Adding a repo is one line in a YAML file. Can hold per-repo settings beyond enable/disable (linting config subfolder, rollout stage). Satisfies UC-13 — no per-repo config changes needed. + +#### Central config file schema + +The central config file lives in the tooling repository and maps each API repository to its rollout stage. Spectral ruleset selection is **not** a per-repo config field — it is derived from `commonalities_release` in the repository's own `release-plan.yaml` (section 3.3). + +```yaml +# validation-config.yaml in camaraproject/tooling +version: 1 +defaults: + stage: disabled # default for repositories not listed below +fork_owners: [hdamker, rartych] # GitHub users allowed to test in their forks +repositories: + QualityOnDemand: + stage: standard # stage 2: runs on PRs, standard profile + DeviceLocation: + stage: standard + ReleaseTest: + stage: standard + NetworkSliceBooking: + stage: advisory # stage 1: dispatch only +``` + +| Field | Type | Description | +|-------|------|-------------| +| `version` | integer | Schema version (currently `1`). Allows future schema evolution without breaking existing configs. | +| `defaults.stage` | enum | Default stage for unlisted repositories: `disabled`, `advisory`, `standard`. | +| `fork_owners` | array of strings | GitHub usernames allowed to run validation in their forks. When the workflow runs in a fork owned by a listed user, stage is overridden to `standard` regardless of the repository's upstream stage (section 8.2). | +| `repositories..stage` | enum | Per-repo rollout stage override. Same values as `defaults.stage`. | + +**Stage mapping** (see also Requirements section 10.3): + +| Stage | Config value | Behavior | +|-------|-------------|----------| +| 0 (dark) | `disabled` | Caller deployed but reusable workflow exits immediately | +| 1 (advisory) | `advisory` | Runs on dispatch only, advisory profile, nothing blocks | +| 2 (standard) | `standard` | Runs on PRs and dispatch, standard profile on PRs | +| 3 (blocking) | `standard` + ruleset | Same as stage 2; blocking is enforced by a GitHub ruleset, not the config file | + +**Extensibility**: Additional per-repo fields (e.g., `features`, optional overrides) can be added without a `version` bump — new fields are additive. Future candidates include `spectral_ruleset_override` and `extra_checks`. + +**Self-validation**: The reusable workflow validates the config file against a JSON Schema on every run. An invalid config file is a hard failure with an explicit error message naming the file and the problematic entry. This catches typos, unknown stage values, and schema drift before any validation logic runs. + +### 6.3 GitHub Rulesets for Blocking + +A new ruleset (org-level or per-repo) requires the v1 validation check to pass before PR merge. The pattern follows the existing `release-snapshot-protection` ruleset. + +- The ruleset references the v1 workflow by check name (workflow name or job name) +- The `camara-release-automation` app can be a bypass actor for automated release PRs that need to merge without validation +- Ruleset management can reuse the existing admin script pattern (`apply-release-rulesets.sh`) + +### 6.4 Rollout Sequence + +1. **Test repo**: `ReleaseTest` — full cycle through stages 0-3, validates all surfacing paths +2. **Template**: `Template_API_Repository` — ensures new repos get v1 caller from creation +3. **Pilot API repos**: 2-3 active repos with engaged codeowners +4. **Batch rollout**: Remaining repos, coordinated with v0 removal + +### 6.5 Feature Branch Testing + +Admins and rule developers test validation changes on feature branches before merging to main and tagging (UC-15). + +The caller workflow in a test repo is temporarily pointed at the feature branch: + +```yaml +uses: camaraproject/tooling/.github/workflows/validation.yml@feature-branch +``` + +Ref resolution (section 5.6) ensures internal checkouts match — admins have write access, so OIDC resolves the exact SHA. `tooling_ref_override` is available as break-glass for composite action changes not on the workflow branch. + +Rule developers can dispatch validation on existing release branches in a test repo while calling the feature-branch version of the reusable workflow. This validates rule changes against known-good content before merging (UC-10). + +No special framework support is needed — pinned refs are a standard GitHub Actions feature. The framework's only requirement is correct ref resolution (section 5.6). + +### 6.6 Caller Update Strategy + +The v1 caller workflow is deployed by copying from `Template_API_Repository` to each API repo. Since the caller is identical across all repos (section 5.7), deployment is a mechanical copy — no per-repo customization. + +Deployment can be batched using the existing admin tooling pattern (scripted multi-repo operations). The caller can be deployed to all repos at once in stage 0 (dark) — it has no effect until the repo is listed in the config file. + +### 6.7 Relationship to tooling#121 + +[tooling#121](https://github.com/camaraproject/tooling/pull/121) fixes ref consistency in the existing v0 reusable workflow. It validates the OIDC ref resolution pattern that v1 reuses and adds the `tooling_ref_override` break-glass input. tooling#121 does not change the v0 caller — callers still call `@v0`. The v1 reusable workflow reuses the same ref resolution pattern with the hardcoded version fallback (section 5.6). + +--- + +## 7. Release Automation Implementation + +### 7.1 Bundling and Snapshot Interaction + +The validation framework produces the bundled API specs as part of its validation pipeline (Requirements section 6.2, steps 2-3). These bundled specs are the same artifacts that become the release content on the snapshot branch. Bundling happens exactly once — during validation. Release automation consumes the bundled output rather than re-bundling independently. + +On the snapshot branch, source API definition files (which contain `$ref` to `code/common/` and `code/modules/`) are **replaced** with the bundled standalone specs produced by the validation framework. This is the "swap strategy" described in the [bundling design document](https://github.com/camaraproject/ReleaseManagement/pull/436): the familiar filename (`api-name.yaml`) is retained, but the content is the fully resolved, consumer-ready artifact. + +#### Handoff model + +The validation framework uploads bundled specs as **workflow artifacts** (artifact handoff). Release automation downloads these artifacts and commits them to the snapshot branch as part of snapshot creation. See section 9.7 for the detailed handoff sequence. + +This model was chosen over the alternative (validation creates the snapshot branch directly) because: +- The validation reusable workflow's checkout is ephemeral — there is no mechanism to hand uncommitted file changes to a different branch without committing and pushing from within the validation job +- Validation would need `contents: write` permission and knowledge of snapshot branch naming conventions, creating tight coupling to release automation internals +- The artifact model keeps validation stateless: it produces files and reports results, release automation owns repository state + +Bundling runs once, during validation. Release automation consumes the bundled output without re-bundling. + +#### Mechanical transformations after bundling + +After release automation downloads the bundled artifacts, the mechanical transformer applies version-specific changes on top of the bundled content: + +- `info.version` replacement (`wip` → calculated release version) +- Server URL version updates +- `x-camara-commonalities` version field +- Feature file version updates +- Link replacements + +These transformations are release automation's responsibility. The validation framework validates the source content (including bundling); the mechanical transformer produces the final release-ready content. + +#### Cache sync at snapshot time + +A cache synchronization mismatch is an error in strict profile, blocking snapshot creation. This ensures that `code/common/` content matches the declared `commonalities_release` version before the bundled output is produced and becomes immutable on the snapshot branch. + +### 7.2 Token and Findings Output for Pre-Snapshot + +**Token**: The `camara-release-automation` app token is passed by the release workflow. This is token priority 1 in the layered resolution (section 5.1). The validation framework does not mint this token; it receives it from the caller. + +**Findings output**: Validation findings are reported in the bot's response comment on the Release Issue. The comment includes a structured findings section with error and warning counts, individual findings with fix hints, and a link to the full workflow run for diagnostic artifacts. + +### 7.3 File Restriction Check + +The context field `is_release_review_pr` (Requirements section 2.2) serves dual roles: profile selection and applicability condition. The file restriction check is the only check currently using it as an applicability condition. + +```yaml +# File restriction check — release review PR only +id: "060" +name: release-review-file-restriction +engine: python +applicability: + is_release_review_pr: true +conditional_level: + default: error +description: "Release review PR may only modify CHANGELOG and README files" +hint: "Only CHANGELOG.md (or CHANGELOG/ directory) and README.md may be modified on the release review branch. API specs and other files are immutable on the snapshot branch." +``` + +### 7.4 Pre-Snapshot Invocation Detail + +Release automation invokes the validation framework on the same branch on which `/create-snapshot` was called. The framework reads `release-plan.yaml` from that branch to derive all context fields (target release type, API statuses, Commonalities version, etc.). + +The only distinction is the `mode` input — a reusable workflow input (alongside `tooling_ref_override` and `profile` from Requirements section 9.2) that tells the framework this is a pre-snapshot invocation rather than a dispatch or PR trigger. Release automation passes `mode: pre-snapshot` when calling the validation workflow. + +When `mode` is `pre-snapshot`, the framework: +- Sets `trigger_type` to `release-automation` +- Selects the strict profile +- Produces bundled API specs as output for consumption by release automation (section 7.1) +- Formats findings for inclusion in a Release Issue comment (not a PR comment) + +The detailed output model (findings format, artifact structure) is defined in section 9. + +--- + +## 8. End-to-End Processing Flow + +This section describes the reusable workflow's internal structure and the validation engine's processing pipeline. It covers what happens from the moment the workflow starts to the point where raw findings are collected — output formatting and surfacing are in section 9. + +### 8.1 Job Architecture + +The reusable workflow uses a **single-job design**. All steps run sequentially within one job. + +This differs from release automation's multi-job architecture. Release automation splits into separate jobs because its phases have fundamentally different conditional logic (trigger classification → state derivation → command validation → command execution, where each command is a separate job). The validation workflow has a single linear pipeline — context flows naturally between steps via environment variables and the shared file system within one job. Multi-job would require serializing the context object through job outputs and re-checking out the repository in each job, adding complexity without benefit. + +#### Step sequence + +| # | Step | Composite action | Skip condition | +|---|------|-----------------|----------------| +| 1 | Checkout repository content | — (inline) | Never | +| 2 | Resolve tooling ref and checkout tooling | `resolve-tooling-ref` | Never | +| 3 | Read central config and check stage | — (inline) | Never (exits here if `disabled`) | +| 4 | Build validation context | `build-validation-context` | Never | +| 5 | Pre-bundling validation | `run-pre-bundling-checks` | `is_release_review_pr` (section 8.6) | +| 6 | Bundling | `run-bundling` | No external `$ref` detected; or `is_release_review_pr` | +| 7 | Full validation | `run-full-validation` | `is_release_review_pr` (runs subset instead — section 8.6) | +| 8 | Post-filter and output | `process-findings` | Never (always produces at least a summary) | + +Steps 5–7 are the engine orchestration phase (section 8.4). Step 8 is the output pipeline (section 9). + +### 8.2 Checkout Strategy + +#### Repository content checkout + +```yaml +- uses: actions/checkout@v6 + with: + fetch-depth: 0 +``` + +Full history (`fetch-depth: 0`) is needed for: +- PR diff analysis: identifying which files changed (used by `release_plan_changed` detection and file restriction check) +- Branch type detection: examining `github.base_ref` for PR triggers + +For dispatch triggers, `fetch-depth: 1` would suffice, but a single checkout configuration avoids trigger-specific branching. + +#### Tooling checkout + +The tooling repository is checked out via sparse checkout at the resolved ref (section 5.6): + +```yaml +- uses: actions/checkout@v6 + with: + repository: camaraproject/tooling + ref: ${{ steps.resolve-ref.outputs.tooling_sha }} + sparse-checkout: | + linting/config + validation + shared-actions + path: .tooling +``` + +The checkout lands in `.tooling/` within the workspace. This path is used by all subsequent steps to locate linting configs, validation scripts, shared actions, and the central config file. + +The sparse checkout scope includes: +- `linting/config/` — Spectral rulesets (per Commonalities version), yamllint config, gherkin-lint config +- `validation/` — Python validation scripts, rule metadata, central config file +- `shared-actions/` — composite actions consumed at runtime + +#### Central config gate + +Immediately after the tooling checkout, the workflow reads the central config file (section 6.2) and looks up the current repository: + +1. Validate the config file against its JSON Schema +2. Extract the repository name from `github.repository` **without the owner prefix** (e.g., `camaraproject/QualityOnDemand` → `QualityOnDemand`) +3. Look up `repositories..stage` (fall back to `defaults.stage`) +4. **Fork override**: If the workflow is running in a fork (`github.repository_owner` is not `camaraproject` or `GSMA-Open-Gateway`) and the owner is listed in `fork_owners` → override stage to `standard`. If the owner is not listed → keep the resolved stage (typically `disabled` during early rollout, which exits the workflow) +5. If stage is `disabled`: exit the workflow with a notice in the summary ("Validation is not enabled for this repository") and set the overall result to `skipped` +6. If stage is `advisory` and trigger is `pull_request`: exit similarly ("Validation is in advisory mode — use workflow_dispatch to run") +7. Otherwise: continue, passing the resolved stage to subsequent steps + +The fork override (step 4) enables trusted testers to run full validation in their forks even before the upstream repository has been onboarded. This is especially useful during early rollout when most repos are still at `disabled`. Once upstream repos move to `standard`, the `fork_owners` list becomes less relevant — forks inherit the upstream stage. + +### 8.3 Context Builder + +The context builder assembles a unified context object from multiple sources. It follows the same principle as release automation's `BotContext`: all fields are always present in the output, with missing or inapplicable values defaulting to empty strings or `null`. Downstream consumers never need to handle missing keys. + +#### Branch type derivation + +| Pattern | Branch type | Source | +|---------|-------------|--------| +| `main` | `main` | `github.base_ref` (PR) or `github.ref_name` (dispatch) | +| `release-snapshot/**` | `release` | Same | +| `maintenance/**` | `maintenance` | Same | +| Everything else | `feature` | Same | + +For `pull_request` triggers, branch type is derived from the PR's **target branch** (`github.base_ref`), not the source branch. This determines which validation rules apply to the content being merged. + +For `workflow_dispatch`, it is derived from the checked-out branch (`github.ref_name`). + +#### Trigger type derivation + +| Source | Trigger type | +|--------|-------------| +| `github.event_name == 'pull_request'` | `pr` | +| `github.event_name == 'workflow_dispatch'` | `dispatch` | +| `mode` input == `pre-snapshot` | `release-automation` | + +The `mode` input (section 7.4) is the only way the trigger type `release-automation` is set. In standard caller workflow operation, `mode` is not set and trigger type comes from the GitHub event. + +#### Profile selection + +| Trigger type | Branch type | Condition | Profile | +|-------------|-------------|-----------|---------| +| `dispatch` | any | — | `advisory` | +| `pr` | `release` | `is_release_review_pr` = true | `strict` | +| `pr` | any | — | `standard` | +| `release-automation` | any | — | `strict` | + +If the `profile` input (Requirements section 9.2) is explicitly set, it overrides the auto-selected profile. This allows dispatch users to preview what a different profile would flag. + +#### release-plan.yaml parsing + +The context builder reads `release-plan.yaml` from the checked-out branch and validates it against the release-plan JSON Schema (section 1.6). Parsing extracts: + +- `target_release_type` — determines release context for all rules +- `dependencies.commonalities_release` — drives Spectral ruleset selection (section 3.3) +- `dependencies.identity_consent_management_release` — version constraint (section 3.2) +- Per-API fields: `api_name`, `target_api_version`, `target_api_status` + +If `release-plan.yaml` is absent (valid for repositories not yet using release automation, or feature branches), the context builder produces a minimal context with `target_release_type: null`. Release-plan-specific checks are skipped; Spectral and yamllint still run. If the file is present but invalid, schema violations are reported as findings and processing continues with whatever fields could be parsed. + +#### Derived per-API fields + +For each API declared in `release-plan.yaml`: + +- **`target_api_maturity`**: Derived from `target_api_version` — `initial` if major version is 0, `stable` if >= 1 (section 1.4) +- **`api_pattern`**: Detected from the corresponding OpenAPI spec file content in `code/API_definitions/` (section 1.4). Detection runs during context building because many downstream rules depend on it + +#### Spectral ruleset selection + +The Commonalities version declared in `release-plan.yaml` (`dependencies.commonalities_release`) determines which Spectral ruleset is used (section 3.3). The framework maps the release tag to a version line: + +- `commonalities_release` matching `r3.*` → `.spectral-r3.4.yaml` +- `commonalities_release` matching `r4.*` → `.spectral-r4.yaml` +- Future version lines added as new rulesets are created +- If `commonalities_release` is absent or unresolvable: default to the latest ruleset (currently r4.x) + +This selection is derived from the repository's own `release-plan.yaml`, not from the central config. Different branches of the same repo can target different Commonalities versions. + +#### Detection of PR-specific context + +- **`is_release_review_pr`**: True when the PR targets a `release-snapshot/**` branch (section 7.3) +- **`release_plan_changed`**: True when `release-plan.yaml` is in the PR diff. Detected via the GitHub API's changed files list for the PR. Relevant for the release-plan non-exclusivity check (section 2.2) + +#### Fork scenarios + +The validation workflow can run in forks as well as upstream: + +- **Fork dispatch**: A codeowner dispatches validation on their fork to check work before creating an upstream PR. Org secrets are not available in forks, so the validation app token cannot be minted — token resolution falls back to `GITHUB_TOKEN`, which has full write access within the fork's own context. +- **Fork-internal PRs**: PRs within a fork (feature branch → fork's main) trigger the caller if deployed. Same degraded token situation as fork dispatch. +- **Fork-to-upstream PRs**: The standard contribution path. The workflow runs in the upstream repo context (triggered by `pull_request`), so `github.repository` is the upstream repo. Org secrets are available to the reusable workflow. + +The context builder does not expose fork identity as a context field — no validation rule needs it. Fork vs. upstream is a workflow-level concern handled by token resolution (section 5.1). + +#### Context object structure + +```yaml +# Validation context — all fields always present +repository: "QualityOnDemand" # repo name without owner prefix +branch_type: "main" # main | release | maintenance | feature +trigger_type: "pr" # pr | dispatch | release-automation +profile: "standard" # advisory | standard | strict +stage: "standard" # from central config (disabled | advisory | standard) + +# Release context (from release-plan.yaml; null if absent) +target_release_type: "pre-release-rc" +commonalities_release: "r4.1" +icm_release: ">= r1.0" + +# PR-specific (null for dispatch) +is_release_review_pr: false +release_plan_changed: true +pr_number: 42 + +# Per-API contexts (array; empty if no release-plan.yaml) +apis: + - api_name: "qos-booking" + target_api_version: "1.0.0-rc.1" + target_api_status: "maintained" + target_api_maturity: "stable" # derived + api_pattern: "request-response" # detected from spec + spec_file: "code/API_definitions/qos-booking.yaml" + - api_name: "qos-on-demand" + target_api_version: "0.11.0-alpha.1" + target_api_status: "initial" + target_api_maturity: "initial" + api_pattern: "explicit-subscription" + spec_file: "code/API_definitions/qos-on-demand.yaml" + +# Workflow metadata +workflow_run_url: "https://github.com/camaraproject/QualityOnDemand/actions/runs/12345" +tooling_ref: "abc1234def5678..." +``` + +The context object is serialized as JSON and made available to subsequent steps via an environment variable or step output. + +### 8.4 Engine Orchestration + +Engines run **sequentially** within the single job. Each step captures its findings and passes them to the post-filter. + +Sequential execution is appropriate because: +- GitHub Actions steps within a job are inherently sequential +- Parallel execution would require multi-job with artifact passing — overhead that exceeds the time saved for a 1-3 minute pipeline +- Later steps may depend on earlier results (bundling produces input for Spectral) + +#### Step 5: Pre-bundling validation + +Runs on **source files** before any ref resolution. These checks must work regardless of whether the repo uses `$ref` or copy-paste: + +| Check | Engine | Target files | +|-------|--------|-------------| +| YAML validity | yamllint | `code/API_definitions/*.yaml` | +| `$ref` existence and pattern validation | Python | `code/API_definitions/*.yaml` | +| release-plan.yaml semantic checks | Python | `release-plan.yaml` | +| Cross-file checks not dependent on schema content | Python | Various | + +yamllint uses a configuration from the tooling checkout (`linting/config/.yamllint.yaml`). Invalid YAML produces error-level findings that block subsequent steps — there is no point running Spectral on syntactically invalid files. + +If all API spec files are syntactically valid and no external `$ref` is detected, step 6 (bundling) is skipped and step 7 runs on the source files directly. + +#### Step 6: Bundling + +Conditional — only runs when external `$ref` to `code/common/` or `code/modules/` is detected in at least one API spec file. + +The framework invokes an **external bundling tool** — it does not implement its own OpenAPI bundler. The tool must satisfy two requirements: + +1. **External ref resolution only** (see section 3.1): Resolve `$ref` to `code/common/`, `code/modules/`, and other local files. Preserve all internal `$ref` (`#/components/schemas/...`, `#/components/responses/...`). Full dereferencing must not be used. +2. **Source map production**: Produce a mapping from bundled output regions back to source file locations. This is needed for line number translation in the output pipeline (section 9.5). This is a selection criterion for tool evaluation — the chosen tool must either support source maps natively or be wrappable to produce them. + +The specific tool choice (e.g., redocly, swagger-cli, prance, custom wrapper) is deferred to implementation, evaluated against these two requirements. + +**Cache sync validation** runs as part of bundling: the content of `code/common/` files is compared against the expected content for the declared `commonalities_release` version. A mismatch produces a finding — `warning` in standard profile, `error` in strict profile. + +**On bundling failure** (unresolvable `$ref`, missing file, tool error): +- Report bundling error as a finding +- Skip step 7's Spectral run on bundled output (Spectral cannot lint incomplete specs) +- Proceed to step 8 with pre-bundling findings only +- Overall result: `error` (incomplete evaluation) + +#### Step 7: Full validation + +Runs on the **effective input** — bundled specs (from step 6) if bundling ran, otherwise source specs. + +| Check | Engine | Configuration | Output format | +|-------|--------|--------------|---------------| +| Spectral linting | Spectral | Version-selected ruleset (section 3.3) | JSON (`--format json`) | +| Test definition linting | gherkin-lint | `linting/config/.gherkin-lintrc` | Text or JSON | +| Cross-field consistency | Python | Rule metadata (section 1) | Common findings model | +| Version checks | Python | Context + spec content | Common findings model | +| Error response structure | Python | Context + spec content | Common findings model | +| API pattern-specific checks | Python | Context + spec content + `api_pattern` | Common findings model | + +**Spectral** is invoked with `--format json` to capture structured output for programmatic post-processing: + +```bash +spectral lint \ + --ruleset .tooling/linting/config/.spectral-r4.yaml \ + --format json \ + code/API_definitions/*.yaml \ + > spectral-output.json +``` + +The JSON output provides per-finding: `code` (rule name), `path` (file), `message`, `severity` (0=error, 1=warn, 2=info, 3=hint), `range.start.line`, `range.start.character`. + +**Python checks** produce findings directly in the common findings model (section 8.4.1). They receive the full context object and have access to the file system for cross-file analysis. + +**gherkin-lint** validates test definition files in `code/Test_definitions/`. Its findings are normalized into the common model by the framework. + +#### 8.4.1 Common Findings Model + +All engine outputs are normalized into a common findings format before post-filtering: + +```yaml +- rule_id: "042" # framework rule ID (sequential, stable) + engine: spectral # spectral | yamllint | gherkin | python + engine_rule: "camara-parameter-casing-convention" # native engine rule name + level: error # engine-reported level (before post-filter) + message: "Path segment 'qualityOnDemand' should be kebab-case" + path: "code/API_definitions/qos-on-demand.yaml" + line: 47 # line in source file (mapped back if bundled) + column: 5 # column (if available from engine) + api_name: "qos-on-demand" # which API this finding belongs to + hint: "Use kebab-case: /quality-on-demand/{sessionId}" +``` + +| Field | Source — Spectral | Source — yamllint | Source — gherkin-lint | Source — Python | +|-------|------------------|------------------|----------------------|-----------------| +| `rule_id` | Looked up from rule metadata by `engine_rule`; auto-assigned if no metadata | Looked up similarly | Looked up similarly | Set directly by check | +| `engine` | `"spectral"` | `"yamllint"` | `"gherkin"` | `"python"` | +| `engine_rule` | `code` field from JSON | Rule name from output | Rule name from output | Check function name | +| `level` | Mapped from severity integer: 0→error, 1→warn, 2→hint, 3→hint | Mapped from yamllint severity | Mapped from gherkin-lint severity | Set directly | +| `message` | `message` field | Error message text | Error message text | Set directly | +| `path` | `source` field | File path from output | File path from output | Set directly | +| `line` | `range.start.line` (0-indexed → 1-indexed) | Line number from output | Line number from output | Set directly | +| `column` | `range.start.character` | Column from output | Column from output (or null) | Set directly (or null) | +| `api_name` | Derived from file path | Derived from file path | Derived from file path | Set directly | +| `hint` | From rule metadata `hint` field (if present); otherwise engine `message` serves as hint | From rule metadata | From rule metadata | Set directly | + +Spectral rules **without** explicit framework metadata entries pass through with identity mapping (section 1.3): `rule_id` is auto-assigned, `hint` defaults to the engine's `message`, and the level maps directly. This means the check inventory does not need to be complete before the framework can run — new Spectral rules work immediately. + +### 8.5 Composite Action Boundaries + +Composite actions encapsulate logic that is independently testable, reusable across contexts, or benefits from encapsulation (e.g., Python scripts with dependencies). Simple shell commands and conditional logic remain as inline workflow steps. + +| Action | Purpose | Key inputs | Key outputs | +|--------|---------|-----------|-------------| +| `resolve-tooling-ref` | OIDC-based ref resolution (section 5.6) | `tooling_ref_override` | `tooling_sha` | +| `build-validation-context` | Context assembly (section 8.3) | Repository checkout path, tooling path, central config | Context JSON, stage, profile | +| `run-pre-bundling-checks` | Pre-bundling validation (section 8.4 step 5) | Context JSON, source files | Findings JSON | +| `run-bundling` | External ref resolution + cache sync (section 8.4 step 6) | Context JSON, source files, cache dir | Bundled specs, source map, findings JSON | +| `run-full-validation` | Spectral + gherkin + Python checks (section 8.4 step 7) | Context JSON, effective input files, ruleset path | Findings JSON | +| `process-findings` | Post-filter + output formatting (section 9) | Context JSON, all findings JSONs, token | Summary, annotations, comment, status | + +The `validate-release-plan` action from release automation is reused (enhanced to produce findings in the common model) rather than reimplemented. Other release automation shared actions (`resolve-tooling-ref`) are consumed directly. + +The boundary between "one large action" and "several small actions" per engine is an implementation choice. The table above shows the logical boundaries; implementation may merge or split actions based on testability and maintenance considerations. + +### 8.6 Release Review PR Short Circuit + +When `is_release_review_pr` is true, the processing flow is shortened: + +- **Skip**: Steps 5 (pre-bundling), 6 (bundling), and the Spectral/gherkin/most-Python portions of step 7 +- **Run**: CHANGELOG format check, README validation, file restriction check (section 7.3) +- **Rationale**: API specs are immutable on the snapshot branch — Spectral checks would be redundant. Release-plan and cache sync were already validated at snapshot creation time (Requirements section 11.2) + +This is implemented as a conditional skip within the engine orchestration steps, not a separate workflow path. The context builder sets `is_release_review_pr` and subsequent steps check it before executing. + +--- + +## 9. Output Pipeline + +This section covers step 8 of the processing flow (section 8.1): post-filter processing, output formatting, truncation, line number mapping, and error handling. It takes the raw findings from all engines and produces the user-visible output. + +### 9.1 Post-Filter Processing + +The post-filter evaluates each raw finding against the rule metadata (section 1) and the current validation context (section 8.3). It produces a filtered, severity-adjusted findings list ready for output formatting. + +**Processing steps per finding:** + +1. **Rule metadata lookup**: Match the finding to its framework rule metadata entry by `engine` + `engine_rule`. Python-produced findings already carry a `rule_id` and skip this step. + +2. **Applicability evaluation**: Apply the rule's `applicability` conditions against the current context (section 1.2). If any condition does not match, the finding is silently removed — it does not apply in this context. + +3. **Conditional level resolution**: Apply `conditional_level` overrides against the context (section 1.2). The first matching override determines the resolved level; if none match, the default level is used. The resolved level replaces the engine-reported level. If the resolved level is `off`, the finding is removed. + +4. **Pass-through**: Spectral rules without explicit framework metadata entries pass through with identity mapping (section 1.3). Their engine-reported severity becomes the resolved level, and the engine's `message` serves as the hint. This is the common case for most Spectral rules — the framework runs without requiring a complete metadata inventory. + +**Per-API evaluation**: Findings associated with a specific API (identified by `api_name`) are evaluated against that API's context fields (`target_api_status`, `target_api_maturity`, `api_pattern`). Repository-level findings (e.g., release-plan.yaml checks) are evaluated against the repository-level context. + +### 9.2 Profile Application and Blocking Decision + +After post-filtering, each finding has a resolved level. The active profile (section 8.3) determines which levels block: + +| Profile | Blocking levels | Typical context | +|---------|----------------|-----------------| +| `advisory` | None — nothing blocks | Dispatch (stage 1) | +| `standard` | `error` | PR validation (stage 2) | +| `strict` | `error` and `warn` | Pre-snapshot, release review PR | + +**Overall result** — one of three values: + +| Result | Meaning | +|--------|---------| +| `pass` | No blocking findings | +| `fail` | At least one finding at a blocking level | +| `error` | An engine failure prevented complete evaluation (section 9.6) — even if no blocking findings were collected, the result is uncertain | + +**Finding grouping for output**: Findings are grouped for display in the following order: by resolved level (error → warn → hint), then by API name, then by file path, then by line number. This ensures the most actionable items appear first. + +### 9.3 Output Formatting + +The framework produces output on multiple surfaces, each with different capabilities and token requirements. All surfaces are generated from the same filtered findings list. + +#### Workflow summary (always available) + +The workflow summary is the primary detailed surface. It requires no write token — it is written via `$GITHUB_STEP_SUMMARY` and is always available, even for fork PRs with read-only tokens. + +Structure: + +```markdown +## CAMARA Validation — {result} + +**Profile**: {profile} | **Branch**: {branch_type} | **Trigger**: {trigger_type} + +### Summary + +| API | Errors | Warnings | Hints | +|-----|--------|----------|-------| +| qos-booking | 0 | 2 | 1 | +| qos-on-demand | 1 | 0 | 3 | + +### Findings + +#### Errors + +| Rule | File | Line | Message | Hint | +|------|------|------|---------|------| +| 042 | qos-on-demand.yaml | 47 | Path segment should be kebab-case | Use: /quality-on-demand | + +#### Warnings +... + +#### Hints +... + +### Engine Status + +| Engine | Status | Findings | +|--------|--------|----------| +| Spectral | Completed | 4 | +| yamllint | Completed | 0 | +| Python | Completed | 3 | + +--- +Commit: abc1234 | Tooling: def5678 | [Full workflow run]({workflow_run_url}) +``` + +#### Check run annotations (write token required) + +One annotation per finding, mapped to the source file and line number: + +- Annotation level: `failure` for error, `warning` for warn, `notice` for hint +- Annotation message: includes the rule ID, engine rule name, and fix hint +- Annotation title: rule name from metadata (or engine rule name if no metadata) + +GitHub limits annotations to **50 per step**. If more findings exist: +- Prioritize: errors first, then warnings, then hints +- Note the truncation in the workflow summary: "Showing 50 of N findings as annotations. See workflow summary for the complete list." + +#### PR comment (write token required) + +A concise summary comment on the PR — not a full findings list. Links to the workflow summary for details. + +```markdown +### CAMARA Validation — {result} + +{errors} errors, {warnings} warnings, {hints} hints | Profile: {profile} + +[View full results]({workflow_run_url}) +``` + +The comment uses a create-or-update pattern with a marker (``) to avoid duplicate comments on subsequent pushes. Each new push updates the existing comment rather than creating a new one. + +**Pre-snapshot context**: When invoked by release automation (`mode: pre-snapshot`), findings are formatted for inclusion in the bot's Release Issue comment (section 7.2) rather than as a standalone PR comment. The framework returns the formatted findings section to the calling workflow. + +#### Commit status (write token required) + +Commit statuses provide at-a-glance results in the PR's checks list: + +- **Overall status**: Context `CAMARA Validation`, state = `success` / `failure` / `error` +- **Per-engine statuses** (optional, for diagnostics): `CAMARA Validation / Spectral`, `CAMARA Validation / yamllint`, etc. — each shows the engine's individual pass/fail state + +The overall status is the one referenced by GitHub rulesets for blocking (stage 3). + +### 9.4 Truncation Strategy + +GitHub limits workflow step summaries to **1 MB per step** and **1 MB total per job**. The framework must stay within this limit. + +**Approach**: Estimate the rendered size during summary generation. If the cumulative size approaches 900 KB: + +1. Show all **errors** first — never truncated +2. Show **warnings** up to the remaining budget +3. If budget exhausted before all warnings are shown: display a count of remaining findings and link to the full report +4. **Hints** are shown only if budget permits after errors and warnings + +```markdown +> Showing 85 of 214 findings. Full Spectral output available in +> [workflow artifacts]({artifact_url}). +``` + +The full Spectral JSON output and the complete findings list (all engines) are always uploaded as **workflow artifacts** regardless of summary truncation. These artifacts are the authoritative complete record. + +### 9.5 Line Number Mapping + +When Spectral runs on bundled output (section 8.4, step 7), finding line numbers reference the bundled file, not the original source file. Annotations and summary tables must show source file locations to be actionable for developers. + +**Requirement on the bundling tool**: The external bundling tool (section 8.4, step 6) must produce — or be augmented to produce — a source map that records which regions of the bundled file originated from which source file and line range. This is a selection criterion for bundling tool evaluation, alongside external-ref-only resolution (see section 3.1). + +**Design choice**: Content pulled in from external refs (e.g., schemas from `CAMARA_common.yaml`) maps back to the **`$ref` line in the source file**, not to the external file itself. The `$ref` declaration is the actionable location — it is the line the developer controls. If Spectral reports an issue at line 247 of the bundled file, and that region was pulled from an external ref declared at line 15 of the source file, the finding is reported at source line 15. + +**Scope**: Line number mapping is only needed for repositories that use `$ref` to `code/common/` or `code/modules/`. Copy-paste repositories have identity mapping — source and effective input are the same file. + +The source map format is an implementation detail. A line-offset table per external ref insertion point is sufficient for MVP. + +### 9.6 Error Handling + +**Design principle**: Always surface what succeeded; never silently skip. If an engine fails, the failure is reported explicitly and remaining engines continue. + +#### Engine failure + +When an engine crashes (Spectral error, Python exception, missing dependency): + +1. Catch the failure in the step (non-zero exit code or exception) +2. Record an engine-level error finding: + ```yaml + - rule_id: "engine-failure" + engine: spectral # the engine that failed + engine_rule: null + level: error + message: "Spectral exited with code 2: Cannot read ruleset file" + path: null + line: null + api_name: null + hint: "Check the workflow log for details" + ``` +3. Continue with remaining engines +4. Set overall result to `error` (distinct from `fail` — signals incomplete evaluation) +5. Workflow summary explicitly lists which engines succeeded and which failed (engine status table in section 9.3) + +Following the release automation pattern: error messages are shown in code blocks, immediately visible, not collapsed. The workflow run URL links to full logs. + +#### Config file missing or invalid + +- Config file **missing** from the tooling checkout: hard failure with an explicit error in the summary. This is a tooling repository issue, not a per-repo issue. +- Config file **invalid** (bad YAML, unknown stage value, schema violation): hard failure naming the config file and the problematic entry. The framework does not proceed with potentially incorrect configuration. + +#### release-plan.yaml missing or invalid + +- **Missing**: The context builder produces a minimal context with `target_release_type: null`. Release-plan-specific checks (version consistency, non-exclusivity, dependency validation) are skipped. Spectral and yamllint still run on source files. This is not an error — repos without release-plan.yaml (or feature branches that haven't added one) are valid. +- **Invalid** (present but fails schema validation): Schema violations are reported as findings. The context builder extracts whatever fields it can and continues with a partial context. Remaining checks run with the available context. + +#### Bundling failure + +When bundling fails (unresolvable `$ref`, missing file in `code/common/`, tool crash): + +1. Report the bundling error as a finding (error level) +2. Skip Spectral on bundled output (step 7 Spectral) — cannot lint incomplete specs +3. Proceed to post-filter with pre-bundling findings (step 5 results) and the bundling error +4. Set overall result to `error` if the active profile would have required checks on bundled output + +Pre-bundling findings are still valuable — yamllint and ref pattern checks provide feedback even when bundling fails. + +#### Token minting failure + +When the validation app token cannot be minted (app not installed, secret missing, API error): + +1. Follow the layered token resolution (section 5.1) — fall back to `GITHUB_TOKEN`, then to read-only +2. Log the degraded state in the workflow summary header: "Write surfaces unavailable — showing findings in workflow summary only" +3. **Never** fail the workflow because of a token issue — validation results are always produced, only the surfacing degrades +4. Check run annotations, PR comments, and commit statuses are silently skipped when no write token is available + +#### Failure mode summary + +| Failure | Behavior | Overall result | +|---------|----------|---------------| +| Engine crash | Record engine-failure finding, continue with remaining engines | `error` | +| Config file missing/invalid | Hard failure, explicit error in summary | `error` (workflow exits) | +| release-plan.yaml missing | Minimal context, skip release-plan checks, run Spectral/yamllint | Normal (`pass`/`fail`) | +| release-plan.yaml invalid | Report violations as findings, partial context, continue | Normal (`pass`/`fail`) | +| Bundling failure | Report error, skip bundled-output checks, show pre-bundling findings | `error` | +| Token minting failure | Degrade surfacing, never fail the workflow | Normal (`pass`/`fail`) | + +### 9.7 Bundled Spec Artifacts + +Bundled API specs produced in step 6 (section 8.4) are a distinct output category from findings — they are build artifacts, not validation results. This section covers how they are persisted and consumed. + +#### Artifact upload + +After bundling completes (step 6), the bundled spec files are uploaded as **GitHub workflow artifacts**, regardless of whether validation passes or fails. Bundled specs are useful for review even when findings exist. + +Artifact naming follows a convention that identifies the content: + +``` +validation-bundled-specs-{commit-sha-short} +``` + +The artifact contains one bundled YAML file per API (retaining the original filename, e.g., `qos-booking.yaml`) plus the source map file produced by the bundling tool (section 9.5). + +Artifacts use the GitHub default retention period (90 days). They are available for download from the workflow run's artifact list. + +#### User surfacing + +The workflow summary (section 9.3) includes a link to the bundled spec artifacts when bundling ran: + +```markdown +### Bundled Specs + +Bundled standalone API specs are available as [workflow artifacts]({artifact_url}). +``` + +This gives PR reviewers visibility into the bundled output — they can download and inspect the fully resolved specs to verify that `$ref` resolution produced the expected result. For copy-paste repositories (no bundling), this section is omitted. + +#### Release automation handoff + +When release automation invokes validation with `mode: pre-snapshot`, it consumes the bundled spec artifacts to populate the snapshot branch. The handoff uses the **artifact model** (section 7.1): + +1. Release automation calls the validation reusable workflow with `mode: pre-snapshot` +2. Validation runs the full pipeline (context → pre-bundling → bundling → full validation → output) +3. Validation uploads bundled specs as workflow artifacts +4. Validation returns its overall result (`pass` / `fail` / `error`) to the caller +5. If result is `pass`: release automation downloads the bundled spec artifacts and commits them to the snapshot branch, replacing the source files that contained `$ref` +6. If result is `fail` or `error`: release automation does not create the snapshot — findings are reported in the Release Issue comment + +The validation framework does not need `contents: write` permission and has no knowledge of snapshot branch naming. It produces files and reports results; release automation decides what to do with them. This keeps a clean separation: validation is stateless, release automation owns the repository state. + +**For non-pre-snapshot runs** (PR and dispatch triggers): bundled specs are uploaded as artifacts for reviewer inspection only. No branch creation or file replacement occurs — the artifacts are informational. + +--- + +## Appendix A: Naming Conventions + +Complete naming convention rules from CAMARA-API-Design-Guide.md and CAMARA-API-Event-Subscription-and-Notification-Guide.md, with current Spectral rule coverage. + +| Element | Convention | Example | DG Section | Spectral Rule | Status | +|---------|-----------|---------|------------|---------------|--------| +| Paths (URLs) | kebab-case | `/customer-segments` | 5.7.1 | camara-parameter-casing-convention (error) | Implemented | +| Path parameters | {entityId} form | `{userId}`, `{accountId}` | 5.7.1 | camara-path-param-id (warn) | Implemented (morphology rule missing) | +| Schemas | PascalCase | `ErrorInfo`, `DeviceResponse` | 5.8.1 | camara-schema-casing-convention (warn) | Implemented | +| operationId | camelCase | `helloWorld`, `retrieveLocation` | 5.7.2 | camara-operationid-casing-convention | Implemented (severity: hint vs error in Linting-rules.md) | +| Properties | lowerCamelCase | `sessionId`, `phoneNumber` | 5.7.4 | camara-property-casing-convention (error) | Listed in Linting-rules.md, not in .spectral.yaml | +| Enum values | SCREAMING_SNAKE_CASE | `INVALID_ARGUMENT`, `PERMISSION_DENIED` | 3.2 | camara-enum-casing-convention (info) | Listed in Linting-rules.md, not in .spectral.yaml | +| Error codes | SCREAMING_SNAKE_CASE | `UNAUTHENTICATED`, `NOT_FOUND` | 3.2 | — | Gap (r4.x explicit requirement) | +| API-specific error codes | API_NAME.SPECIFIC_CODE | `CARRIER_BILLING.PAYMENT_DENIED` | 3.2.1 | — | Gap | +| Tags | Title Case with spaces | `Quality On Demand` | 5.7.3 | — | Gap | +| Headers | kebab-case | `x-correlator` | 5.8.5 | — | Implied by convention, no rule | +| API name | kebab-case | `location-verification` | 1.2 | — | v0_6 validator only | +| Scope names | kebab-case with : separators | `qod:sessions:create` | 6.6 | — | v0_6 validator only | +| Event type | org.camaraproject.\.\.\ | `org.camaraproject.device-roaming-subscriptions.v1.roaming-status` | Event Guide 3.1 | — | Gap | +| Examples (named) | SCREAMING_SNAKE_CASE | `SESSION_CREATION_EXAMPLE_WITH_DEVICE_RESPONSE` | OAS 3.0.3 | — | Gap | diff --git a/documentation/SupportingDocuments/CAMARA-Validation-Framework-Requirements.md b/documentation/SupportingDocuments/CAMARA-Validation-Framework-Requirements.md new file mode 100644 index 0000000..6b968c5 --- /dev/null +++ b/documentation/SupportingDocuments/CAMARA-Validation-Framework-Requirements.md @@ -0,0 +1,523 @@ +# Validation Framework — Requirements + +**Status**: Work in progress +**Last updated**: 2026-03-17 + +> For design and implementation detail, see [Validation Framework — Detailed Design](CAMARA-Validation-Framework-Detailed-Design.md). + +--- + +## 1. Use Cases + +### 1.1 Contributor (including dev agents) + +UC-01: Check a fix/feature branch on a fork before creating a PR (via GitHub workflow dispatch). + +UC-02: See validation results of a PR check in an understandable form, with hints how to resolve failures. + +UC-03: Run most validation rules locally (on local clones, via scripts). + +UC-04: Check the code base before starting work, to be aware of existing issues (relevant for dev agents and new contributors). + +### 1.2 Codeowner + +UC-05: Rely on PR checks — if green, the branch fulfills formal requirements for the intended release as defined in release-plan.yaml. + +UC-06: Check at any time if the main branch is ready to be released, or which issues remain open (via workflow dispatch on main). + +UC-07: See the consequences of a release-plan.yaml change — a PR that only changes release-plan.yaml triggers a re-run, surfacing newly relevant rules. + +### 1.3 Release Automation + +UC-08: Use validation as a strict gate before creating a release snapshot. + +UC-09: Use validation to check the release review PR (strict, also restricting allowed changes to README and CHANGELOG). + +### 1.4 Rule Developer + +UC-10: Run rule changes against existing release branches as regression tests or to assess impact. + +UC-11: Define rule applicability conditions and conditional severity based on context (see section 2.2). + +UC-12: Define fix hints for failed checks. + +### 1.5 CAMARA Admin + +UC-13: Introduce the validation framework incrementally, with central configuration of enabled functionality per repository (no per-repo config files that create change noise). + +UC-14: Validate that an API repository is correctly configured for the validation framework. + +UC-15: Test the validation framework on feature branches with pinned refs (same mechanism as release automation). + +### 1.6 Release Manager (independent) + +UC-16: See a dashboard indicating compliance of API repository main branches against their declared intent. *(Independent work — Release Progress Tracker can collect last validation run on merged PR or main branch, trigger validation as camara-release-automation bot.)* + +UC-17: Trigger a validation check on selected repositories to update dashboard status. *(Independent — requires validation to be operational but does not depend on specific framework version.)* + +--- + +## 2. Validation Contexts + +### 2.1 Validation Profiles + +Three profiles control blocking behavior. The framework selects the profile automatically based on how validation is invoked. + +| Profile | Blocking behavior | Typical trigger | +|---------|-------------------|-----------------| +| **advisory** | Nothing blocks; all findings shown | Local run, dispatch | +| **standard** | `error` blocks; `warn` and `hint` shown | PR (fork-to-upstream or upstream branch) | +| **strict** | `error` and `warn` block; `hint` shown | Release automation gates (snapshot, release review PR) | + +### 2.2 Rule Model + +Each rule defines two things: + +1. **Applicability conditions** — when the rule fires at all +2. **Conditional level** — what severity it produces, which can vary by context + +The profile (section 2.1) then determines which levels block. This separates three concerns cleanly: + +- **Rule developer** controls: when a rule applies and what level it produces +- **Framework** controls: which profile is active +- **Profile** controls: which levels block + +#### Levels + +| Level | Meaning | Blocked by profile | +|-------|---------|-------------------| +| **error** | Must be fixed | standard, strict | +| **warn** | Should be fixed | strict | +| **hint** | Recommendation, never blocking | *(none)* | +| **off** | Suppressed, finding not shown | *(n/a)* | + +#### Applicability conditions + +A rule is skipped silently if its conditions don't match the current context. Multiple fields are combined with AND; multiple values within an array field are combined with OR. + +| Field | Type | Values | Source | +|-------|------|--------|--------| +| `branch_types` | array | `main`, `release`, `maintenance`, `feature` | Target branch (PR) or checked branch (dispatch) | +| `trigger_types` | array | `pr`, `dispatch`, `release-automation` | How validation was invoked | +| `target_release_type` | array | `none`, `pre-release-alpha`, `pre-release-rc`, `public-release`, `maintenance-release` | `repository.target_release_type` in release-plan.yaml | +| `target_api_status` | array | `draft`, `alpha`, `rc`, `public` | `apis[].target_api_status` in release-plan.yaml (per-API) | +| `target_api_maturity` | array | `initial` (0.x.y), `stable` (x.y.z, x>=1) | Derived from `apis[].target_api_version` | +| `api_pattern` | array | `request-response`, `implicit-subscription`, `explicit-subscription` | Detected from OpenAPI spec content (per-API) | +| `commonalities_release` | string | Range expression, e.g. `">=r3.4"` | `dependencies.commonalities_release` in release-plan.yaml | +| `release_plan_changed` | boolean | `true`, `false` | Whether release-plan.yaml is in the PR diff (PR trigger only) | + +Range comparison for `commonalities_release` uses `packaging.specifiers` (Python) with release tags normalized to comparable values. + +#### Conditional level + +`default` is always present. `overrides` is a list of `{condition, level}` pairs evaluated in order; first match wins. Conditions use the same field/value model as applicability (AND across fields, OR within arrays). The level `off` can be used in overrides to suppress a finding in specific contexts. + +#### Execution context + +The framework constructs a context object at runtime from the trigger, branch, and release-plan.yaml content. Rules are evaluated per-API for API-specific checks (a repository with three APIs at different statuses produces three evaluation contexts). + +| Context field | Type | Source | +|---------------|------|--------| +| `branch_type` | string | Derived from branch name | +| `trigger_type` | string | Workflow event | +| `release_plan_changed` | boolean | PR diff (PR trigger only) | +| `target_release_type` | string | release-plan.yaml `repository.target_release_type` | +| `commonalities_release` | string | release-plan.yaml `dependencies.commonalities_release` | +| `icm_release` | string | release-plan.yaml `dependencies.identity_consent_management_release` | +| `target_api_status` | string | release-plan.yaml `apis[].target_api_status` (per-API) | +| `target_api_maturity` | string | Derived from `apis[].target_api_version` (per-API) | +| `api_pattern` | string | Detected from OpenAPI spec content (per-API) | +| `is_release_review_pr` | boolean | Detected by framework (profile selection + applicability condition for release-review-specific checks) | + +### 2.3 Execution Contexts + +The validation framework must support these execution contexts: + +| Context | Trigger | Profile | Token | Notes | +|---------|---------|---------|-------|-------| +| **PR (fork-to-upstream)** | `pull_request` event | standard | read-only | Default GitHub behavior: fork PRs get read-only GITHUB_TOKEN regardless of author's repo permissions. Limits output options (no check run annotations, no PR comments via token) | +| **PR (upstream branch)** | `pull_request` event | standard | write | PR from upstream branch. Note: codeowners should normally use forks too | +| **Dispatch (upstream repo)** | `workflow_dispatch` | advisory | write | Main (default), maintenance, release branches. Warning on non-standard branches | +| **Dispatch (fork repo)** | `workflow_dispatch` | advisory | write (fork scope) | Fork owner triggers on own fork. Inherited — no extra work if dispatch trigger exists | +| **Local** | CLI / script | advisory | n/a | No GitHub context; subset of rules | +| **Release automation: snapshot** | Called by release workflow | strict | write (app token) | Gate before snapshot creation (section 11.1) | +| **Release automation: review PR** | `pull_request` event (push to PR branch) | strict | write or read-only | Same trigger as normal PR; profile is strict based on release review PR detection (section 11.2) | + +**Profile is independent of token permissions.** A fork-to-upstream PR gets the same **standard** profile as an upstream-branch PR. The read-only token only limits *how results are surfaced* (e.g. no check run annotations via GITHUB_TOKEN), not validation strictness. + +**Dispatch is always advisory.** Dispatch runs have nothing to block (except the workflow itself). They surface errors, warnings, and hints for the user to review. + +**Who can dispatch**: Standard GitHub permission model — write access required. On upstream: codeowners and org-wide teams (admin, release_reviewer). On forks: fork owner. + +--- + +## 3. MVP Scope + +The MVP replaces `pr_validation` v0 and delivers the minimum useful validation on PRs and dispatch. + +### MVP includes + +- UC-01 through UC-07 (contributor and codeowner use cases) +- UC-13 (incremental rollout with central config) +- UC-15 (feature branch testing with pinned refs) +- Profiles: advisory and standard +- Execution contexts: PR (fork-to-upstream), PR (upstream branch), dispatch (upstream repo), dispatch (fork repo) +- Existing Spectral rules and YAML linting +- Understandable output with fix hints +- Caller workflow with all triggers from day one (PR, dispatch), deployed alongside `pr_validation` v0 +- Central enable/disable per repository (no per-repo config files) + +### Post-MVP priorities + +- UC-08, UC-09 (release automation strict gates, section 11) — high priority, requires strict profile; depends on PR integration being operational first +- UC-10 (regression testing against release branches) — rule developer tooling +- UC-14 (repository configuration validation) — admin tooling +- Automated cache synchronization for `code/common/` and strict version enforcement — bundling itself (ref resolution via `$ref`) is MVP scope (section 6) +- Python-based consistency checks beyond what Spectral covers — may partially land in MVP + +### Independent work (not sequenced with MVP) + +- UC-16, UC-17 (dashboard, section 12.3) — Release Progress Tracker can collect last validation run results and trigger validation with appropriate permissions + +### MVP rollout model + +1. Deploy caller workflow with all triggers to repositories alongside `pr_validation` v0; reusable workflow starts as no-op for most triggers +2. Enable advisory checks on dispatch +3. Enable standard checks on PRs +4. Enable blocking via GitHub rulesets (errors only) +5. Expand repository set incrementally via central include list + +--- + +## 4. Check Inventory + +### 4.1 Authoritative Sources + +The validation rules are derived from the following upstream documents. Rules must be validated against the version of these documents matching the declared `commonalities_release` in release-plan.yaml. + +**Commonalities** (version-dependent — r3.4 = v0.6.x, r4.1/r4.2 = v0.7.x): +- `CAMARA-API-Design-Guide.md` — API design rules, versioning, error handling, naming conventions +- `CAMARA-API-Event-Subscription-and-Notification-Guide.md` — subscription and notification patterns + +**Release Management:** +- `release-plan-schema.yaml` — field definitions and allowed values for release-plan.yaml +- `release-metadata-schema.yaml` — field definitions for generated release-metadata.yaml on release branches +- `api-readiness-checklist.md` — release asset requirements matrix by API status and maturity + +The `release-plan-schema.yaml` and `release-metadata-schema.yaml` are the authoritative source for execution context field values. The framework must accept exactly the values defined in these schemas. + +### 4.2 Check Engines + +| Engine | Scope | Current status | +|--------|-------|----------------| +| **Spectral** | OpenAPI structural and naming rules against `code/API_definitions/*.yaml` | Active in `pr_validation` v0 (~47 rules) | +| **yamllint** | YAML formatting for `code/API_definitions/*.yaml` | Active in `pr_validation` v0 (via MegaLinter) | +| **gherkin-lint** | Test file structure for `code/Test_definitions/*.feature` | Active in `pr_validation` v0 (via MegaLinter) | +| **Python** | Cross-file consistency, release-plan validation, context-dependent checks | Partially active (release-plan validation only); api-review v0.6 deprecated | +| **manual** | Checks too semantic to automate; framework provides prompts for human/AI review | Not yet implemented | + +### 4.3 Implementation Classification + +Each check falls into one of five categories: + +| Category | Description | +|----------|-------------| +| **spectral** | Implementable as a Spectral rule (single-file OpenAPI checks, pattern matching, schema structure) | +| **python** | Requires Python implementation (cross-file checks, release-plan awareness, file existence, version consistency) | +| **manual** | Too semantic to fully automate (e.g., data minimization, meaningful descriptions). Framework generates a prompt or checklist | +| **obsolete** | Handled by release automation — validation framework should not re-check (e.g., readiness checklist file management, release tag creation, version field replacement on release branches) | +| **existing** | Already implemented in current Spectral/yamllint/gherkin rules. Severity mapping to be determined when individual rules are classified | + +--- + +## 5. Rule Architecture + +### 5.1 Processing Model + +The framework uses a **post-filter and severity mapping** architecture: + +1. Framework determines the execution context (branch type, trigger, release-plan.yaml content) +2. Engines run (Spectral, yamllint, gherkin-lint, Python checks) and produce findings +3. Framework collects all findings +4. For each finding: look up rule metadata → is it applicable in this context? → what level? → apply profile (advisory/standard/strict) to determine blocking +5. Produce unified output with hints + +### 5.2 Key Design Decisions + +- **Flat ID namespace**: Sequential IDs (e.g., `042`) that are stable across engine changes. A rule migrating from Python to Spectral keeps its ID. +- **Engine as metadata**: The `engine` field records which engine implements the check. The `engine_rule` field maps to the native rule ID (e.g., Spectral rule name) for configuration synchronization. +- **Condition language**: Plain YAML with AND across fields, OR within arrays. Range comparison on `commonalities_release`. No custom DSL or policy engine. +- **Per-API evaluation**: API-specific rules are evaluated once per API in the repository, using the API's own `target_api_status`, `target_api_maturity`, and `api_pattern` from release-plan.yaml and OpenAPI spec content. + +--- + +## 6. Bundling & Refs + +The overall bundling model — controlled local copy on `main`, source-only `main`, bundled release artifacts — is defined in the [Commonalities Consumption and Bundling Design](https://github.com/camaraproject/ReleaseManagement/pull/436) document. This section captures what the validation framework checks about that model, not the model itself. + +### 6.1 Repository Model Validation + +The framework validates the repository layout and `$ref` patterns used in API source files: + +- **Layout**: When API source files contain `$ref` to `../common/` or `../modules/`, the framework validates that referenced files exist, that `code/common/` contains only cache copies from external repositories, and that `code/modules/` contains project-local reusable schemas. +- **Ref patterns**: API source files must use only relative `$ref` for normative schema consumption. Remote URL-based `$ref` used for normative schema content is an error. Internal component references (`#/components/...`) are always valid. +- **Transition**: Repositories not yet using `code/common/` (current copy-paste model) remain valid. Layout and ref pattern checks only fire when `$ref` to `../common/` or `../modules/` is detected. The framework does not force migration. + +### 6.2 Validation Pipeline + +The framework applies a uniform validation flow regardless of context: + +1. **Pre-bundling validation** (always runs on source files): YAML validity, ref existence and pattern validation, release-plan.yaml consistency checks, cross-file checks that do not depend on schema content +2. **Bundling** (if `$ref` to `code/common/` or `code/modules/` is detected): Resolve all external refs and produce a standalone spec per API. Internal `$ref` (`#/components/...`) are preserved. The framework uses bundling (external ref resolution only), not full dereferencing. +3. **Full validation** (runs on the effective input — bundled output when refs are present, source directly when no refs): Full Spectral ruleset, Python checks that depend on schema content, standalone API spec validation +4. **Artifact surfacing**: Upload bundled specs as workflow artifacts or make them available for further processing (section 7) + +The validation **profile** (advisory, standard, strict) controls which findings block; it does not change which steps run. + +**Bundling failure**: If the bundling step fails (e.g., unresolvable `$ref`, missing file), the framework reports the failure as an error. Full validation is skipped; only pre-bundling results are available. + +**Repositories without `$ref`**: Repositories using the copy-paste model skip step 2. Source files are standalone by construction, so the full Spectral ruleset runs directly on source. No bundling overhead is introduced. + +**Bundling happens once**: The bundled API specs produced during validation are consumed by release automation for the snapshot branch. Release automation does not re-bundle independently. + +### 6.3 Cache Synchronization + +When `code/common/` exists, the framework validates that cached files match the declared dependency versions in `release-plan.yaml`. Mismatch severity depends on the profile: + +- **standard** (PR): warning — codeowner is informed, merge is not blocked +- **strict** (release automation): error — snapshot creation is blocked + +If no `code/common/` directory exists, the sync check is skipped. In the MVP, cache management may be manual; the validation check still applies regardless of how the cache was populated. + +### 6.4 Placeholder Handling + +- The framework detects unreplaced placeholder patterns (`{{...}}`) in bundled output and reports them as errors — this is a permanent safety net +- Placeholder replacement is not a framework responsibility. If placeholder replacement becomes part of a bundling/transformation pipeline, it is a post-MVP enhancement +- Detection of undefined/unresolvable placeholders remains an error regardless of whether replacement is implemented + +--- + +## 7. Artifact Surfacing + +The framework produces bundled artifacts for reviewer visibility. The [bundling design document](https://github.com/camaraproject/ReleaseManagement/pull/436) defines a priority order: + +1. **Source diff** — primary review surface, no framework action needed (standard git diff) +2. **Bundled artifact** — the framework uploads the bundled standalone API spec as a GitHub workflow artifact for each API affected by the PR +3. **Bundled diff** — a diff between the bundled API from the PR base and the bundled API from the PR head, uploaded as a workflow artifact or included in the workflow summary +4. **API-aware summary** — optional semantic change summary (post-MVP) + +**Line number mapping**: When checks run on bundled output, findings report line numbers in the bundled file. The framework must map finding locations back to source file and line number, so that findings are actionable. + +**API-aware change summaries**: The framework should support pluggable diff tools for generating semantic change summaries (breaking changes, new endpoints, modified schemas). This capability is post-MVP. + +--- + +## 8. Findings Surfacing + +### 8.1 Output Surfaces + +Two categories of output are surfaced to users: + +**A. Findings output** (validation errors, warnings, hints with fix guidance): + +| Surface | Description | Requires write token | +|---------|-------------|---------------------| +| **Workflow summary** | Structured markdown in the Actions run summary. Primary detailed surface with all findings, fix hints, and links to diagnostic artifacts | No | +| **Check run annotations** | Inline annotations in the PR diff, mapped to file and line. Findings appear directly where the issue is | Yes | +| **PR comment** | Concise summary on the PR: error/warn/hint counts, blocking status, link to full workflow summary | Yes | +| **Commit status** | Per-check context visible in the PR checks list (e.g., `--> Validate: release-plan.yaml`). Continuation of the v0 pattern | Yes | + +**B. Diagnostic artifacts** (log files, detailed engine output): + +| Artifact | Description | +|----------|-------------| +| **Spectral JSON log** | Full structured Spectral output for debugging rule behavior | +| **Engine reports** | Detailed per-engine output (replaces v0 MegaLinter report artifacts) | +| **Bundled spec artifacts** | Standalone bundled API specs (see section 7) | + +Diagnostic artifacts are always uploaded as GitHub workflow artifacts regardless of token permissions. + +### 8.2 Summary Size Limit + +GitHub limits workflow step summaries to 1 MB per step and 1 MB total per job. The framework must: + +- Truncate the summary when approaching the limit, showing a count ("50 of 127 findings shown") with a link to the full diagnostic artifact +- Prioritize errors over warnings over hints in the truncated view +- Always include the full findings in the Spectral JSON log artifact (no size limitation on workflow artifacts) + +### 8.3 Surfaces by Resolved Capability + +| Token capability | Findings output | Diagnostic artifacts | +|-----------------|-----------------|---------------------| +| **Write** (validation app, camara-release-automation, or GITHUB_TOKEN) | Workflow summary + check run annotations + PR comment + commit status | Workflow artifacts (always) | +| **Read-only** (all write paths failed) | Workflow summary only | Workflow artifacts (always) | + +A dedicated validation GitHub App provides write token capability even for fork PRs where `GITHUB_TOKEN` is read-only. This app is separate from `camara-release-automation` and is introduced from day one (MVP). + +--- + +## 9. Workflow Contract + +### 9.1 Input Design Principle + +The reusable workflow does not accept inputs that duplicate information derivable from the checked-out branch. The framework reads branch type from the branch name, and all release context from `release-plan.yaml` on the checked-out branch. These values are derived at runtime, never accepted as inputs. + +No per-repo inputs exist. All per-repo configuration lives in the central config file (section 10.2) read by the reusable workflow. The caller workflow is identical across all repositories. + +### 9.2 Reusable Workflow Inputs + +| Input | Type | Required | Default | Description | +|-------|------|----------|---------|-------------| +| `tooling_ref_override` | string | no | *(empty)* | 40-character SHA for non-standard tooling ref. Supports testing in contexts where OIDC may not work. Documented as pilot/break-glass only. | +| `profile` | choice | no | *(auto)* | Validation profile: `advisory`, `standard`, `strict`. Default: framework selects based on context. Dispatch users can set explicitly to see what a different profile would flag. | +| `mode` | string | no | *(empty)* | Execution mode. Set to `pre-snapshot` by release automation to invoke validation as a pre-snapshot gate. Affects trigger type derivation, profile selection, bundled spec handoff, and findings output target. | + +### 9.3 Trigger Summary + +| Trigger | Target branches | Default profile | +|---------|----------------|-----------------| +| `pull_request` | `main`, `release-snapshot/**`, `maintenance/**` | standard (strict for release review PRs) | +| `workflow_dispatch` | Any branch | advisory | +| Release automation call | Base branch (`main` or maintenance) | strict | + +--- + +## 10. Rollout Strategy + +### 10.1 v0/v1 Coexistence + +The v0 caller (`pr_validation_caller.yml`) and v1 caller (new workflow file) coexist in each repository during the transition period. Both run on every PR and appear in the PR checks list. Lifecycle per repository: + +1. **v1 deployed**: v1 caller added alongside v0. Both run on every PR. Codeowners see both check results. +2. **v1 validated**: Codeowners confirm v1 findings are correct and consistent with expectations. +3. **v1 required**: GitHub ruleset makes the v1 check blocking for PR merge. +4. **v0 removed**: v0 caller file deleted from the repository. Any v0-specific ruleset removed. + +The transition is per-repository. Repositories move through stages independently based on pilot experience and codeowner readiness. + +### 10.2 Central Enable/Disable + +A YAML configuration file in the tooling repository maps each API repository to its validation settings. The reusable workflow reads this file from its own checkout. This is version-controlled, PR-reviewable, and scalable. Adding a repo is one line in a YAML file. Satisfies UC-13 — no per-repo config changes needed. + +A repository not listed in the config file is treated as `disabled` — the reusable workflow exits immediately with a notice. This is the safe default for the 60+ repos that have the caller deployed but are not yet onboarded. + +### 10.3 Stage Model + +The config file controls the validation stage per repository: + +| Stage | Config value | Behavior | +|-------|-------------|----------| +| **0: dark** | `disabled` (or not listed) | Caller deployed, reusable workflow exits immediately | +| **1: advisory** | `advisory` | Runs on dispatch only, all findings shown, nothing blocks | +| **2: standard** | `standard` | Runs on PRs and dispatch, standard profile on PRs | +| **3: blocking** | `standard` + GitHub ruleset | Same as stage 2, plus GitHub ruleset requires the v1 check to pass for PR merge | + +Stage 3 is the combination of the config file setting (`standard`) and a GitHub ruleset. The config file does not control blocking — rulesets do. This separation keeps the config file focused on validation behavior and rulesets focused on merge policy. + +--- + +## 11. Release Automation Integration + +The validation framework integrates with CAMARA release automation at two points in the release lifecycle: + +1. **Pre-snapshot gate** (UC-08) — validation runs on the base branch before snapshot creation. Failure blocks the release process. +2. **Release review PR validation** (UC-09) — validation runs on the release review PR with strict profile and content restrictions. + +Both touchpoints use the strict profile (section 2.1): errors and warnings block. + +### 11.1 Pre-Snapshot Validation Gate + +Release automation invokes validation as part of the `/create-snapshot` command, before creating any branches. The validation runs on the current HEAD of the base branch — this is exactly the content that will become the snapshot. + +**Timing**: The release state is PLANNED when `/create-snapshot` is invoked. If validation fails, no snapshot branch is created and the state remains PLANNED. + +**Profile**: Strict — both errors and warnings block snapshot creation. Rationale: once a snapshot is created, content becomes immutable. Issues found post-snapshot require discarding and recreating the snapshot. + +**Scope**: The validation framework runs all applicable checks — Spectral rules, Python consistency checks, bundling validation, and release-plan semantic checks. This subsumes the existing `validate-release-plan.py` preflight. + +| Aspect | Current release-plan preflight | With validation framework | +|--------|-------------------------------|---------------------------| +| **Checks** | release-plan.yaml schema + semantics | Full framework: Spectral, Python, bundling, release-plan | +| **Blocking** | Errors only | Errors and warnings (strict profile) | +| **Findings output** | Bot comment with error list | Bot comment with structured findings + fix hints | +| **Token** | camara-release-automation app | Same (passed to framework) | + +### 11.2 Release Review PR Validation + +A release review PR is created by release automation on the `release-review/rX.Y-` branch, targeting the `release-snapshot/rX.Y-` branch. It contains only CHANGELOG and README changes — API specs and other files are immutable on the snapshot branch. + +**Detection**: The framework detects a release review PR by its target branch pattern (`release-snapshot/**`). + +**Profile**: Strict — errors and warnings block the PR. + +**Which checks run**: Only a subset of checks is meaningful on a release review PR: + +| Check category | Runs on release review PR | Rationale | +|----------------|--------------------------|-----------| +| CHANGELOG format validation | Yes | CHANGELOG is editable on the review branch | +| README content validation | Yes | README is editable on the review branch | +| File restriction check | Yes | Only CHANGELOG and README may be modified | +| Spectral / API definition checks | No | API specs are immutable on the snapshot branch | +| release-plan.yaml checks | No | Already validated at snapshot creation time | +| Bundling validation | No | Source files are immutable | +| Cache sync validation | No | Already validated at snapshot creation time | + +**File restriction check**: The framework examines the PR diff and produces an error if any file outside `CHANGELOG.md` (or `CHANGELOG/` directory) and `README.md` is modified. + +### 11.3 Pre-Snapshot Invocation + +The pre-snapshot gate follows the same input design principle as PR and dispatch contexts (section 9.1): the framework derives all release context from `release-plan.yaml` on the checked-out branch. The `mode` parameter distinguishes a pre-snapshot invocation from other triggers. + +### 11.4 Post-MVP Extensions + +The following integration enhancements are post-MVP: + +- **API-aware change summaries for release notes**: Semantic change summaries (breaking changes, new endpoints, modified schemas) for inclusion in release notes or the Release Issue. +- **Snapshot transformer validation**: Validating the transformer's configuration before snapshot creation — verifying that version replacement patterns and server URL formats are correct. + +--- + +## 12. Operational Views + +### 12.1 Repository Configuration Validation (UC-14) + +An admin needs to verify that an API repository is correctly configured for the validation framework. The following aspects must be checkable: + +- **Caller workflow**: The v1 caller workflow file exists in `.github/workflows/` and matches the expected template content +- **Central config listing**: The repository is listed in the tooling config file (section 10.2) with a valid stage value +- **GitHub ruleset** (stage 3 only): The v1 validation check is required in the applicable ruleset +- **Validation app installation**: The validation GitHub App is installed for the repository +- **v0 cleanup** (post-transition): The v0 caller file has been removed after v1 is stable at stage 3 + +### 12.2 Minimal Change Noise Principle + +The design choices throughout this document follow a consistent principle: **minimize the number of changes that require codeowner interaction in API repositories.** The only per-repo change required for onboarding is adding the v1 caller workflow file — a mechanical copy from `Template_API_Repository`. After that, all configuration changes happen centrally in the tooling repository (section 10.2). + +### 12.3 Release Manager Dashboard (UC-16, UC-17) + +UC-16 and UC-17 are **independent work** — the Release Progress Tracker already exists and is operational. This section defines the integration points between the validation framework and the tracker. + +**Compliance indication**: For each repository with validation enabled, the dashboard shows a compliance status derived from the most recent validation run on `main`: + +| Status | Meaning | +|--------|---------| +| **compliant** | No errors, no warnings (would pass strict profile) | +| **issues** | Has warnings but no errors (would pass standard, not strict) | +| **failing** | Has errors (would fail standard profile) | +| **unknown** | No validation data available | + +**Data collection**: The tracker queries the GitHub API for the most recent completed run of the v1 validation workflow on the default branch (conclusion, URL, timestamp). No changes to the validation framework are needed for this. + +**On-demand trigger** (UC-17): The tracker or an admin can dispatch validation on selected repositories via `workflow_dispatch` to update dashboard status. + +### 12.4 Cross-System Integration Map + +| System | Integration point | Data flow | Priority | +|--------|------------------|-----------|----------| +| Release automation (`/create-snapshot`) | Pre-snapshot gate (11.1) | Validation → release automation: pass/fail + findings + bundled specs | Post-MVP (high) | +| Release automation (release review PR) | PR trigger with strict profile (11.2) | Standard PR validation flow, profile auto-selected | Post-MVP (high) | +| Release Progress Tracker | Workflow run query (12.3) | Tracker reads validation run data from GitHub API | Independent | +| Release Progress Tracker | Dispatch trigger (12.3) | Tracker dispatches validation via GitHub API | Independent | +| Tooling config file | Central enable/disable (10.2) | Validation reads config at runtime | MVP | +| GitHub rulesets | Blocking enforcement (10.3) | Ruleset references validation check name | Stage 3 | +| Validation GitHub App | Token minting (8.3) | App provides write token for findings surfacing | MVP | diff --git a/documentation/SupportingDocuments/CAMARA-Validation-Framework-Testing-Guidelines-Audit.md b/documentation/SupportingDocuments/CAMARA-Validation-Framework-Testing-Guidelines-Audit.md new file mode 100644 index 0000000..99d769d --- /dev/null +++ b/documentation/SupportingDocuments/CAMARA-Validation-Framework-Testing-Guidelines-Audit.md @@ -0,0 +1,221 @@ +# Testing Guidelines Audit + +**Date**: 2026-03-19 +**Scope**: Machine-checkable rules from API-Testing-Guidelines.md +**Sources**: API-Testing-Guidelines.md, .gherkin-lintrc (25 rules), api_review_validator_v0_6.py (test checks V6-071–V6-080) + +## Methodology + +1. Walked API-Testing-Guidelines.md section by section, extracting every machine-checkable requirement +2. Cross-referenced against: .gherkin-lintrc rules, v0_6 validator test alignment checks, pr_validation.yml MegaLinter integration + +## Legend + +**Coverage column**: +- `gherkin: ` — rule exists in .gherkin-lintrc +- `v0_6: V6-NNN` — covered by api_review_validator_v0_6.py +- `gap` — no current implementation + +**v1 Engine**: `gherkin-lint` / `python` / `manual` + +--- + +## File Structure and Location + +| ID | Rule | Coverage | v1 Engine | Notes | +|----|------|----------|-----------|-------| +| TG-001 | Test files MUST be in `code/Test_definitions/` | v0_6: V6-071 | python | Directory existence check | +| TG-002 | Each API MUST have at least one `.feature` file | v0_6: V6-072 | python | Matched by api-name prefix | +| TG-003 | Single-feature file: filename = api-name (kebab-case) | v0_6: V6-072 | python | e.g. `location-verification.feature` | +| TG-004 | Multi-feature files split by operation: `{api-name}-{operationId}.feature` | v0_6: V6-076 | python | operationId must exist in spec | +| TG-005 | Multi-feature files other grouping: `{api-name}-{description}.feature` | gap | python | No validation of description part | +| TG-006 | Filenames follow `{api-name}` or `{api-name}-{operationId}` convention | v0_6: V6-072/V6-076 | python | gherkin-lint `file-name` (kebab-case) cannot be used: convention mixes kebab-case api-name with camelCase operationId | +| TG-007 | `.feature` files must not be empty | gherkin: no-empty-file | gherkin-lint | | +| TG-008 | `.feature` files must contain scenarios | gherkin: no-files-without-scenarios | gherkin-lint | | + +--- + +## Feature Description + +| ID | Rule | Coverage | v1 Engine | Notes | +|----|------|----------|-----------|-------| +| TG-009 | Feature MUST have a name | gherkin: no-unnamed-features | gherkin-lint | | +| TG-010 | Feature name MUST include API name | gap | python | Not checked by gherkin-lint | +| TG-011 | Feature name MUST include API version | v0_6: V6-073 | python | Checks lines 1-2 for version pattern | +| TG-012 | Feature name max 250 characters | gherkin: name-length (Feature: 250) | gherkin-lint | | +| TG-013 | Feature names MUST be globally unique | gherkin: no-dupe-feature-names | gherkin-lint | | + +--- + +## Feature Context (RECOMMENDED) + +| ID | Rule | Coverage | v1 Engine | Notes | +|----|------|----------|-----------|-------| +| TG-014 | Feature context SHOULD reference API spec file location | gap | python | Pattern: `in {apiname}.yaml` | +| TG-015 | Feature context SHOULD include "Implementation indications" section | gap | manual | Recommended, not mandatory | +| TG-016 | Feature context SHOULD include "Testing assets" section | gap | manual | Recommended, not mandatory | +| TG-017 | Feature context SHOULD include "References to OAS spec schemas" | gap | manual | Recommended, not mandatory | + +--- + +## Background Section + +| ID | Rule | Coverage | v1 Engine | Notes | +|----|------|----------|-----------|-------| +| TG-018 | Background MUST NOT be empty | gherkin: no-empty-background | gherkin-lint | | +| TG-019 | Background MUST NOT exist without scenarios | gherkin: no-background-only-scenario | gherkin-lint | | +| TG-020 | Background SHOULD include environment setup (apiRoot) | gap | manual | Recommended pattern, not strictly checkable | +| TG-021 | Background SHOULD include Content-Type header setup | gap | manual | Recommended pattern | +| TG-022 | Background SHOULD include Authorization header setup | gap | manual | Recommended pattern | +| TG-023 | Background SHOULD include x-correlator header setup | gap | manual | Recommended pattern | + +--- + +## Scenario Structure + +| ID | Rule | Coverage | v1 Engine | Notes | +|----|------|----------|-----------|-------| +| TG-024 | Each scenario MUST have a name | gherkin: no-unnamed-scenarios | gherkin-lint | | +| TG-025 | Scenario names MUST be unique within feature | gherkin: no-dupe-scenario-names (in-feature) | gherkin-lint | | +| TG-026 | Scenario name max 250 characters | gherkin: name-length (Scenario: 250) | gherkin-lint | | +| TG-027 | Step name max 250 characters | gherkin: name-length (Step: 250) | gherkin-lint | | +| TG-028 | Steps MUST follow Given → When → Then order | gherkin: keywords-in-logical-order | gherkin-lint | | +| TG-029 | Repeated step keywords MUST use `And` | gherkin: use-and | gherkin-lint | | +| TG-030 | Scenario Outline MUST have Examples section | gherkin: no-scenario-outlines-without-examples | gherkin-lint | | +| TG-031 | Max 50 scenarios per file | gherkin: max-scenarios-per-file (50) | gherkin-lint | | +| TG-032 | All scenario variables MUST be defined | gherkin: no-unused-variables | gherkin-lint | | + +--- + +## Scenario Tagging + +| ID | Rule | Coverage | v1 Engine | Notes | +|----|------|----------|-----------|-------| +| TG-033 | Scenario tag format: `@{feature_identifier}_{number}_{optional_detail}` | gap | python | Pattern check on tag naming | +| TG-034 | Tag identifiers: lowercase with underscores | gap | python | Could be gherkin-lint allowed-tags pattern | +| TG-035 | Every scenario MUST have at least one tag | gherkin: required-tags (pattern: `^@.*$`) | gherkin-lint | | +| TG-036 | Tags `@watch` and `@wip` are forbidden | gherkin: no-restricted-tags | gherkin-lint | | +| TG-037 | No duplicate tags on same scenario | gherkin: no-duplicate-tags | gherkin-lint | | +| TG-038 | No superfluous tags | gherkin: no-superfluous-tags | gherkin-lint | | +| TG-039 | Single space between tags | gherkin: one-space-between-tags | gherkin-lint | | +| TG-040 | No inline comments on tag lines | gherkin: no-partially-commented-tag-lines | gherkin-lint | | + +--- + +## Request Step Syntax + +| ID | Rule | Coverage | v1 Engine | Notes | +|----|------|----------|-----------|-------| +| TG-041 | operationId in `When the request "{operationId}" is sent` MUST exist in API spec | v0_6: V6-075 | python | Cross-file: test ↔ API spec | +| TG-042 | Path parameters: `the path parameter "{name}" is set as "{value}"` | gap | manual | Standardized syntax, hard to enforce | +| TG-043 | Query parameters: `the query parameter "{name}" is set as "{value}"` | gap | manual | Standardized syntax | +| TG-044 | Headers: `the header "{name}" is set as "{value}"` | gap | manual | Standardized syntax | +| TG-045 | Request body: `the request body property "{json_path}" is set as "{value}"` | gap | manual | JSON path notation | + +--- + +## Response Validation Syntax + +| ID | Rule | Coverage | v1 Engine | Notes | +|----|------|----------|-----------|-------| +| TG-046 | Status code validation: `the response status code is {code}` | gap | manual | Standardized syntax | +| TG-047 | Schema compliance: `the response body complies with the OAS schema {ref}` | gap | manual | Standardized syntax | +| TG-048 | Error code: `the response property "$.code" is "{ERROR_CODE}"` | gap | manual | Standardized syntax | + +--- + +## URL and Version Alignment (cross-file) + +| ID | Rule | Coverage | v1 Engine | Notes | +|----|------|----------|-----------|-------| +| TG-049 | Test file URLs MUST contain correct api-name matching API spec | v0_6: V6-078 | python | Regex extraction + comparison | +| TG-050 | Test file URL version suffix MUST match API version | v0_6: V6-079 | python | Version mapping rules (wip→vwip, 1.0.0→v1, 0.3.0→v0.3) | +| TG-051 | Test file URLs SHOULD have leading slash | v0_6: V6-080 | python | Style convention (LOW severity) | +| TG-052 | Operation-specific test filenames MUST reference valid operationIds | v0_6: V6-076 | python | Cross-file: filename ↔ API spec | +| TG-053 | Orphan test files (not matching any API name) MUST be flagged | v0_6: V6-077 | python | Multi-file name matching | +| TG-066 | If operation-specific test files are used, all operationIds MUST be covered | gap | python | Completeness check: `{api-name}-{operationId}.feature` for every operationId (RC and later) | + +--- + +## Coverage Requirements + +| ID | Rule | Coverage | v1 Engine | Notes | +|----|------|----------|-----------|-------| +| TG-054 | RC: minimum basic test plan with sunny-day scenarios | gap | manual | Semantic — requires human judgment | +| TG-055 | Public release: full test plan with sunny and rainy day scenarios | gap | manual | Semantic | +| TG-056 | All HTTP statuses documented in spec MUST have test scenarios | gap | python | Cross-file: response codes in spec → scenario coverage. Feasible but complex | +| TG-057 | 3-legged token responses: MUST NOT include device identifier | gap | manual | Response content validation at runtime | +| TG-058 | 2-legged token responses: SHOULD include device identifier | gap | manual | Response content validation at runtime | + +--- + +## Indentation and Formatting + +| ID | Rule | Coverage | v1 Engine | Notes | +|----|------|----------|-----------|-------| +| TG-059 | Feature: 0 spaces indentation | gherkin: indentation | gherkin-lint | | +| TG-060 | Background/Scenario/Step: 2 spaces | gherkin: indentation | gherkin-lint | | +| TG-061 | Examples header: 4 spaces | gherkin: indentation | gherkin-lint | | +| TG-062 | Example rows: 6 spaces | gherkin: indentation | gherkin-lint | | +| TG-063 | Tags: 2 spaces indentation | gherkin: indentation | gherkin-lint | | +| TG-064 | No trailing spaces | gherkin: no-trailing-spaces | gherkin-lint | | +| TG-065 | No multiple empty lines | gherkin: no-multiple-empty-lines | gherkin-lint | | + +--- + +## Gherkin-lint Rules Not Mapped to Testing Guidelines + +These rules exist in `.gherkin-lintrc` but do not correspond to a specific testing guidelines requirement: + +| Gherkin Rule | Status | Notes | +|-------------|--------|-------| +| no-homogenous-tags | ON | Quality heuristic, not in guidelines | +| allowed-tags | ON | Patterns: `^@watch$`, `^@wip$`, `^@.*$` — enforces tagging | +| new-line-at-eof | OFF | Not mentioned in guidelines | +| scenario-size (15 steps) | OFF | Not enforced; guidelines allow complex scenarios | +| only-one-when | OFF | Guidelines explicitly allow multiple When blocks | +| no-restricted-patterns | OFF | Debugging step patterns configured but disabled | + +--- + +## Gap Summary + +### Gaps addressable by gherkin-lint configuration changes + +| ID | Rule | Suggested Change | +|----|------|-----------------| +| ~~TG-006~~ | ~~Filenames use kebab-case~~ | Not a gap: covered by v0_6 Python checks; gherkin-lint `file-name` cannot be used due to mixed casing (`{api-name}-{operationId}`) | +| TG-033/034 | Tag naming convention `@{feature}_{number}_{detail}` | Could use `allowed-tags` pattern: `^@[a-z][a-z0-9]*(_[a-z0-9]+)*$` | + +### Gaps requiring Python checks + +| ID | Rule | Priority | +|----|------|----------| +| TG-010 | Feature name includes API name | Medium | +| TG-014 | Feature context references API spec file | Low (RECOMMENDED) | +| TG-056 | All documented HTTP status codes have test scenarios | Medium (cross-file, complex) | + +### Gaps that remain manual + +| ID | Rule | Reason | +|----|------|--------| +| TG-015–017 | Feature context sections | Semantic content, RECOMMENDED | +| TG-020–023 | Background setup patterns | Recommended patterns, not strict syntax | +| TG-042–048 | Standardized step syntax | Natural language with alternatives allowed | +| TG-054–055 | Test coverage completeness | Requires human judgment on adequacy | +| TG-057–058 | Device identifier in responses | Runtime validation, not static | + +--- + +## Statistics + +| Category | Count | +|----------|-------| +| Total rules extracted | 65 | +| Covered by gherkin-lint | 25 | +| Covered by v0_6 validator | 10 | +| Gaps (no implementation) | 30 | +| Of which: addressable by config change | 1 | +| Of which: addressable by Python | 3 | +| Of which: remain manual | 25 | +| Gherkin rules not mapped to guidelines | 6 |