Develop a CVE Record Format Versioning RFD

[alilleybrinker]

There is currently not a clear answer to how the CVE Record Format ought to be versioned. While the QWG has collectively suggested SchemaVer as a potential scheme, there doesn't appear to be consensus among the group on that question.

In particular, while SchemaVer expresses backward compatibility of new schema versions (the level of backward compatibility determines which of the three numbers in a SchemaVer version gets bumped at release-time), it does not express anything about forward compatibility.

Since CVE is a multi-stakeholder system, and the CVE Record Format impacts both CVE data producers (CNAs, ADPs), and consumers, it would be good to be more rigorous in the versioning applied to the record format so we can better communicate to our stakeholders about the implications of a new release.

This question has also come up in some specific conversations recently, including:

• RFD to introduce an RFD process #405
• Add packageURL field to product in affected array. #409
• Add a formal semver 2.0.0 version type #371

In #405, the issue is how versioning impacts the "Compatibility and Migration" section of the RFD template.

In #409, the issue is whether a new packageURL field on the product object of the affected array should be added to the set of "identifier-like" fields available to fulfill the requirement for an "identifier-like" entry in every product object, which presents a forward compatibility hazard.

In #371, the issue is what level of breakage we consider adding a new version type to the schema to be. Under SchemaVer, it's an ADDITION-level change, but it does present a forward compatibility issue (users of the current version of the format may need to implement or import a Semantic Version parser).

From these, we can see that there are two layers to the question:

• Should the CVE Record Format's stability guarantees include some level of forward compatibility in addition to backward compatibility? If so, what forward compatibility guarantees should we make?
• How should compatibility be expressed in the version number published with new releases of the CVE Record Format?

[alilleybrinker]

During today's QWG meeting it was raised that this conversation will almost certainly need to loop in the AWG (Automation Working Group) since they operate CVE Services which uses the schema for CVE submissions. Currently CVE Services only uses one version of the schema at a given time, so they'll likely have thoughts on how the rules around versioning the schema affect them.

[alilleybrinker]

Also @boblord raised in #405 the idea that we should do consumer engagement, which we probably will want to do for this topic given the clear user impacts of our versioning rules.

Options for this include surveys (broad coverage, feedback is perhaps not nuanced / detailed) and "field studies" (direct engagement, fewer data points but deeper).

[ElectricNroff]

Reviewing #405 (comment) the "Forward compatibility" Wikipedia page was mentioned. It suggests that "Forward compatibility" is often applied to data formats or data-processing systems that cannot easily be replaced, because they are tied to hardware (or perhaps expensive proprietary software). In particular, it might apply well to situations in which many consumers cannot realistically update, and yet they want to handle some new objects (perhaps in a way that's somehow degraded relative to how new objects would be handled by a new system).

In the CVE Program use cases, the CVE Record Format schema is not like that. Obtaining the new schema itself is easy and free. Also, many consumers want to handle all new objects, not merely some of them.

For these reasons, it may be best to use the term backward compatibility from two perspectives: the schema and the consumer. In the example from #405 (comment)

the old schema is:

{
  "type": "object",
  "properties": {
    "x": { "type": "string" },
    "y": { "type": "string" }
  },
  "required": ["x", "y"]
}
and the new schema is:

{
  "type": "object",
  "properties": {
    "x": { "type": "string" },
    "y": { "type": "string" }
  },
  "required": ["x"]
}

this is not backward compatible from the consumer perspective. From the schema perspective, if we rely on:

Forward compatibility: When a new version of the schema does not
permit records which would not be permitted by the past version of
the schema.

it is not forward compatible from the schema perspective. It is a common property that when the perspective shifts (consumer to schema), the direction flips (backward to forward).

This may address concerns that support for "forward compatibility" is not needed.

[alilleybrinker]

One piece of prior art on versioning is Ember.js, in particular that they never pair the introduction of new features with the removal of deprecated features. To put it another way: their minor versions add new features and may mark existing features as deprecated, and their major features only remove deprecated features. See: https://emberjs.com/releases/

For us this would be something like adding new fields in "minor" versions, and then in "major" versions only removing deprecated fields.

[alilleybrinker]

IMO, I'd propose the following:

• Adopt SemVer 2.0 for the CVE Record Format.
  • Define the precise breakage rules for the API guarantee.
• To enable the QWG to catch unintentional breakage early: Build a static breakage-detection tool (like cargo-semver-checks).
• To catch when technically non-breaking changes would nonetheless break ecosystem tools: Build a crater-like ecosystem testing tool to catch ecosystem breakage in advance of schema changes.
• To catch breakage against historic CVE records: Run all historic CVE Records against new versions of the schema.

[alilleybrinker]

Resolved during the QWG meeting: we will use SemVer 2.0 as the versioning scheme for the CVE Record Format, and will begin work to define our precise API guarantees.

[ccoffin]

Started brainstorming today on how we might use semver Major, Minor, and Patch when changing the CVE Record Format. I am probably missing a lot of things, but this is a start.

MAJOR (semver definition: which you're supposed to use when you make backwards-incompatible API changes)

• changes to existing and commonly used properties (would need to define commonly used)
• removal of existing and commonly used properties (would need to define commonly used)
• adding requirement for previously existing properties
• adding requirement for new properties

MINOR (semver definition: when you add backwards-compatible functionalities)

• new properties (cannot affect required properties)
• changes to properties that are NOT required and rarely used (would need to define rarely used)
• removal of properties that are NOT required and rarely used (would need to define rarely used)
• removing requirement for property to exist

PATCH (semver definition: when you make backwards-compatible bug fixes)

• documentation additions, changes, or removals
• internal scripting changes in repo

[alilleybrinker]

IMO, we shouldn't incorporate any notion of "commonly used" in the versioning rules; the rules should apply equivalently to all fields.

We might want to define a way to experiment with "unstable" fields in public, but I think that would be in the form of a common prefix, like unstable, for relevant field names; unstable fields could exist in public stable versions without committing to stability.

[alilleybrinker]

In today's QWG, the point was raised that whatever versioning policy we settle on, there will likely be a need to retain wiggle room for risk-informed decisions to permit breakage in minor versions.

I offered as prior art the stability policy of the Rust programming language, which permits minor-version breaking changes in cases where the broken code was mistakenly accepted in a prior version, or where an API was determined to be unsound (thus violating Rust's normal guarantees).

There seemed to be general consensus among the group that there's value in leaving this kind of wiggle room in the CVE Record Format versioning rules.

[alilleybrinker]

Another issue raised is whether the versioning of the Record Format is as a schema or as part of the CVE Services API. This is probably something to resolve early, before even the specific versioning rules, as it will inform everything else.

[alilleybrinker]

IMO, from discussions so far, my views are the following:

• The Record Format has consumers besides CVE Services, such as Vulnogram.
• We should version the Record Format separately from CVE Services, but coordinate with the AWG around any breakage concerns.
• The versioning rules must leave room for risk-assessments to permit minor-version breakage. We should specify the rules for when those considerations come into play and breakage is permissible (for example, to correct mistakes in prior versions).
• We should have a specified way for indicating unstable parts of the Record Format, which can be broken in a minor version.

[alilleybrinker]

Regarding stability guarantees, we could imagine publicly documenting stability of fields, with different states:

• Proposed: The field has been proposed in an RFD.
• Provisional: Under an RFD rollback period, meaning it's new and hasn't yet passed its associated RFD's success metrics, and may be rolled back.
• Stable: The field follows the normal stability rules.
• Deprecated: Will be removed in a future breaking version.
• Unused: Previously stable, but identified as "unused" based on data analysis; may be removed in a minor version after a consideration period of some fixed time.
• Removed: The field has been removed from the schema.

flowchart TD
    proposed[Proposed]
    provisional[Provisional]
    stable[Stable]
    unused[Unused]
    deprecated[Deprecated]
    removed[Removed]

    proposed-- RFD approved, major or minor version released -->provisional
    provisional-- Passes RFD success metrics -->stable
    provisional-- Does not pass RFD success metrics -->removed
    stable-- Marked for removal by the QWG -->deprecated
    stable-- Marked as unused by the QWG -->unused
    deprecated-- Major version released -->removed
    unused-- Minor version released -->removed

Loading