Support semantic OpenAPI expressions for targets

[lornajane]

For example, how to target all parameters with a name matching a pattern. Or all Array schemas. Rather than needing some JSONPath expression that checks in all the places that these items can go in an OpenAPI file, consider the OpenAPI file as a data structure and access everything more semantically rather than syntactically.

You probably already have linting rules that do some of this stuff, feel free to add them as example here.

[lornajane]

Syntax example could be something like these (and if you comment on this, I'll assume you think the overall idea is excellent):

target:
  item: operation
  tag: x-special-operation
  method: POST
  

We also talked about:

target:
  item: schema
  criteria:
    pattern: '*_date'

[mikeschinkel]

@lornajane — Thanks for posting those syntax examples.

I know you risk people bikeshedding syntax and not focusing on the question "Does Overlay need to support semantic OpenAPI expressions for targets?", but having these examples really helps someone like me who thinks in code to conceptualize what you were thinking in your question.

@darrelmiller — Can you see the signatures you are working on being reused here as a type of target?

[lornajane]

We chatted more about this idea in the meeting today and the idea is to allow people to target particular types of things, for example, everything that's got an "X Object" heading in the specification. Then we'd also need the ability to target children of that thing, with some filtering.

[handrews]

My main concern is that any semantic selection feature be usable consistently across Overlay and Arazzo with OAS 3.x and Moonwalk (and within OAS if relevant).

[kevinswiber]

Prior art:

• Structural search and replace in rust-analyzer: Structural Search Replace rust-lang/rust-analyzer#2267
• Structural search and replace using tree-sitter in Neovim: https://github.com/cshuaimin/ssr.nvim
• Structural search and replace in JetBrains: https://www.jetbrains.com/help/idea/structural-search-and-replace.html#to_search_structurally
• Redocly: https://redocly.com/docs/cli/rules/configurable-rules#configurable-rule-object

Many of these have variable capture and reuse for replacement operations. We may not use this exact syntax, but I think this demonstrates there's a real use case for semantic selection expressions.

[lornajane]

We talked about what we should do when something that is semantically targeted, but it is a $ref. In this situation, an overlay should NOT follow the $ref and update something that might additionally be referenced from other locations. If a $ref is found, then for the targeted instance, it should be deferenced and updated only within the target scope.

It is totally valid to target something that is referenced from lots of places - but target it in the place it is actually declared, not via reference.

[mikeschinkel]

Even though @handrews was strongly opposed(!) I wanted to document for posterity the suggestion to support an embeddable programming language such as JavaScript or Lua, both of which have significant tooling available.

There is also Common Expression Language, which might be reasonable to consider.

And any of these could piggyback off ADA.

[mikeschinkel]

I dislike the implication that I am somehow against this.

I did not mean to imply that in any way. I only meant to imply that my desire for making things possible probably outweighs other important concerns, which in order to be intellectually honest I will admit should be addressed. So please take that specific comment there characterizing my perspective but not your position.

What I object to is just saying one language is the language. And that's a hill I am more than willing to die on and argue to exhaustion.

Well, since you bring that up, objecting to saying "one language is the language" means effectively the same as saying you advocate that there not be (just) one language, right?

Thus — given the need for a standard to be interoperable means effectively — you are advocating there to be no language — at least not recognized by the spec — and to me that is the perfect being the enemy of the good.

I am willing to concede the point that maybe JavaScript and Lua are a bridge too far, but I don't think it is fair to have one person's opinion summarily kill any consideration of the potential to support something like CEL.

IOW, I'd like to get other's opinion on (something like) CEL before we would table it with prejudice.

[handrews]

I haven't had time to look at CEL.

Thus — given the need for a standard to be interoperable means effectively — you are advocating there to be no language — at least not recognized by the spec — and to me that is the perfect being the enemy of the good.

That is not the point at all. We are hardly the first standard to define a language-neutral spec that will have bindings in many languages. Whether any/all of those bindings are "official" has a lot more to do with the resources of the project than anything else.

I appreciate your clarification regarding your perspective vs my intentions, but by taking my perspective back to "enemy of the good," I really don't know what to do with this. I don't see a producitve discussion here. You want a programming language to be required, and I will fight that to every degree possible.

Again, I have not had time to look at CEL. If it is programming-environment neutral (meaning that it, like CSS which I brought up, is a spec that can be supported in many environment), then I'd be open to it as a syntax. But if it requires a specific programming language, then no.

[mikeschinkel]

@handrews — A key point of distinction. I was proposing an embedded language, not language bindings.

IOW, the ability to write expressions in the YAML doc and be able to know they would just work. That is different from having many language able to process the documents with external tooling.

I expect you probably disagree with that just as much, but I wanted to make the distinction since it might not have been clear.

Again, I have not had time to look at CEL.

Please take a look at it, when you get a chance.

If it is programming-environment neutral ...then I'd be open to it as a syntax.
But if it requires a specific programming language, then no.

That really depends on how you define "specific programming language" as CEL could be considered a programming language by some but not by others.

If you mean a Turing-complete language, no it is not. From Claude: CEL is "A non-Turing-complete expression language designed for lightweight evaluation of expressions, often used in policy engines and configuration management."

CEL examples:

// Simple expression
name.startsWith("foo") && age > 18

// Using built-in functions
size(list) > 5 || list.exists(x, x > 10)

// Boolean logic
request.path.startsWith("/api") && request.method == "GET"

// String manipulation
request.user + ":" + request.path

// Mathematical operations
resource.cpu * 2 + resource.memory / 1024

// Collection operations
request.headers.exists(h, h.startsWith("X-"))

End user cannot define functions in CEL, but the spec could define OpenAPI-specific functions to handle the 80 percentile use-case, and then the vendors could implement those in Go, C, Java, or whatever.

[handrews]

@mikeschinkel I just want to acknowledge that my reactions here have been overly harsh. I'm going to focus on getting DOM/ADA specified as part of Moonwalk, and leave whether or how Overlays involve those interfaces the up to you and others who have been more consistently involved with Overlays.

[mikeschinkel]

@handrews — Thank you for the comment.

No problem, I didn't take it as harsh, just passionate. I know you mean well. 🙂

[mikeschinkel]

I did some more research on prior art that provide transforms on YAML or JSON, in descending order of what I found relevant, more or less (though I found them all somewhat relevant):

• yaml-cli - Transforms YAML through direct operations on nodes via path selectors for getting, setting, or deleting values and matching objects.

• Koreo - Transforms Kubernetes resources using CEL (Common Expression Language) in function-based pipelines.

• Yamale - Validates YAML against schema definitions using built-in validators (str, int, num, map, etc.) and supports custom validator extensions.

• Jsonnet - Transforms data with variables, functions, arithmetic, and conditionals extending JSON syntax.

• Jenkins Job Builder - Transforms YAML job definitions into Jenkins XML configurations via templates and macros.

• Kustomize - Transforms Kubernetes YAML through bases, overlays, patches (strategic merge and JSON), and built-in transformers (namePrefix, commonLabels).

• BigQuery SQL Translation - Transforms SQL using YAML-defined object rewriters, attribute type transformations, and name mappings.

• kpt - Transforms Kubernetes YAML via function-based processors that apply standard or custom mutations.