"Nocode" instrumentation without modifying source

[johnbley]

This PR allows "nocode" instrumentation of custom methods for those who don't have the ability
to modify the original Java source code. It uses a yml file to direct the complexity
of how the spans should look:

- class: myapp.BusinessObject
  method: update
  spanName: this.getName()
  attributes:
    - key: "business.context"
      value: this.getDetails().get("context")

- class: mycustom.SpecialClient
  method: doRequest
  spanKind: CLIENT
  spanStatus: 'returnValue.code() > 3 ? "OK" : "ERROR"'
  attributes:
    - key: "special.header"
      value: 'param0.headers().get("special-header").substring(5)'

Each yml entry above encodes a "rule" for how to instrument the chosen method.

Java-like expressions are written in JEXL which was chosen after looking through a number of expression languages, trying to balance (a) expressivity for common use cases, (b) compatibility with the license
and design of the otel java agent, and (c) not having too much weight or being a full
separate language like groovy (or a complete java compiler/interpreter).

Future work can include:

• Configuration moving to declarative configuration
• More complex ability to select classes and methods besides exact-name-match (this
  requires some help with the ability to parameterize advice classes by rule)
• Other instrumentation features (e.g., propagation inject/extract)

Fixes #11046.

This is an upstream donation of a Splunk-specific feature based on requests from and discussions with our field and customer base.

[johnbley]

I was able to get a local -PindyTest=true test to fail but with a different error message. When I hardcoded isIndyModule to false (a hack to be sure) it passed locally. I see now that it is not passing on github CI. So, I will look at this more tomorrow.

[PeterF778]

Without going deep into the capabilities of this feature as is, or its future enhancements, the thing that strikes me first is the name "nocode".

It is very misleading. In order for it to work, the user has to write code, even if disguised behind YAML cover. The feature instruments (byte)code. Furthermore, for a meaningful result, the user needs to have access to and at least partial understanding of the application source code.

But what could be a good name for it? What sets this feature apart from other forms of instrumentation is that it can be applied at the very last step with respect to application development/deployment lifecycle, and can bypass the regular processes involved in code review, functionality/performance testing or security reviews. This should get reflected in the name, but I struggle to find a really good one.

I have some proposals, though, even if I'm not entirely happy with them:

• last-minute instrumentation (LMI)
• post-deployment instrumentation
• sneaky instrumentation

[johnbley]

Without going deep into the capabilities of this feature as is, or its future enhancements, the thing that strikes me first is the name "nocode".

It is very misleading. In order for it to work, the user has to write code, even if disguised behind YAML cover. The feature instruments (byte)code. Furthermore, for a meaningful result, the user needs to have access to and at least partial understanding of the application source code.

But what could be a good name for it? What sets this feature apart from other forms of instrumentation is that it can be applied at the very last step with respect to application development/deployment lifecycle, and can bypass the regular processes involved in code review, functionality/performance testing or security reviews. This should get reflected in the name, but I struggle to find a really good one.

I have some proposals, though, even if I'm not entirely happy with them:

• last-minute instrumentation (LMI)
• post-deployment instrumentation
• sneaky instrumentation

Yes, I wrestled with this a bit too. The best alternative to "nocode" I have at the moment is "nonsource"/"nosource"/"sourceless" - meaning that you don't need access to source code to do it. I'm obviously happy to refactor the basic name if the group can come to a consensus.

[laurit]

Instead of adding a new instrumentation you could consider adding this functionality to the existing methods instrumentation. That would solve the naming problem.

[johnbley]

Instead of adding a new instrumentation you could consider adding this functionality to the existing methods instrumentation. That would solve the naming problem.

Could definitely do that - would you want to keep the current -D-based mechanism (for simple instrumentation) as well or completely replace it?

[PeterF778]

Instead of adding a new instrumentation you could consider adding this functionality to the existing methods instrumentation. That would solve the naming problem.

It would not change the nature of the instrumentation, which is that unreviewed and unverified instrumentation can be added as configuration for any environment.

[johnbley]

unreviewed and unverified instrumentation can be added as configuration for any environment.

I hear you! What about adding some stronger wording like

By using this feature you are essentially changing your application's code.  Doing so without understanding
the behavior of the code you are adding may cause any or all of:
  - Data corruption
  - Deadlocks or application crashes
  - Memory leaks
  - Greatly increased latency or timeouts
  - Errors reported back to your application's clients

?

[jackshirazi]

Is there a way to enforce "access only" expressions? Probably would need a more restrictive or dedicated expression language, but that would eliminate most potential security bugs. Though still leaving insecure access to internal info, it would at least eliminate the possibility of arbitrary code execution slipping through at some point. We don't want to become an "Otel4Shell" debacle

[johnbley]

We discussed this in the SIG meeting last week and settled on moving this to -contrib as an optional extension. The TypeTransformer changes Lauri introduced here will be split off into a separate -instrumentation PR to make things easier.