chore(deps): Widen dependency ranges for non-SDK packages

[mydea]

Related to #4815

I noticed that e.g. @opentelemetry/instrumentation-http or @opentelemetry/instrumentation-fetch all hard-depend on specific versions of e.g. @opentelemetry/instrumentation, core, semantic-conventions etc. Is this really desired?

Based on https://github.com/open-telemetry/opentelemetry-specification/blob/a03382ada8afa9415266a84dafac0510ec8c160f/specification/upgrading.md?plain=1#L97-L122, if I understand correctly, instrumentation should continue to work for following minor releases of core packages. By relaxing e.g. @opentelemetry/core dependencies to e.g. ^1.26.0 etc. we could make compatibility much easier. As of now, it can be pretty hard to update instrumentation.

FWIW all the instrumentation from opentelemetry-js-contrib seems to already do it that way.

Closes #4975

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.17%. Comparing base (6096f72) to head (e7c86ed).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5165      +/-   ##
==========================================
- Coverage   93.18%   93.17%   -0.02%     
==========================================
  Files         314      314              
  Lines        8076     8076              
  Branches     1622     1622              
==========================================
- Hits         7526     7525       -1     
- Misses        550      551       +1     

see 1 file with indirect coverage changes

[pichlermarc]

At the moment, these being pinned is fully intentional, but not optimal - a few reasons for this:

• this change will likely be reverted by release automation because it strict-updates dependencies from this monorepo
• we don't have proper documentation on @opentelemetry/core when exports were added, which makes it difficult to ensure that we depend on the correct minimum, which will drastically increase the time needed to review changes that involve @opentelemetry/core
• we don't have any way to automate checking if we depend on the correct minimum version in this repo and always the latest version from this repo will be linked, which introduces significant potential for human error during reviews

Given these problems, I don't think we can merge this PR as-is. I completely agree that this is annoying the way it is and I'd love to see this changed in the future, but that change has to be sustainable in the sense of how much effort we need to put in to maintain it long-term.

We do have challenges with regards to maintainability of the project already, and I think this change would not exactly improve the situation at this point in time. I'd much rather do other things like #5148 first and then re-visit this at a later point in time when we've reached a point where this is sustainable.

[mydea]

I guess I don't fully understand why this would not work for instrumentation? The spec explicitly says:

When new functionality is added to the OpenTelemetry API, a new minor version of
the API is released. These API changes are always additive and backwards compatible
from the perspective of existing Instrumentation packages which import and call
prior versions.

Which to me sounds as if e.g. relying on @opentelemetry/core: ^XXXX is what the spec describes? To be clear, what I'm saying is this could still bump the dependencies on every release, but simply preprend it with ^ instead of hard-pinning the dependency version.

Also, as I mentioned this is already the case for all instrumentation in opentelemetry-js-contrib and I don't see this being a problem there either?

[pichlermarc]

I guess I don't fully understand why this would not work for instrumentation?

Oh it does work, it's just slightly more complex than it initially seems. 😅

Let me explain:

First for the easy blocker: Our release automation currently bumps strict, so If I merge this PR, the release automation will revert it when we're prepping the next release. A PR making changes like this PR does will actually have to update the release automation too.

Then for the more complex one (the really annoying one) - the API package does add features in a way that are breaking for implementers. We do have packages that implement these interfaces - so the actual implementations of the API are also technically breaking the public API when we implement new features because the implementations are not compatible with other versions of itself. So when another package uses the SDK package in the public API, and it's not pinned, the public API of that package also breaks (very annoying). That's not the only problem - when I mentioned that exporting classes is a problem in #5148, I meant that private properties end up as part of the public interface - so technically we're often breaking these parts of the SDK that don't implement the API's functionality even when making small changes that shouldn't even concern the user (the user might not care about these internal changes, but the typescript compiler does, unfortunately), and even in packages that don't implement the API but everywhere. So while for the most cases we strive to adhere to SemVer these caveats that still come up and ruin people's days, because if we wouldn't allow ourselves to break private properties, then, in many cases, we'd not even be able to fix the most basic of bugs.

Now, imagine that breakage happening depending on which version your npm install resolves - now it's not only consistently annoying, but it's inconsistently annoying in many different ways depending on then different versions that npm decides to resolve at that given time.

That's a bummer, I agree, but it's the status quo (and it also haunts me in my dreams). Hence why some packages need to be pinned while others don't need to be pinned. In the contrib repo, we mostly depend on the API to do the work and the packages are leaf packages. Since we're only consumers there, that's perfectly fine to have a more permissive range. As long as the types don't show up in the public API of the package you're writing, it's unlikely to cause problems. It would even work here but we have to decide on a case-by-case basis what's fine and what is not fine.

I think this PR is unpinning prematurely, as removing the class exports and introducing factory-functions with interfaces as return-types will drastically improve the situation, as the internal stuff that's pseudo-public now will be completely hidden away and breaking changes will become significantly less common. This will drastically simplify the change you're trying to make today because there's less stuff that can go wrong - we'll be able to actually adhere to SemVer and open the floodgates for un-pinning - that's part of the reason why I proposed doing it in the first place. I just don't want unpin right now where it'll cause additional maintenance overhead and possibly prevent us from delivering that change, because we're stuck putting out fires, making us stuck for another year or two.

It will not solve all problems (the API implementer break one still applies), but it'll get us to a better place than we're in now. 🙂

In any case, once we've selected the packages that are fine to un-pin, we'll have to enforce that other packages that are intended to be pinned actually stay that way so that we don't accidentally un-pin it in a PR (I know the pitfalls but if there's no automation to enforce it, we're just putting ourselves in the world of hurt that I described above, as I also miss stuff during those reviews).

...
Which to me sounds as if e.g. relying on @opentelemetry/core: ^XXXX is what the spec describes? To be clear, what I'm saying is this could still bump the dependencies on every release, but simply preprend it with ^ instead of hard-pinning the dependency version.

Ah yes that's slightly better than what I originally though the PR would be - but most of my points above would still apply (even though I wish they did not), they'd just gradually start to haunt users 🙂

[pichlermarc]

A side note - I know that this change would likely be fine for the packages modified in this PR, but PR's like this tend to start of a series of PR by other people tying who are blindly trying to do the same to the rest of them - and that's where it will become a problem.

We'll eventually unpin it - but I want to make sure that

• a) we're ready and have a plan written down to point people to so that this can happen in a somewhat orderly fashion
• b) we have the bandwidth to deal with any problems that arise from it