-
Notifications
You must be signed in to change notification settings - Fork 834
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[instrumentation-fetch] Use msw
for fetch instrumentation tests
#5282
base: main
Are you sure you want to change the base?
Conversation
18cb1ae
to
71d13f5
Compare
Discussed this at the Dec 18 SIG meeting, the group seems generally onboard with trying this out, so I'll keep pushing forward on this and report any issues with the approach |
71d13f5
to
360a65e
Compare
experimental/packages/opentelemetry-instrumentation-fetch/test/fetch.msw.test.ts
Fixed
Show fixed
Hide fixed
experimental/packages/opentelemetry-instrumentation-fetch/test/fetch.msw.test.ts
Fixed
Show fixed
Hide fixed
experimental/packages/opentelemetry-instrumentation-fetch/test/fetch.msw.test.ts
Fixed
Show fixed
Hide fixed
experimental/packages/opentelemetry-instrumentation-fetch/test/fetch.msw.test.ts
Fixed
Show fixed
Hide fixed
experimental/packages/opentelemetry-instrumentation-fetch/test/fetch.msw.test.ts
Fixed
Show fixed
Hide fixed
experimental/packages/opentelemetry-instrumentation-fetch/test/fetch.msw.test.ts
Fixed
Show fixed
Hide fixed
msw
for fetch (and XHR) instrumentation testsmsw
for fetch (and XHR) instrumentation tests
Discussed in the SIG meeting today, the group is generally in favor with the approach, I'll clean it up and get things ready for merge. |
c764f0d
to
549841b
Compare
experimental/packages/opentelemetry-instrumentation-fetch/test/fetch.test.ts
Fixed
Show fixed
Hide fixed
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #5282 +/- ##
=======================================
Coverage 94.58% 94.58%
=======================================
Files 318 318
Lines 8069 8069
Branches 1701 1701
=======================================
Hits 7632 7632
Misses 437 437 |
549841b
to
ed3d379
Compare
msw
for fetch (and XHR) instrumentation testsmsw
for fetch instrumentation tests
f405cbc
to
b4c7074
Compare
Also ran `npx msw init test --save` in inst-fetch and adjusted the karma config to make the generated worker file available at root.
These tests doesn't actually need the complicated setup to work, so seperated them out into a new suite. Spiritually should be the same coverage as before, with a new test for `{ enabled: false }`.
Without this, the next set of test fails intermittently
b4c7074
to
07ec36c
Compare
Tightened some of the assertions in the old tests where they were unnecessarily vague (e.g. protocol is either "http" or "https" when the test setup already fully dictated which one will be used). This shows the promise of the new appraoch – we are able to test the instrumentation without mocking the browser APIs at all, fully execrising the instrumentation code including `PerformanceObserver` and the resource-timing matching code. We did cheat and added a 500ms delay to make this work, so we do have to come back and deal with that at some later point.
For the most part, it's a pretty straightforward port, but also improved clarity with a `describe` grouping and actually assert the headers in each test.
This explictly tests that the trace propagation headers are NOT added when there are no global propagator set. This wasn't in the old tests but seemed like good coverage to have.
As pointed out open-telemetry#5122, the expectations in the existing tests are not the real browser behavior (which was ultimately the motivation for refactoring these tests to rely on real browser behavior as much as possible). So instead of porting them as-is, this adjust the tests behavior to match what actually happens.
As with the CORS tests, the preflight span does not exist IRL, so those tests were dropped.
It's unclear what these tests are trying to validate. For the most part, the `fetch()` API and its instrumenation behave similarly between successful and unsuccessful, so this seems to be largely duplicating coverage from the other tests without making any other meaningful assertions.
07ec36c
to
9f8149c
Compare
@pichlermarc and other reviewers, this is not ready for merge just yet (see TODOs) but the code should be ready for review! For your convenience, I broke up this PR into a series of commits, each porting one group of related tests, I'd recommend reviewing it that way. I also mostly kept the tests in the same order, if you want to just open both files side-by-side and review that way. Would appreciate it if anyone has time to do a pass, even if incomplete, so we can discuss any issues/feedback on the Wednesday SIG 🙏🏼 |
Which problem is this PR solving?
Currently the test suite heavily relies on mocking to work, specifically it uses
sinon.stub(window, 'fetch')
to replace it with a custom function that is intended to work similarly to the real one. However, the actual behavior offetch
is very complicated and is very difficult to get right, with a lot of subtle interactions around CORS, redirects, resource timing, etc.Issues like #5122 is kind of an extreme example of the problem (assuming the report is correct), where the current code and the "fake fetch" are assuming a browser behavior that didn't actually exist, and the current tests are explicitly confirming that non-existent behavior based on the bad stub.
This is a problem, considering that the point of the this package (and consequently, what we are trying to test) is:
Essentially, we are testing that "if our assumptions about how the browser works is correct, then our instrumentation code behaves correctly".
The current factoring of the tests – a big setup block that needs to handle all the various edge cases under test – is also quite difficult to work with. It would be quite complex to add additional test coverage for things like redirects and opaque responses.
In the case of
fetch
, there is a better way to go about this and use as much of the real browser functionalities as possible.Service Workers allows webpages to intercept network requests (like
fetch()
and XHR, but also other resources), and msw is a library to make that work nicely for testing/development workflows. Essentially, it allows for mocking/intercepting anyfetch()
requests in the tests while still invoking the actual nativefetch()
function in the browser.Improvements Made
fetch()
requests are actually made, and all timing information came from the real resource-timing browser APIsany
)Future Improvements
Potentially these tests can be run in additional browsers (e.g. Firefox) to improve coverage by changing the karma config. This will ensure we aren't accidentally targeting any chrome-specific behavior in the instrumentation code.
This approach also works with XHR, so we could also do the same refactor to those tests. (I'm unsure if I'll personally have time to finish that, but I think set up a good foundation/framework to follow if someone wants to do that.)
TODOs
fetch()
to ensure the resource-timing information made it to the instrumentation'sPerformanceObserver
(plus the instrumentation code also has its own 300ms artificial timeout)