Report v3 attachments to VSTest

[bradwilson]

xunit/xunit#3063
xunit/xunit#2457 (comment)

[matt-richardson]

Digging into this, we found that this tidbit - the attachments are available on ITestFinished, but the test results are sent on ITestFailed/ITestPassed/etc.

Thinking about it, we could cache the attachments in the OnTestFinished event, and send them on the OnTestFailed event (or vice-versa. depending on which event comes first).

Or, we could revisit the decision to put the attachments on ITestFinished.

I'm happy to contribute a PR if it helps - would be very useful to get your input on it before I start on it @bradwilson.

[bradwilson]

I've been thinking about this, and it's not a simple answer.

The test messages come in this order: test starting => test result (passed/failed/etc.) => test finished

Probably the best way to do this is to short-lived cache the result message (via a ConcurrentDictionary, keyed by the test's unique ID) while waiting for the finish message, and then construct the test result during finish rather than result.

If you want to PR this, I'd be happy to accept it; if not, I'm also happy to do it. 😄 I definitely want to get it done before shipping 3.0.2 (which will co-ship with v3 1.1.0, which seems like its not-quite-10 feature/bug commits is probably close to ready to ship).

[matt-richardson]

I've had a play with this, trying to get it working, and I'm not having much luck.

Rider doesn't appear to support test attachments, and it's not working so well via TeamCity (maybe a faulty assumption (that it should work) on my behalf in there).

There also seems to be a mismatch how things plug together - xunit messages have a string/byte array, but the attachments (on Microsoft.VisualStudio.TestPlatform.ObjectModel.TestResult) expect a filename.

Finally, the only way I could get the tests to work was to #if NETCOREAPP - it was hanging when accessing args.Message.Attachments under net472.

I'm just about to head on vacation for a couple of weeks, where I wont be able to look at this, so if you're happy to tackle it, feel free to go for it.

[bradwilson]

Thanks for the research. The hang on net472 is surprising and worth looking into.

As for everything expecting filenames, I think the answer is to write things to files in the Temp folder, and the assumption is that VSTest owns those things. If it doesn't take a copy up into itself, then the Temp folder will be cleaned out when the computer reboots. Not optimal, but I suspect that's what they intend so they'll take ownership of that file (and hopefully delete the Temp version).

[matt-richardson]

Pushed up the code in case it's helpful 👍