Could not load file or assembly 'nunit.engine.api, Version=3.0.0.0,

[runehalfdan]

#475

I trying out 3.12 beta of the console-runner. Only change is to
<PackageReference Include="NUnit.Console" Version="3.11.1" />
to
<PackageReference Include="NUnit.Console" Version="3.12.0-beta1" />,
and referencing 'nuni3-console.exe' from the beta-folder.
All other packages is latest version, and targetting .NET Framework 4.8

Full output:

Running only unit-tests
NUnit Console Runner 3.12.0-beta1 (.NET 2.0)
Copyright (c) 2020 Charlie Poole, Rob Prouse
mandag 3. august 2020 15:20:33

Runtime Environment
   OS Version: Microsoft Windows NT 6.2.9200.0
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    .\ProjectTests3.nunit

Test Filters
    Where: cat != MANUAL and cat != DATABASE and cat != NIGHTLY

Could not load file or assembly 'nunit.engine.api, Version=3.0.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb' or one of its dependencies. The system cannot find the file specified.

[ChrisMaddock]

Hi @runehalfdan. Thanks for trying the Beta, and reporting this! I’ve got a suspicion as to what the problem might be - will try and dig into this later this week.

[CharliePoole]

@ChrisMaddock I'm wondering if this relates to loading of extensions built against the 3.0 api. If that's the case, changing the API to be versioned could be a breaking change. OTOH if it were that, I'd think that our tests would have caught it!

[ChrisMaddock]

@CharliePoole yes, I think so. That puts us between a bit of a rock and a hard place with the API versioning breaking things - we get a break both if we change it and if we don't! The only think we could have done right would be to never change add those properties, but I think that ship has sailed at this point, unfortunately.

Long-term, I think this is another great argument towards separating out the 'extensions' part of the API from the 'test-running' part, like we mentioned in #770. (Admittedly not with this in mind!) That's definitely a v4 type change however. In the short-term, as much as I hate to suggest it, I think we might need to put in a binding redirect?

I don't think we have any tests covering running extensions - that's definitely on the integration test wishlist! As part of my manual testing when doing a release, I normally run --list-extensions to check extension loading is still happy - I expect an issue like this wouldn't show until runtime however, which wouldn't be picked up by --list-extensions.

[CharliePoole]

@ChrisMaddock Not sure why the ship has sailed! This is just a beta so far. What if we back out the version change? Workaround for the user then is to upgrade both NUnit and whatever (R#?) was exposing the conflict.

[ChrisMaddock]

@CharliePoole

My reason for not being keen on that idea is that I don't think this is such a niche case that R# has exposed - it's just the first time anyone has been able to diagnose it correctly! I think this issue was also reported as #424 a good number of times over the last few years. In some of those cases, it's being reported by people who have the console and the adapter installed in the same project - which I would say is a much more 'normal' thing to do, now solution level NuGet packages no longer exist.

I'm concerned that if we don't start versioning the API, this will continue affecting people every time we change to API assembly. That's not really a corner I want to be backed in to.

(And, just to reduce confusion, with "the ship has sailed", I was referring to the fact that we shipped API assembly changes back in 3.8 and 3.11. Fully agree we can change things in the Beta, but I don't think there's a lot we can do about those previous versions of the engine that are already out there! 😅)

[CharliePoole]

@ChrisMaddock As a mitigation, what if we put out a docs page that lists compatible console and adapter versions for folks who install both in their projects.