Structural fix the issue "Could not load file or assembly" in the Visual Studio Extension

[304NotModified]

I know this isn't easy, but there are many reports regarding this issue. It would be great if we could structural fix this issue.

Maybe also useful, this analysis: reqnroll/Reqnroll.VisualStudio#75 (comment)

[304NotModified]

@gasparnagy are there anu ideas to load the assemblies in another way?

[Code-Grump]

Am I being stupid to suggest loading the assemblies into their own isolated load context?

[clrudolphi]

Am I being stupid to suggest loading the assemblies into their own isolated load context?

No. That is the heart of the solution. The complicating factor is that we mix user supplied binding code and Reqnroll generated code in the same "root" assembly. The trick then is to isolate use supplied code assemblies from Reqnroll assemblies.

[clrudolphi]

In other words the ideal structure would be for feature code-behind generated code to be in a separate assembly from the user supplied binding code. Today these are in the same project and thus end up in the same assembly.

Are there approaches to the build that could get us that?

[Code-Grump]

I'm afraid there's no trivial way to make a .NET build produce a second assembly. Essentially we'd have to create our own call to CSC.exe to produce an additional assembly. That's not simple but understandable. The tricky part is all the dependency resolution required to assemble a workable build, plus the challenge of actually loading the information.

However, no matter which way you slice it, generated code takes a dependency on user code. If you want to load the types that are generated, they will have a dependency to the user's steps that need to be invoked.

Fundamentally, if the user's code is able to be loaded into a test runner and executed without running into dependency problems, I don't understand what makes repeating that isolated load a challenge for the Visual Studio Extension. Could you elaborate?

[clrudolphi]

My understanding is that we have reported instances of dependency version conflicts in both the VS Extension and at run-time. The solutions for these might in fact differ. I suspect that for the VS Extension that the solution will involve loading user types into a metadata-only load context. This would involve changing the techniques we use for binding discovery.
For run-time problems, the I'm presuming that the solution would involve loading user types into an isolated assembly load context.

[gasparnagy]

Let me summarize the problem and the possible approaches that have been considered in the past, because I am afraid that you don't have the full picture and therefore you are not necessarily optimizing in the right direction.

Long post alert...

How the binding discovery works in VS extension right now

There are also some legacy approaches but currently the most of the "binding discovery" is done via the "generic reqnroll connector" (https://github.com/reqnroll/Reqnroll.VisualStudio/tree/main/Connectors/Reqnroll.VisualStudio.ReqnrollConnector.Generic). This basically works in the following way:

1. It is a standalone process that is invoked by the VS extension via process execution with some command line arguments and the response is provided via console output. So this process is fully isolated and does not depend on the assemblies loaded into the VS extension or to VS.
2. The connector process does two things:
   1. Queries the binding information from the complied test assembly by loading the assembly and invoking the BindingProviderService.DiscoverBindings static method. This method sets up a basic simplified execution environment using the DryRunBindingInvoker (see here), so that no user code is actually invoked but otherwise everything is like in a normal execution: plugins are loaded, configuration is processed and external assemblies are also loaded and investigated for binding data. So finally the BindingProviderService.DiscoverBindings will return the complete binding data, including external assemblies and customizations made by plugins. (Note: the plugin codes must run for this, so reflection-only models would not be sufficient)
   2. As the binding data only contains the names of the binding methods, but we need the source location as well (if available), the connector also tries to get the source location for all mentioned binding data using the information in the PDB file of the test assembly.
3. Although the connector is fully isolated, it still creates an Assembly Load Context and applies some assembly loading resolution magic, because this is an executable in the VS extension folder and normally .NET would only load dependencies from the connector folder using the "deps.json" file of the connector itself. But we need to load the dependencies of the test assembly from the folder of the test assembly and resolve the dependencies based on the "deps.json" file of the test assembly itself. This is the real problematic part, because framework assembly resolution is pretty magical and it also depends on the SDK being used (e.g. an ASP.NET project dependencies are resolved differently). ‼️Please remember this point because this is the root of the issues with the current solution.

This connector model generally supports test assemblies depending on different versions of assemblies than Reqnroll itself. In that case the "deps.json" file of the test assembly contains the calculated dependency versions and the connector will load that. The problem is that for some tricky dependency combinations are simulated loading resolution does not work correctly.

The current connector can handle the most project needs. I can have an exact value from the telemetry but it most be something above 95%.

Why do we load the binding information from the compiled test assembly

Loading the binding details seems to be an easy task: just go through the types and look for the Given, When and Then attributes. But in reality the thing is much more complicated. Here is a list of problems that we have encountered in the past and why we decided to process the binding attributes from the compiled test assembly. (I try to make a complete list, but not sure I can recall all of them.)

• Usage variations

  • constant values and expressions can be used in attribute parameters: [Given(SomeClass.MY_EXPRESSION + "foobar")]. The constant can be defined even in another assembly.
  • derived attribute classes can be used (e.g. for language localization): GegebenSeiAttribute: GivenAttribute
  • type aliases for the attribute names can be used
  • bindings from external assembly can be used that is provided as a project reference
  • bindings from external assembly can be used that is provided as a package or dll reference

• Version mix (there is one VS extension and it has to work with multiple Reqnroll versions if the user works in different projects)

  • The attribute parameters might be different in different Reqnroll versions (e.g. we added an ExpressionType setting recently)
  • The attribute values (e.g. expressions) might be parsed differently in different Reqnroll versions
  • Validation rules might be different in different Reqnroll versions
  • We sometimes introduce new configuration options that changes the binding discovery behavior
  • Cucumber expression parsing depends on an external package (Cucumber.Expressions) and different Reqnroll versions might depend on different versions of this package

• Plugin support

  • Plugins can register step definitions
  • Plugins can modify the processing behavior of the standard attributes
  • Plugins can change the configuration

• Maintanance

  • The Reqnroll runtime currently collects the binding information via reflection. Any other processing option (source code, reflection-only info, Roslyn AST, etc) would require us to duplicate the processing logic. We have tried to make a generalized model. You see the remaining of it with the interfaces like IBindingType or IBindingMethod that suppose to be abstractions over the different source models, but altogether this was pretty hard to maintain.

For the reasons above at some point we decided to only keep the runtime binding discovery but that comes with the price of the need to execute the binding discovery logic on the compiled test assembly.

Options that have been considered so far

When considering improved options, we can generally choose from two main strategies:

1. Make a "perfect" binding discovery solution that is able to provide the complete binding registry, with the same functionality as the connector has
2. Make a "good enough" secondary alternative solution that provides some result for the most important usage scenarios if the connector approach does not work. E.g. if the project has some tricky dependency structure that the connector cannot handle, the normal Given/When/Then attributes can be still processed and the user gets an almost perfect IDE experience

The concrete options that I have seen or considered:

1. We could further fine-tune the connector and somehow find out why it fails to process some projects and fix it.
2. Replace the connector with a "dry-run" approach. This would work in a way that the VS extension would invoke "dotnet run", but with an environment variable setting that enabled "dry-run" mode (so no actual test code execution is done) and also somehow configure the execution to save the binding information to a file. (The actual binding discovery is done in before-test-run time, so the dry-run test executions are not important, but we need at least one test to trigger before-test-run.) The benefit of this would be that the tests would be executed exactly in the same way/context as with their normal execution, so if the tests can execute the discovery can also be executed and we would not need to deal with the tricky assembly loading issues. There are some down side of this:
   • If the project has no feature file and scenarios yet, the test frameworks will not detect any test so Reqnroll will not be invoked. So you need to have at least one scenario for this to work (or we need to generate a dummy test if there is no scenario yet)
   • If there are many (>1000) tests, then even a dry-run would take some time (5000 tests in dry-run takes 20 seconds on my machine). So for bigger projects this approach is not ideal, unless we can filter the test execution somehow to limit the number of tests, but somehow the VS extension could randomly select a scenario and somehow construct a filter condition that limits the execution to only that scenario.
   • We would still need to solve the source code reference resolution from PDB, but that's an easier problem.
3. Process the standard attributes from Roslyn AST as a preliminary binding information (you don't even need to compile the project for that) and once the project is complied, invoke the current connector solution. If that succeeds, we replace the binding info based on that, otherwise we just keep the preliminary one. This could be further optimized, for example if you modify a step definition class after a successful compilation, we could still process the standard attributes of the modified class and extend the existing binding info with them. This would have the benefit that it would give some basic support even without compiling the project.
4. The concept of @Code-Grump in PR Roslyn Analyzers and Compile-Time Binding Registry #541 is also a possible improvement. Basically that collects the core binding information during compilation and saves it in some form to the test assembly. The VS extension could read the info out without the need to processing the bindings. The downside of this approach is that it can only handle the standard cases, anything that comes from plugins or special customizations cannot be discovered this way. Still this could work as a "good enough" strategy that works when the connector fails to build the complete discovery info.

[clrudolphi]

so when you say version mismatches, I guess you mean the non-reqnroll dependencies that the generic connector has (e.g. system.collections.immutable, etc) and the dependencies that the users reqnroll version or the test project has, right?

Yes.

[clrudolphi]

Regarding first: Could we make simply a .NET Framework compilation of the generic connector? I wonder why we didn't do that originally...

That is my intent as a follow up experiment.

The generic connector will not compile under net48 as the concept of an AssemblyLoadContext does not exist.

[clrudolphi]

Regarding the second issue, I spent the day yesterday attempting to use MetadataLoadContext by building a prototype replacement for the RuntimeBindingRegistryBuilder.

Here is a demonstration of using a MetadataLoadContext to build the binding registry during VS Extension discovery. I addressed the issue that MLC cannot read attribute Properties by taking a hybrid approach that creates instances of Reqnroll attributes in order to read their properties.

While this works, I suspect we may not need it if @gasparnagy's solution to the assembly loading issue (below) is sufficient.

[gasparnagy]

While this works, I suspect we may not need it if @gasparnagy's solution to the assembly loading issue (below) is sufficient.

I think the biggest issue with this approach that it will not be able to run plugins that may influence the result. But could be considered as a plan-B approach.

Have you checked that the metadata loading solves the assembly loading issues? Because my initial suspicion would be that this still needs to load the references and if something cannot be found, it will not be found this way either. But this is only a speculation, I might be wrong.

[clrudolphi]

The global container is constructed as normal; is my assumption correct that this causes the plug-ins to be loaded?

The MetadataLoadContext is used only when loading types to be passed to the BindingSourceProcessor.

I have confirmed that this approach does Not solve the assembly loading issues that we are seeing.
For one example project the problem is an attribute that lives in the GAC.
The other example project has a version conflict between the test project and a plug-in. Since plug-ins are loaded prior to this code executing the exception occurs before the use of the MLC.

[Obelixx]

Here is a reproduction of not working step navigation with .Net 10 and MSDI: https://github.com/Obelixx/ReqnrollTestOnDotNet10AndMSDI
If the project dependency ("TestWithNet10AndMSDI") is removed from the "TestWithNet10AndMSDI.Tests" project navigation works. But I want to test things inside TestWithNet10AndMSDI project.

[gasparnagy]

Thx, this is definitely useful.

[gasparnagy]

I did analyze this. The reference "TestWithNet10AndMSDI" uses the Microsoft.NET.Sdk.Web. If the reference is not added, the TestWithNet10AndMSDI.Tests.deps.json file in the output dir that the connector uses to resolve the dependencies contains the details about Microsoft.Extensions.DependencyInjection package and the connector can load the dependency. I can even force using a different version of it by specifying an explicit package reference. Everything works fine.

But once the reference is added, the compilation removes all mention of the Microsoft.Extensions.DependencyInjection package completely. Like it would be obvious that that one is needed. And because it is not mentioned, the connector has no idea how to find it and fails.

It is clear from this that we still miss an additional trick that can be used to load the references of such projects. I just don't know what is the trick. Keep trying...

[gasparnagy]

Just after hitting send, I found a fix for this particular one! Needed to change the line https://github.com/reqnroll/Reqnroll.VisualStudio/blob/fdbc1d688b6e7aca6dc0c0a9be1d1128220dbfa8/Connectors/Reqnroll.VisualStudio.ReqnrollConnector.Generic/AssemblyLoading/TestAssemblyLoadContext.cs?plain=1#L169 to Path.Combine(assemblyName.Name!, $"{assemblyName.Version.Major}.{assemblyName.Version.Minor}.{assemblyName.Version.MajorRevision}".ToString()), (remove the 4th version number).

I have to leave now, but continue on Monday.

[clrudolphi]

Did you mean line 165 of the TestAssemblyLoadContext.cs file?
I tried this and did not see any improvement (same load exception). Also tried using assemblyName.Version.Build instead of MajorRevision.

I assumed that rebuilding the VisualStudio Extension solution and launching the experimental hive is sufficient. Or does the changed connector need to be rebuilt using the ps1 file before attempting to use it?

[clrudolphi]

For the specific situation of failed loading of attributes from the GAC, I've put together PR #1006. This simply puts a try/catch around calls to .GetCustomAttributes() so that we can continue processing rather than hard-failing. Let me know what you think.

[gasparnagy]

That might be definitely valuable. I'm trying to simplify and refactor the connector now, so that we can do the improvements and prototyping more easily (hopefully finishing today) and then get back to that one.

[clrudolphi]

yeah, I see your PR for the refactor. While you're at it, I am curious about a few things.

First, this method in BindingRegistryFactory

    public IBindingRegistryAdapter GetBindingRegistry(AssemblyLoadContext assemblyLoadContext, Assembly testAssembly, Option<FileDetails> configFile)
    {
        var configFileContent = LoadConfigFileContent(configFile.Reduce((FileDetails)null));

        var reqnrollAssembly = assemblyLoadContext.LoadFromAssemblyName(new AssemblyName("Reqnroll"));
        var bindingProviderServiceType = reqnrollAssembly.GetType("Reqnroll.Bindings.Provider.BindingProviderService", true)!;
        var bindingJson = bindingProviderServiceType.ReflectionCallStaticMethod<string>("DiscoverBindings", new[] { typeof(Assembly), typeof(string) }, testAssembly, configFileContent);
        var bindingData = JsonSerialization.DeserializeObjectDefaultCase<BindingData>(bindingJson, Log);
        return new BindingRegistryAdapter(bindingData.Reduce(new BindingData()));
    }

Why have the static method within the Reqnroll BindingProviderService return a string? Why not return a structured DTO? Is the effort to establish a shared contract assembly too complicated? The extra serialization/deserialization costs time. And does it not also raise the risk of conflicts of versions of System.Text.Json (that is, invoking serialization within the Reqnroll BindingProviderService)?

And since you've gone to the trouble of loading Reqnroll into an isolated assembly context, why bother with executing it as an Out-of-Proc connector?

[gasparnagy]

Why have the static method within the Reqnroll BindingProviderService return a string? Why not return a structured DTO?

This is a very important question. If we would return a DTO, the reflection infrastructure would try to load the same assemblies to both the connector and the load context. Loading it in the connector causes extra issues (this is what we did many years ago), so finally the best option is to use data types on the interface that do not depend on any additional assemblies, so string is good. The time penalty is valid (fortunately not significant), but the System.Text.Json is not a problem, as the string does not require to be serialized/deserialized with the same version of it (one is loaded to the default load context of the connector process the other to the load context we create for the assembly).

The discovery is performed asynchronously, so the user does not need to wait for the result actively. Therefore this is not time critical: if it finishes within a few seconds, that's enough.

And since you've gone to the trouble of loading Reqnroll into an isolated assembly context, why bother with executing it as an Out-of-Proc connector?

This is another good question. The assembly context ensures isolation, but cannot fully be disposed. If we would load it to the VS process, the memory usage would keep growing as we perform discover more time. Also sometimes it locks the assemblies causing issues with subsequent builds. So this is safer this way.

[gasparnagy]

The refactoring of the connector is completed. The dev workflow should be much easier now.

Additional testing tools:

• If you work with the Reqnroll generic connector only, you don't need to run Connectors\build.ps1 anymore because the local build handles it. It can now also build the .NET 10 version as well.
• The Reqnroll.VisualStudio.ReqnrollConnector.Tests now contains a test class called ExternalSampleTests. This is primarily used as a smoke test to ensure that the projects the connector could handle are still handled. It works in a way that it clones a repo (currently https://github.com/reqnroll/Reqnroll.ExploratoryTestProjects and https://github.com/reqnroll/Sample-ReqOverflow) to the temp folder, detects any Reqnroll projects in it, compiles them and tries to process them with the connector.
• I have already identified two projects in Reqnroll.ExploratoryTestProjects that the connector cannot handle currently (these are ignored now):
  • SpecFlowCompatibilityProject.Net472 - this is because of Using App.config with SpecFlowCompatibility plugin does not work #1010, basically the entire test project does not work, if we decide not to fix 1010, we should delete this
  • CleanReqnrollProject.Net481.x86 - this problem is related to the fact that the generic connector is currently not compiled of .NET Framework. Not that critical, but let's keep it in mind
• I have started to port relevant repro project to https://github.com/reqnroll/Reqnroll.ExploratoryTestProjects/tree/main/VsExtConnectorTestSamples. Basically I would like to have all those projects here that we want to use as a regression detector. (Maybe these should be moved to the https://github.com/reqnroll/Reqnroll.VisualStudio repo eventually. Let's see.)
• The ExternalSampleTests allows you to add a test for a local folder as well. This is practical for local dev when you try to test/debug a connector for a specific local project.
  • The "debug into the connector" experience is not complete yet in ExternalSampleTests, but we can improve that later.

Now I will make a PR for my one-line fix, but there is a problem with it anyway... I will comment it to the PR.

[gasparnagy]

Update: We have made two small (but hopefully effective) step to address this problem:

• Reqnroll v3.3.3 improves the cases where the binding discovery has failed because of a type or member that was anyway not related to Reqnroll bindings
• Reqnroll Visual Studio extension v2026.1 improved the dependency handling in some situations and also made the logging a bit better

This two steps fixed the two problems that we had a repro for, but hopefully even more. Let's see.

I will ask the issue submitters to re-test their problems on Friday, because it takes a few days to have the new Visual Studio extension to be auto-updated to all users.