Skip to content

Conversation

shantyk
Copy link
Collaborator

@shantyk shantyk commented Sep 24, 2025

A test framework proposal for NuGet Inspector because

  1. It is a neglected and under tested code base.
  2. Internal cascading logic that is not tested.
  3. NuGet version support upgrade isn't straightforward: .NET developers most often use VS or dotnet to build their projects and add dependencies. Version of NuGet is under the hood and most likely our users just run NI without paying much attention to our compatibility matrix.

There is plenty of room for improvement here. Because the tests are not dockerized and expect the machine that is running them to be set up a certain way, we will need to make sure the build machine for our pipelines is and stays configured in a certain way. Good thing is, our Jenkins machine already IS configured with multiple versions of dotnet and newer versions have been added. This does make running them locally difficult. But maybe worth it if it means usually we run these tests only in Jenkins anyway?

Essentially:
-> Set up environment with appropriate version of dotnet.
-> Create test solution, add projects, add dependencies all with that dotnet version.
-> run dotnet list/restore as source of truth.
--> manipulate solution directory as needed to trigger different cascade branches.

Pros:

  • increased test coverage by 11.33% -> 44.92%
    (ran dotnet test --collect:"XPlat Code Coverage" and got coverage line-rate="0.44920000000000004" branch-rate="0.3803" version="1.9" timestamp="1759933557" lines-covered="1076" lines-valid="2395" branches-covered="286" branches-valid="752" for this branch. For master the output is: coverage line-rate="0.1133" branch-rate="0.1103" version="1.9" timestamp="1759933682" lines-covered="271" lines-valid="2391" branches-covered="83" branches-valid="752")
  • Less overhead than existing QA tests, which would only expand as we try to confirm support for new versions. Though I guess expecting different versions of dotnet to be presintalled is not exactly 0 overhead.
  • easier debugging locally
  • Performance and memory (I think the docker approach would require more memory? Or maybe the "have and keep a bunch of dotnets on the machine" is worse off down the line?)
  • unlike dockerized tests, we save time on start up/tear down and the need to download dotnet each and every time.
    Cons:
  • Jenkins and local set up differences can be subtle and hard to pin down.
  • Difficult to isolate tests that use MSBuild APIs (important for when updating our own .NET version). Usefulness of using MSBuild APIs is huge question.
  • Pro is also performance? Dockerized tests require time to start up/tear down ... download dotnet each and every time ...

Closes IDETECT-4830, 4827, 4826

@shantyk shantyk changed the title Skeleton for parameterized test framework Working skeleton for parameterized test framework Sep 24, 2025
@shantyk shantyk self-assigned this Oct 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant