Tree format 'Test list' and grouping 'By Category' => no categories are displayed

[rowo360]

Describe the bug
This bug is a continuation of the work to support the different tree view display formats (#1101).

Maybe this issue is no bug at all, but instead the expected system behavior. But I was irritated for a long time and thought something wasn't working properly. Now that I have analyzed, debugged and understood it, I can explain the behavior. But maybe other users feel the same way and we can support them better.

To Reproduce
Steps to reproduce the behavior:

1. Open test project which contains some test categories
2. Switch tree format to 'test list'
3. Switch grouping to ' By Category'
   => Categories in test project are not found!

However if I switch to grouping 'By Category (extended)' the categories are suddenly displayed:

Expected behavior
I expected that the categories are already shown when selecting 'By category' grouping.

Environment (please complete the following information):

• TestCentric Version - latest developer build 2.0.0-dev00451
• OS: Windows 11

[rowo360]

The actual root cause of this behavior is the location at which I defined the test category attribute.
In my case I defined them on the TestFixture level, only.

I was not aware of that it make a difference!
But while debugging this use case, I noticed that when working in 'test list' mode, the TestGrouping only receives the XML nodes for single test cases. And it only checks, if this test case has the category set - which is not the case. Therefore it won't find the catgories.
Having understood this, I added a test category to test cases directly and yes indeed, now the 'By category' grouping work as expected.

For a long time I didn't understand the difference between 'By category' and 'By category (extended)'. Now, as I'm aware of the XML and debugged it, I know that the extended mode will search also in the anchestors for a matching category.
Therefore the categories were found in my initial experiement when I used 'By category (extended)' - because the category at the test fixture were found.

When I started this research, I thought that it didn't matter where I defined the attribute (Fixture or test case level). In both cases I expected that my test case will have the category.

Overall, I wonder if users will understand the difference between 'By category' and 'By category (extended)'? Maybe a different naming will help? Or we can decide that we only support the extended mode?

[CharliePoole]

By Category (extended) was added at somebody's request. I never liked it that much. Let's think about what we really want here. Maybe if you do a test list, you should be able to group by the categories on tests and if you do a fixture list you should be able to group by the categories on fixtures.

There's a similar discussion that regularly takes place around properties. People get confused because they imagine that putting a property on a fixture is the same as putting it on each test in the fixture, but that has never been true in NUnit.

Let's think about where this should go before doing anything big on it.

[rowo360]

Before we continue the discussion, here is one more observation. I also did not expect this behavior and find it irritating
Now that I know that it makes a difference where the test category attribute is placed, I made some experiment with 'TestFixture' TreeDisplayFormat:
If I place a test category attribute on a test case, this category is not displayed in the tree when using 'TestFixture' TreeDisplayFormat and grouping 'By category' nor 'By category (Extended)'.
In both cases the tree lists the test fixture/cases beneath the category node 'None'.

From technical point of view this behavior can be explained again easily: the grouping logic only considers 'fixture nodes' and in extended mode it considers also the anchestors. But here the test category attribute is only present at some child test case node.

[rowo360]

Here's an attempt to describe my expectation:
I use the test categories to execute specific subsets of tests in a test suite. This is particularly helpful in large test suites and long-running tests. I specifiy the test category in the '-where' command line argument when using the NUnit.ConsoleRunner. Or I specifiy the test category in the 'category' tab using the TestCentric.GUI 1.6.4

So far I have never bothered to check whether the category attribute was defined on the test case or on the test fixture. This has never made a difference to me so far. And when I look at this use case, I think it is essential that the tests belonging to a test category are always selected reliably and without any doubts. If users start to wonder why a test was not executed, they may also quickly lose confidence.

So overall, when I select a category node in the tree, I expect all tests of that category to be executed. And I think we should aim for the same behavior when I execute a dedicated category using the '-where' parameter in the NUnit.ConsoleRunner. So in other words:

• Category grouping in 'TestFixture' tree display format: contains all tests with category regardless if defined on test fixture or on test case
• Category grouping in 'TestList' tree display format: contains all tests with category regardless if defined on test fixture or on test case

So the default 'By category' grouping will assemble category attributes from all locations. At least this would be my expectation - but it's difficult for me to anticipate the expectations of all the other users.

Of course, we can and maybe should also support another type of category grouping. Perhaps I need a little more time to think about the use cases for this. So the next proposal is more based on the current behavior than considering all possible use cases.

• 'TestFixture' tree display format: replace menu item 'By category (extended)' with 'By test fixture category'. This will only consider test category on test fixtures, but ignore them on test cases.
• 'TestList' tree display format: replace menu item 'By category (extended)' with 'By test case category'. This will only consider test category on test cases, but ignore them on test fixtures.

Depending on the active tree display format either one or the other special category grouping menu item is available.
Well, that's my first idea about it!

[CharliePoole]

I too will need some time to think about this.

This is similar to an issue that comes up with NUnit all the time: should properties be inherited? That is, when a property is added to the test fixture, is it also "present" on the individual cases in some sense? Does it affect them?

For properties in general, the answer is "it depends". An Order property on a fixture might impact the order in which that fixture is executed as compared to others in the same namespace. The same property on a test would change the order of executing that test as compared to other test cases in the same fixture. So "fixture order" should not be inherited by the individual cases.

But if we had an Author property, putting it on a fixture would most likely mean that the same person had written all the test cases. So, some properties are "inherited" (in our minds) while others are not.

Categories are basically degenerate properties. They have a name but no value. Their purpose is exactly what you say: to execute subsets of tests. So I think we should be able to come up with a standard way to display them without having two different menu items. Your proposal might do it.

But as I said at the start, I'd like to think about it for a while. Going back to bed now. :-)

[rowo360]

This issue is almost completly solved by PR #1118.
It was decided to remove the context menu 'By Category (Extended)'. The user has only one context menu entry 'By Category' to group by categories. The tree display format 'Fixture list' and 'Test list' will identify the test category attribute regardless if it's defined on test fixture or test case level and display the category in the test tree. (see also discussion in PR #1108)

Reading carefully through the previous comments I noticed that the case 'Fixture list' + 'By category' and test category is defined on test case is not working as expected yet. I need to continue to look at this case.

[CharliePoole]

What isn't working yet?

[rowo360]

I decided to create a separate issue for the last observation #1134. Allthough it might be technical related, it's a different use case.
Therefore I propose to close this issue, because the described use case is fixed already.