Living Documentation

[Code-Grump]

Having seen a few mentions of the SpecFlow Living Docs plugin, it seems like there's a demand for an equivalent feature in Reqnroll.

What I'd like to understand is:

• Does it matter when this output is produced?
• Should it be the output of a build, or a test-run?
• Would running a stand-alone tool to extract this information be preferrable to something built into Visual Studio?
• What do we think should be the output?

I can think of many ways to approach this, so knowing the constraints is probably the easiest way to find a path forward.

[gasparnagy]

Thx for taking care of this.

I don't think there are single answers for these questions (there are so many situation and needs), but let me share my thoughts based the usages of SpecFlow+ LivingDoc I have seen.

Does it matter when this output is produced?

The most common usage is to produce the report at the same time of the test execution or directly after in CI (or locally). SpecFlow+ LivingDoc CLI enforced a separate step (generated the data during test execution and run the HTML gen after), but I know that many teams tried to merge this by calling the HTML gen from an AfterTestRun hook.

Should it be the output of a build, or a test-run?

I think the most common usage is to use it with test-run, but I have seen usages where they were only interested in the nice rendering of the feature file and not interested in the test result. So both, but more commonly related to test-run.

Would running a stand-alone tool to extract this information be preferrable to something built into Visual Studio?

Yes, definitely, as it is going to be used mostly as part of the CI. I would even say a "stand-alone command line tool" (no UI).

What do we think should be the output?

I think there are diverse answers for this. The most simple and common usage would require a single-file active HTML (maybe multi-file HTML is also acceptable, but more problematic to share). But there are teams who use this as part of their QA process, where they might need to produce other output (PDF, Word, ???). But I think this latter is more the special case.

When I say active HTML, I mainly mean that something where you can browse the folders and feature files, search, etc. within the offline HTML.

Besides these questions, there are plenty of "nice-to-haves" that I have heard of. These are not necessarily part of the primary goal (or a goal at all), but good to think about them when we set the design. (SpecFlow+ LivinDoc supported some of these.)

• Customizable HTML output (templating?)
• Have summary controls (charts) in the result
• Non-HTML output (PDF, Word, JSON)
• Seamless integration of the result document to common ALM systems, like Azure DevOps, Jira, GitHub
• Manage result history

[tula-engineered-quality-hub]

Hi @gasparnagy! How is the live documentation part going? Publishing the HTML extension on Azure is a prerequisite for us to finally migrate from Specflow to Reqnroll and take advantage of all the improvements made. I must admit, I'm very excited about this.

I would like to understand the status of this request, if there is anything in progress, any planning, and if https://github.com/praticandobdd and I can actively contribute (if so, what contribution would add value), especially in terms of development and testing. We have a rich environment for testing this entire process, as well as experience to share with this community.

[gasparnagy]

@clrudolphi is currently working on the cucumber messages support that is a prerequisite of this (see #233), but we realized that it is a bigger task than we (or at least I) expected. Once this is done, we can work on the HTML generation.

In principle we could work on the HTML generation even before, because we can get sample Cucumber Messages reports from other Cucumber implementations. But to start with that, we would need some design concept at least. My preference would be to use React (like the old one did), so that we could reuse something from https://github.com/cucumber/react-components. But that's all I have and I'm not a React expert. If you (or anyone else) is fluent in React, you could start working on a prototype...

[blaisep]

Hi @gasparnagy ... (I just learned about this proiect from Lisa Crispin)
If there is support for python, then you can use the tooling like Sphinx or MKdocs that currently power https://readthedocs.io
Sphinx will output to many formats (PDF, Confluence, ePub, Man Pages, HTML, etc) and if you only need HTML then there are many more options.

The tricky thing about live docs is not so much the HTML design but rather the mapping of source code to abstractions like definitions (Glossary, API, etc) and other cross references. In this sense Sphinx, AsciiDoctor and Antora would be the tools to consider.

[gasparnagy]

@blaisep Thx for the comment. Yes, that's why we currently focus on generating cucumber messages that contain all these information. Once we have that we can find the right tooling to produce HTML. Do you know maybe any tool that can generate report from Cucumber Messages .ndjson file directly?

[a-korevaar]

Having a LivingDoc equivalent feature would allow us to migrate from SpecFlow to Reqnroll. I would gladly contribute to the development or testing of such a feature.

[Code-Grump]

Can you tell us what you need the output to be? Any technical constraints?

[a-korevaar]

We're currently using the SpecFlow Azure DevOps extension and the CLI to generate LivingDoc in HTML format and publish it externally. I have added some custom code to the build pipeline to include images (mainly screen mockups) as well.

For us, the integration in Azure DevOps is a nice to have, as developers are focused on the feature and step files in their local repositories and product owners and stakeholders use the published documentation.

The HTML format is ok, but it would be nice to have other options such as markdown or pdf as well.

I'm not familiar with Cucumber reports but it looks interesting. We are for the most part bound to on-premises solutions though.

[Code-Grump]

We are for the most part bound to on-premises solutions though.

This is good to know as a constraint. I'm certain it's a common one too.

[gasparnagy]

I agree. Test results are highly confidential in many environments I worked with and storing any test result data introduces a lot of complication (GDPR, etc.).

[adrichem]

I love this is getting attention! LivingDocs reports are the only useful source of test results for us. Especially the test runner mixes output from concurrently running scenarios.

1. We need to generate and store results in our CI/CD pipelines .My org needs flexibility in deciding where/when this work is done as we have different projects using different setups in their pipelines. Some are YAML, some are classic. Some are build- some are release. Some pipelines only run tests and have no 'real' branch associated with them. Having LivingDoc invoked by shell script ticks all our boxes w.r.t. integration into our devops.
2. Having HTML output is critical for us as we include the HTML into viewers configured in DevOps on the pipelines..
3. Its very nice LivingDocs contains the markdown from the scenarios. This helps communicate why tests do what they do.
4. When running locally we sometimes manually create a report (both for test results but also to show the content of scenario to someone in a nice format). So ability to execute multi platform is needed.
5. The Integration with Azure DevOps is not something we use yet. We are entering a situation where its going to be used. Even then not always.....for end-to-end tests there is no 1-to-1 correlation between test results and a specific branch.
6. We need LivingDoc 's ability to filter on outcomes.
7. The analysis on step-definitions for us adds no value as they are shared over multiple projects. Its normal for a step-definition to not be used in a test run.
8. We use the ISpecFlowOutputHelper to include OpenTelemetry TraceId's and links to logfiles (Blobstorage / S3 etc etc) into the LivingDoc report.

[gasparnagy]

Thx. The "filter on outcomes" is especially a good point that I did not think of. Also good to know your feedback about the analysis on step-definitions. I had the same feeling.

[Digbigpig]

Hi I'd like to chime in
In the past i've had a great experience using specflow and specrunner+. I'm now setting up a new test automation solution and realized we are without a reporting feature in reqnroll. Like others have mentioned, it could be a reason to delay reqnroll adoption over specflow.

The main feature we need is the ability to generate an html report that can be easily navigated and parsed. We'd like it to support navigation/ filtering between test categories. We'd like for easy screenshot integration into the report. Being able to see the step, the timing, and console output for each test is an expected feature.

I come from using spec runner and that was everything I needed. The livingdoc stuff looks cool and has nice features but some of it seems overkill for my general use case, however I never used it besides failing to integrate it with reqnroll :)

PS reqnroll sounds closer to rick-roll than rock n roll to me🕺

[gasparnagy]

There is no realistic chance that we will be able to redo the SpecFlow+ Runner. What would be good though is to make the livingdoc generator HTML "template" more extensible, so one could make a more "classic test report" kind of template for it (we can even put it to the product itself).

[brajpal-jangir]

Thanks for bringing this in discussion!

We used CLI command to generate reports. These are integrated in DevOps pipelines.

• In the active development phase, we generate living doc without results to let Product owner and other stakeholders review the test coverage. Living doc also have links to UserStory (from TFS) to validate the requirement.
• In times of GA, we generate living doc with results and archive them as test metrics as part of GA.

I also tried to use SpecFlow+LivingDoc DevOps extension, but it has some limitations and that’s where I switched to use CLI. Few limitations which I came across are

• We have multiple Specflow Projects in GitRepo and it’s not possible to publish multiple Living doc for single git branch. It will always override the previous uploaded document for same git branch.
• We also came across the issue when Living document HTML file size is more and it failed to publish document on DevOps. I reported this issue but am not able to find the link now.

[MathiasLopes]

I've found a way to generate LivingDoc HTML from Reqnroll unit tests. I've made a GitHub project to explain it, here's the link if you're interested:
https://github.com/MathiasLopes/ReqnrollTemporaryLivingDoc

[RajeshPoola]

They pulled the plug on Specflow https://support-hub.tricentis.com/open?id=post&number=NEW0001432

[tula-engineered-quality-hub]

Yes @RajeshPoola. That's why I came to change this idea of ​​what you're doing to view the post-deploy report.

[RajeshPoola]

@tulavalle They are currently not allowing new signups or logins. Mine is still working as I signed up a long time ago. However, if my browser or Azure forces me to reauthenticate, it could become a problem for me too.

[rdvanbuuren-gasunie]

Did anyone manage to run livingdoc with the test-assembly option and .NET 9?
I'm getting the following error:

Tool 'specflow.plus.livingdoc.cli' (version '3.9.57') was successfully installed.
Framework: .NET 8.0.12
Warning: Unable to load one or more of the requested types.
Could not load file or assembly 'System.Runtime, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

I noticed that Framework: .NET 8.0.12 is being selected, however the project is .NET 9. I can reproduce this locally so I'm wondering if I can force the Framework somehow. According to the NuGet it should also support .NET 9 and in one pipeline run I've seen it work. Maybe someone knows how it picks the framework?

[gasparnagy]

I haven't tried, but you can try with setting the DOTNET_ROLL_FORWARD env variable to LatestMajor (or maybe Major) and run it again.

[ShanmukhaGajula]

Yes I do.. thanks for the input. One question how can i add base64 image to the reporter?

…

On Wed, Aug 14, 2024, 11:41 AM dominik-niebuhr ***@***.***> wrote: @MathiasLopes <https://github.com/MathiasLopes> , small correction here. I gave the command as livingdoc feature-folder "mypath to features folder" --output-type HTML -t bin/Debug/net8.0/TestExecution.json instead of livingdoc feature-folder . --output-type HTML -t bin/Debug/net8.0/TestExecution.json but after removing the path it worked. Thanks again for the workaround. Appreciate your inputs. Have you got the test results to show up? — Reply to this email directly, view it on GitHub <#196 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFBWBLKU5SQDZMCEN5KKEYLZRLYHTAVCNFSM6AAAAABKFOZSCGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMZTGM2DQNI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Pressacco]

Status Update

As I understand it, the current game-plan is:

• to be compatible with the Cucumber eco-system by supporting Cucumber messages
  • in order to integrate with tools like Pickles, Report Portal, etc.
• which is being tracked by Feature/Cucumber Messages #233

yes?

[clrudolphi]

Yes. Support for Cucumber Messages is the first step.

[gasparnagy]

FYI: https://reqnroll.net/news/2025/08/reqnroll-v3-0-released-with-reporting/

[CRohm42]

Thank you for your hard work! This is awesome!

[jrod567]

Way to go!! Looks great - thank you so much!

[SierraNL]

Very cool, but we're running into a CSP issue, causing empty pages. Because the generated pages use inline scripts. Is it possible to add a hash/nonce to the pages? Or should I create an issue for that?

[clrudolphi]

Very cool, but we're running into a CSP issue, causing empty pages. Because the generated pages use inline scripts. Is it possible to add a hash/nonce to the pages? Or should I create an issue for that?

Yes , please submit an issue.

[blaisep]

Gaspar, I'm sorry for the delayed response. I have very little practical experience with ndjson, and for HTML output I typically use Python (or pandoc) Python handles ndjson beautifully, but I'm not here to sell you on Python. My dear friend Lisa Crispin thinks highly of you, and I am a fan of cucumber; so I am eager to contribute in any way I can. You will be hearing more from me. Blaise LinkedIn <https://www.linkedin.com/in/blaisepabon/> | Github <https://github.com/blaisep> “If you want to go fast, go alone. If you want to go far, go together.” --African proverb

…

On Fri, Aug 22, 2025, 4:01 AM Gáspár Nagy ***@***.***> wrote: FYI: https://reqnroll.net/news/2025/08/reqnroll-v3-0-released-with-reporting/ — Reply to this email directly, view it on GitHub <#196 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB6D6HRVNSLOAQUFWZMLPD3O3E4FAVCNFSM6AAAAACERFIVPKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTIMJYGY4TSMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[AndreasLauvhjell]

Good to see version 3.0 already here with living doc. However, my .html file is empty. It gets created but it's 0 KB. Is .NET8 required or something? We run on .NET Framework 4.8

[clrudolphi]

Please submit a bug report on this.

[AndreasLauvhjell]

I fixed it by installing Microsoft.NET.Test.Sdk. Works flawlessly now.

[nithinbandaru1]

Feedback:

Report:

1. Scenario Outline - I don't see the multiple results. I see only one. Not sure if it is limitation of Cucumber messages itself or the integration.
   • Update - I see it works, just that it's not easy to realize being a new user 😊. I have to hover and see the mouse pointer changing.
2. Server, Git, Tags as UNKNOWN at top of document. I did not see them in sample at https://reports.cucumber.io/.
3. No partial search for the tags.
4. No duration's for the steps, background etc.
5. No count of pass, fail, skip at overall level and just percentage is shown.
6. Good to have pass, fail, skip percentage and count at each level.
7. Good to have dynamic hierarchy customization instead of flattening to features. With this I could list scenarios or group by folders by levels. Nothing new I see this in code coverage reports.

Usage:

1. I see we can use reqnroll.json or environment variable. I am just wondering about the possibility of having separate NuGets (One for HTML, One for cucumber messages) and on addition just auto register and produce report.
2. I see that path always works only relative to bin folder. How can I provide the path in such a way that it will be relative to execution location. Probably if we specify ./ then consider relative to current path where being executed, if /x/ or x/ consider relative to bin folder.

[clrudolphi]

Feedback:

Thanks @nithinbandaru1 for taking the time to review and provide feedback.

Server, Git, Tags as UNKNOWN at top of document. I did not see them in sample at https://reports.cucumber.io/.

These depend upon our ability to detect which build server is being used. If you are running these locally on your workstation without a build server, then these UNKNOWN values are expected. The reports.cucumber.io server is using a customized version of the renderer and they seem to have omitted the build info. We do not yet have built-in integration with the cucumber reports server.

No partial search for the tags.
No duration's for the steps, background etc.
No count of pass, fail, skip at overall level and just percentage is shown.
Good to have pass, fail, skip percentage and count at each level.
Good to have dynamic hierarchy customization instead of flattening to features. With this I could list scenarios or group by folders by levels.

These would all be behaviors of the Cucumber HtmlFormatter and React rendering components (which we consume from Cucumber). We'll take these feedback items and pass them along to the Cucumber team for consideration of enhancements.

I see we can use reqnroll.json or environment variable. I am just wondering about the possibility of having separate NuGets

We considered this early in the development and decided instead to simply build these two basic formatters into the core project. Other formatters can be custom built and added as plugins, but the configuration of them would be similar (reqnroll.json or environment variables).

I see that path always works only relative to bin folder.

Can you provide an example? The default directory location for the output is the currently executing assembly directory (eg, <Project>/bin/<Configuration>/<framework>, such as myproject/bin/Debug/net9.0)

[hung-doan]

Thank you for the great work on v3.0 and the new reporting mechanism!
I’d like to ask if there are any plans to support a "step definition report".

In SpecFlow's LivingDoc, it’s possible to generate a StepDefinitionReport.html, which is extremely helpful for providing the team with a clear overview of all available steps.

The current report mainly focuses on test execution results, which is valuable, but not as useful for this particular use case.

[gasparnagy]

This is not in our short term plans, but if you choose the message formatter, the generated cucumber messages ndjson file contains all the info for that, so you (or someone) can easily make a tool that extracts that.

[mohamedilyeshel]

Hello, Thank you for releasing the report feature to reqnroll.
I have a question, is it possible to integrate the generated report (html version or .ndjson) into azure devops ?
because the extension Specflow + livingDoc is no longer available.
Is there a way to do this?

[gasparnagy]

I succeeded to do that by generating the HTML report and using this plugin to show it: https://marketplace.visualstudio.com/items?itemName=awardedsolutions.azure-pipelines-html-report-awardedsolutions

I will try to find time to make some description here about the steps, but it was not hard to get that working.

[ghost]

I would like to do something similar. Can you please provide some details on how to get this working? Thanks

[kwlin]

tl;dr;
Does anybody have experience with another tools (licensed is oke) for generating Living docs files based on cucumber messages for large code bases & applications? The tool/site should really focus for functional users/readers.

Hi,

We are currently researching a way to migrate our feature & test result files from the old Pickles tool (https://github.com/picklesdoc/pickles) to cucumber messages, cucumber-htmlformatter & cucumber-reactcomponents.

Our current workflow with Pickles is

• aggregate all the feature & test result files from various testruns and run the Pickles tool
• We push the html output to a our living doc site (static content site)

The html output is very user/reader friendly because it has grouping on features (with treeview & folders) & has a search functionality on all the content.

When using cucumber messages, cucumber-htmlformatter & cucumber-reactcomponents:
Pros:

• Default support "& standardized way of working(?)"
• Search works like a charm (on all content)
• It's a great start for small applications or low feature sets.
  Cons:
• The reqnroll - htmlformatter only generates per cucumber message a html page; There is a workaround: Writing our own code to merge all the cucumber messages & generate the html page.
• The html page from cucumber-reactcomponents is a simple list of all the scenario's. So when we have a lot of features & test results, the html page will be very large and hard to find a specific group of features.

Alternative we have also research ExpressiumOSS:
Pros:

• Features are nicely grouped
• Possible to merge various feature & testresult files
  Cons
• Search only works on feature title & tags
• 3rd party tool/library with still 1 active developer

Does anybody have experience with other tooling for living documentations with focus on readability & user friendly?

[egorshulga]

At our project we adopted Allure reports

[hung-doan]

using Allure reports as well. So far, they’ve been ok, and the Reqnroll integration has been well documented

[MarcWils]

We use feature descriptions to describe additional context and business rules. Am I missing something? I can't find this in the Allure reports.