IReqnrollOutputHelper.AddAttachment() -> Formatter Messages: embedding attachment file content

[clrudolphi]

Reqnroll outputs Formatter reports using the Cucumber Messages format. Messages supports adding Attachments to the report as embedded content. However, in an upcoming release of Cucumber Messages, this will change. It will add the ability to indicate that an attachment was added by the test runner and identify it by its URI (aka, file name) only; the content of the attachment will not be included.

In the current support for this in Reqnroll Formatters, when a call to IReqnrollOutputHelper.AddAttachment(string filename) is made, the Formatters subsystem reads the file in order to embed that as Base64 encoded content directly in the report (as an Attachment message).

I think we have several options to consider and would like feedback on direction:

1. Do nothing and keep current behavior.
2. Replace current behavior by generating the new message type that is being added to Cucumber Messages - ExternalAttachment which will have a property that includes the filename as passed in the call to IReqnrollOutputHelper.AddAttachment(string filename).
3. Add an option within reqnroll.json configuration to allow the user to specify which of the above behaviors to use when handling attachments.

[Code-Grump]

I think this is more a question of what compatibility is expected in the Cucumber Messages ecosystem. It's a format that's consumed by tooling, so the value is only provided through those tools.

If the consuming tooling moves in lockstep with the output format, then making the change to the new format seems the natural answer.

If tooling updates are rolled out after message-producers are upgraded, then it feels like something needs to maintain a level of backwards compatibility for the interim period, which might be a config switch.

Not knowing enough about this ecosystem, I can't really provide an answer, but you may be able to suggest the kinds of compatibility model we should follow.

[gasparnagy]

First however: if the concept of packaging two attachment objects inside a single Envelope is objectionable, how about we publish two Envelopes (one for each type of Attachment)? We do know what type of Attachment message each formatter would like to see, so we would publish only the types we know to be requested. A further possibility is that the Broker could filter that flow of Attachments messages and send to each formatter only the type of Attachments it has requested.

Let's pause this question until we learn more about the usage patterns. Because for example if we decide to have a separate IReqnrollOutputHelper.RegisterLinkedAttachment API call, we don't need to change anything in the infrastructure: RegisterLinkedAttachment will generate external attachment envelope, AddAttachment generates embedded one.

[gasparnagy]

I'm very interested to hear more real-world examples

@MrSoerenAndersen Do you have maybe additional ones?

[gasparnagy]

(Added a thread to the Ideas section to get more input from end-users: https://github.com/orgs/reqnroll/discussions/1018. I have asked them to post here, however.)

[MrSoerenAndersen]

@gasparnagy The idea of having two separate methods - E.g. AddAttachment and AddLinkedAttachment - seems like a simple and sufficient solution. Additional I have been requested to support IReqnrollOutputHelper.WriteLine() as a message in Expressium LivingDoc, but not sure if this fits into the Cucumber Messages concept...?

[clrudolphi]

but not sure if this fits into the Cucumber Messages concept...?

It does. A call to:

    _reqnrollOutputHelper.WriteLine("hello");

results in an Attachment message like this:

{"attachment":{"body":"hello","contentEncoding":"IDENTITY","mediaType":"text/x.cucumber.log+plain","testCaseStartedId":"Uv0yD5NtQzo9rxHctB4Dmg","testStepId":"7c9eP_E_kc8Lh17WV-t4Fg","testRunStartedId":"VwMsUR9gTYah2lIp4yWcGA","testRunHookStartedId":"","timestamp":{"seconds":1769103680,"nanos":354108300}}}

The 'body' is not BASE64 encoded, the 'contentEncoding' property has a value of "IDENTITY" and the 'mediaType' property has a value of "text/x.cucumber.log+plain"

The string argument to WriteLine is also forwarded to the underlying test framework using its respective 'log' or WriteLine() call.

[MrSoerenAndersen]

@gasparnagy I am mostly engaged in test automation projects for large UI-based enterprise applications...

Why would you want linked files instead of embedded ones?

• To easily retrieve attachments later for bug reporting
• To avoid very large embedded files (e.g., 5GB videos) inside a single HTML file
• To be able to link to already existing documents and URLs

In which cases (for which files) would you want to use linked files?

• Screenshots and video recordings
• Test automation execution and application logs
• Direct links to Test Management stories and issues

What file/folder structure would you like to use for linked files?

• For smaller projects, a single subfolder would be sufficient
• For larger projects, a deeper subfolder structure would provide better overview

Do you expect the tool (ReqnRoll) to copy files into that structure, or would you manage it yourself?

• ReqnRoll should not be responsible for copying attachments into a file structure
• Developers should be able to link to any file or URL using both absolute and relative paths

[clrudolphi]

Which test execution framework are you using?

How do you know where the attachments will be permanently located? Is that something that you configure the test framework? Or do you later retrieve that location path from the test framework?

[MrSoerenAndersen]

I am using NUnit...

Attachments may come from multiple sources during test execution. Some internally from e.g. Selenium and log4net like screenshots and logs and others from application servers via a custom API e.g. application error logs. All attachments will be fetched and saved to a local subfolder relative to the generated test report and added as an external link via the AddAttachmentAsLink method. The test report and attachments will finally be distributed in the CI pipeline to a test report server to easy the access for the business...

Does it answer your questions...?

[clrudolphi]

Is Expressium relying on the underlying test execution framework for handling these attachments or does it directly read, copy, move them where it needs them?

[MrSoerenAndersen]

Non of my attachments are related to NUnit and it directly read, copy, move them where it needs them!

[MrSoerenAndersen]

Example saving a Selenium screenshot...

            if (browserManager.IsInitialized())
            {
                if (configuration.Screenshots)
                {
                    var filePath = Path.Combine(configuration.ScreenshotsPath, GetTestName() + ".png");
                    browserManager.SaveScreenshot(filePath);
                    logger.Info($"Screenshot has been saved: {Path.GetFileName(filePath)}");
                }
            }

[MrSoerenAndersen]

Have noticed that ReqnRoll has been update using Cucumber Messages V32...
<PackageReference Include="Cucumber.Messages" Version="32.0.1" />

Is the handling of External Attachments included - If so what do I need update my Expressium LivingDoc PlugIn to enable this...?
https://github.com/ExpressiumOSS/Expressium.LivingDoc/tree/main/Expressium.LivingDoc.ReqnrollPlugin

[clrudolphi]

No.

Reqnroll is not yet leveraging the new message type.

[clrudolphi]

I just double-checked. I suspect you are looking at a preview build.
The official release (v3.3.3) depends upon Messages v30.1.0

[MrSoerenAndersen]

Sorry my bad - My core project is using Cucumber.Messages directly and not Cucumber.HtmlFormatter - Tests are running green anyway ;-)