XSD for Version 3 output format #13

CharliePoole · 2016-08-15T21:25:59Z

@oznetmaster commented on Sun Oct 19 2014

Is there an XSD for the 3.0 output format? If not, should there be one?

@rprouse commented on Sun Oct 19 2014

I only see an XSD for v2 in the source. There probably should be one for 3.0.

@CharliePoole, is this an oversight or were you waiting until the format stabilized?

@CharliePoole commented on Sun Oct 19 2014

Two reasons we don't have it:

I'm waiting for the format to settle.
I really dislike XML, and so tend to not pay enough attention to it.

Only #1 is a valid reason, of course. :-)

I imagined we would do this as a part of resolving #254, but there is no reason we can't develop the XSD earlier and modify it if we make changes. Some people might prefer to review the format through an XSD anyway.

I'm marking this for the first beta milestone, meaning that it needs to be done at least by that point.

@oznetmaster commented on Tue Oct 21 2014

I have a tool which generates an xsd from any xml file.

Given a large enough and comprehensive enough xml file, the generated xsd is pretty good.

If someone can provide a very large version 3 xml output file, I will run it through the tool, and we can see what it produces.

@CharliePoole commented on Tue Oct 21 2014

Hi Neil,

Just run the NUnit tests. Running
nunit-console.exe nunit.engine.tests.dll nunit-console.tests.dll
net-4.5/nunit.framework.tests.dll
will give you a pretty large output file.

Charlie

On Tue, Oct 21, 2014 at 3:50 AM, oznetmaster [email protected]
wrote:

I have a tool which generates an xsd from any xml file.

Given a large enough and comprehensive enough xml file, the generated xsd
is pretty good.

If someone can provide a very large version 3 xml output file, I will run
it through the tool, and we can see what it produces.

—
Reply to this email directly or view it on GitHub
nunit/nunit#281 (comment).

@oznetmaster commented on Tue Oct 21 2014

The problem is that those tests all run correctly. I will need a large output file that contains as many different elements as possible. Errors, failures, failed assumptions, different site failures.

Perhaps a few contributors might have 3.0 output files with these in them?

@ilya-murzinov commented on Tue Oct 21 2014

@oznetmaster I'm gonna need similar XML too for #294. Please share it with me once you have it. Or I will share it if I get it earlier :)
Thanks!

@oznetmaster commented on Tue Oct 21 2014

Will do :)

From: Ilya Murzinov [mailto:[email protected]]
Sent: Wednesday, October 22, 2014 5:48 AM
To: nunit/nunit
Cc: oznetmaster
Subject: Re: [nunit] XSD for Version 3 output format (#281)

@oznetmaster https://github.com/oznetmaster I'm gonna need similar XML too for #294 nunit/nunit#294 . Please share it with me once you have it. Or I will share it if I get it earlier :)
Thanks!

—
Reply to this email directly or view it on GitHub nunit/nunit#281 (comment) . https://github.com/notifications/beacon/5798625__eyJzY29wZSI6Ik5ld3NpZXM6QmVhY29uIiwiZXhwaXJlcyI6MTcyOTU0NzMwOCwiZGF0YSI6eyJpZCI6NDYyNDIxNDN9fQ==--24b50c5574eb9ce7e76a64a6147b284c3fa0f91e.gif

@CharliePoole commented on Fri Oct 24 2014

I'm wondering about the emphasis here on generating the XSD from an actual file. It's as if the XML output were some strange language, for which we don't have a key. Yet the file as it exists was designed. It contains what it contains as a result of some decisions we (principally I) made. It seems as if there should be a better way to document those decisions, then use that to create an XSD.

The decisions are embodied in the code, of course, in certain specific places I could point you to. Alternatively, doing some reading, discovered the compact form of RELAX NG. What if I simply provided that?

None of this resolves the further question I put in a separate thread: How do we get people to review the format? Can we generate web pages that document the meaning of each element in plain English?

@rprouse commented on Fri Oct 24 2014

I did a quick Google search and found XSDDoc, a freeware tool that generates HTML documents describing the XSD.

This results look like this, http://www.filigris.com/docflex-xml/xsddoc/examples/html/XMLSchema/PlainDoc.html

There are probably others, that was just the first freeware program I found.

@OsirisTerje commented on Fri Oct 24 2014

Sorry to pop in a bit late here.

I've been doing quite of lot of work with XML/XSD for miscellaneous projects. My experience is to keep the XML as a transport format (equal to the NUnit 3 idea, afaic see), and the XSD as a representation of the design - as Charlie points out. What I normally do is to represent the design first as a C# class structure, then generate the XSD from that one, which can be turned into documentation as Rob says with XSDDoc or similars (I have normally used XmlSpy (Oxygen is a similar tool). The XSD "may" have to be tweaked somewhat even from the classes, but normally gives a much better result than generating from the XML. The XSDs you get from the XML mostly are pretty wild, because there are more degrees of freedom in that direction. Multiple different XSDs can represent the same XMLs. There are different xml to xsd tools around, and they all geenrate different xsd's. The Microsoft XSD tool is one that generates some pretty heavy XSDs, compared to some off the web, which in fact generates better XSDs.

When we are satisfied that our XSD schema is the correct representation of the design, I think a tool like Rob pointed to is good enough to generate some documentation. The details are often not that interesting for review purposes (it becomes quite a lot, I doubt people want to go through all of that.), but the drawings are often easily understandable, and may create some feedback.

I do believe more people are used to XSDs than Relax NG, so - although very simple - I fear the relax ng format will simply confuse more than clarify.

Having plain English documentation together with an XSD generated drawing, and a sample XML file, I think would be the best choice.

My 5 cents .....

@CharliePoole commented on Fri Oct 24 2014

Thanks Rob, I'll take a look at this.

Charlie

On Fri, Oct 24, 2014 at 10:31 AM, Rob Prouse [email protected]
wrote:

I did a quick Google search and found XSDDoc
http://www.filigris.com/docflex-xml/xsddoc/, a freeware tool that
generates HTML documents describing the XSD.

This results look like this,
http://www.filigris.com/docflex-xml/xsddoc/examples/html/XMLSchema/PlainDoc.html

There are probably others, that was just the first freeware program I
found.

—
Reply to this email directly or view it on GitHub
nunit/nunit#281 (comment).

@CharliePoole commented on Fri Oct 24 2014

Hi Terje,

Well, we do have a class structure in the framework of course. Take a look
at TestResult for example. The XML representation was designed to match
that class and is created programmatically. Could we use that as a basis
even though the implementation does not use standard Xml auto-serialization?

We also need to have a class representations for the TestResult in the
runners (or maybe the engine, for use by the runners). Should we start by
creating that class, making it similar to the Framework TestResult?

Charlie

On Fri, Oct 24, 2014 at 11:13 AM, Terje Sandstrom [email protected]
wrote:

Sorry to pop in a bit late here.

I've been doing quite of lot of work with XML/XSD for miscellaneous
projects. My experience is to keep the XML as a transport format (equal to
the NUnit 3 idea, afaic see), and the XSD as a representation of the design

as Charlie points out. What I normally do is to represent the design
first as a C# class structure, then generate the XSD from that one, which
can be turned into documentation as Rob says with XSDDoc or similars (I
have normally used XmlSpy (Oxygen is a similar tool). The XSD "may" have to
be tweaked somewhat even from the classes, but normally gives a much better
result than generating from the XML. The XSDs you get from the XML mostly
are pretty wild, because there are more degrees of freedom in that
direction. Multiple different XSDs can represent the same XMLs. There are
different xml to xsd tools around, and they all geenrate different xsd's.
The Microsoft XSD tool is one that generates some pretty heavy XSDs,
compared to some off the web, whic h in fac t generates better XSDs.

When we are satisfied that our XSD schema is the correct representation of
the design, I think a tool like Rob pointed to is good enough to generate
some documentation. The details are often not that interesting for review
purposes (it becomes quite a lot, I doubt people want to go through all of
that.), but the drawings are often easily understandable, and may create
some feedback.

I do believe more people are used to XSDs than Relax NG, so - although
very simple - I fear the relax ng format will simply confuse more than
clarify.

Having plain English documentation together with an XSD generated drawing,
and a sample XML file, I think would be the best choice.

My 5 cents .....

—
Reply to this email directly or view it on GitHub
nunit/nunit#281 (comment).

@OsirisTerje commented on Fri Oct 24 2014

I'll have a look. The Xsd requires the properties to be public, so if that is the case it will work. There are other tools too, but the Xsd is very easy to use. I think it is always good to have a c# class as the de-facto design representation. I don't think I like XML any more than Charlie, but have been forced to work with it :-)
For the Test Result - absolutely!

@ravichandranjv commented on Thu Oct 08 2015

Unable to attach file for some reason.

The suggestion to use a tool to generate the XSD from the XML file is not so simple because there are some restrictions that need to be taken care of in the output XML file as specifying only certain values to be present for some elements.

@CharliePoole The "Description" value is missing in the TestResult format described in the github so the below XSD is still not complete..

Plus, there are many other maintenance related points like global or local change impact of the schema in future that may impact the XML validation, which should be discussed here.

This schema I have tested for only two elements - TestRun (test-run) and Environment but still would like someone to test if the schema validates correctly.

The schema does not incorporate all the attributes of other elements described in the format at github. If this schema looks like moving in the right direction, I will complete the declarations for the remaining elements.

Assumption: the root element is named TestRun (Will this cause rewrite to code? Because usage of '-' in schema declaration of element names is not a good practice) without the '-' as in the below extracted XML of the actual TestResults.xml.

The XML


<?xml version="1.0" encoding="utf-8" standalone="no"?>
<TestRun xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.nunit.org testresults.xsd">
id="2" testcasecount="2" total="2" passed="2" failed="0" inconclusive="0" skipped="0" asserts="0" start-time="2015-10-07 13:30:06Z" end-time="2015-10-07 13:30:07Z" >
<environment nunit-version="3.0.5715.30860" clr-version="4.0.30319.42000" os-version="Microsoft Windows NT 6.1.7601 Service Pack 1" platform="Win32NT" cwd="E:\nunit3_tests" machine-name="JV-PC" user="Jv" user-domain="Jv-PC" culture="en-US" uiculture="en-US" os-architecture="x86" />
</TestRun>

The XSD

<?xml version="1.0"?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
 xmlns:nunit="http://www.nunit.org"
 elementFormDefault="unqualified"
 attributeFormDefault="unqualified">
<xsd:element name="TestRun" />
 <xsd:complexType name="etr">
    <xsd:sequence>
     <xsd:element ref="Environment" minOccurs="1" maxOccurs="unbounded" />
     <xsd:element ref="TestSuite" minOccurs="1" maxOccurs="unbounded" />
     <xsd:element ref="CommandLine" minOccurs="1" maxOccurs="unbounded" />
     <xsd:element ref="Settings" minOccurs="1" maxOccurs="unbounded" />
     </xsd:sequence>
 </xsd:complexType>
       <xsd:complexType name="tr">
        <xsd:attribute name="id" type="xsd:int" use="required"/>
        <xsd:attribute name="testcasecount" type="xsd:int" use="optional"/>
        <xsd:attribute name="result" type="xsd:string" use="required"/>
        <xsd:attribute name="total" type="xsd:int" use="optional"/>
        <xsd:attribute name="passed" type="xsd:int" use="required"/>
        <xsd:attribute name="failed" type="xsd:int" use="required"/>
        <xsd:attribute name="inconclusive" type="xsd:int" use="required"/>
        <xsd:attribute name="skipped" type="xsd:int" use="required"/>
        <xsd:attribute name="asserts" type="xsd:int" use="required"/>
        <xsd:attribute name="start-time" type="xsd:date" use="required"/>
        <xsd:attribute name="end-time" type="xsd:date" use="required"/>
        <xsd:attribute name="duration" type="xsd:decimal" use="required"/>
    </xsd:complexType> 
            <xsd:element name="Environment"> 
        <xsd:complexType>
            <xsd:attribute name="nunit-version" type="xsd:string" use="required"/>
            <xsd:attribute name="clr-version" type="xsd:string" use="required"/>
            <xsd:attribute name="os-version" type="xsd:string" use="required"/>
           <xsd:attribute name="platform" type="xsd:string" use="optional"/>
           <xsd:attribute name="cwd" type="xsd:string" use="required"/>
           <xsd:attribute name="machine-name" type="xsd:string" use="optional"/>
           <xsd:attribute name="user" type="xsd:string" use="required"/>
           <xsd:attribute name="user-domain" type="xsd:string" use="required"/>
           <xsd:attribute name="culture" type="xsd:string" use="required"/>
           <xsd:attribute name="uiculture" type="xsd:string" use="required"/>
           <xsd:attribute name="os-architecture" type="xsd:string" use="required"/>
        </xsd:complexType> 
            </xsd:element>
    <xsd:element name="TestSuite" type="xsd:string"/>    
    <xsd:element name="CommandLine" type="xsd:string"/>    
    <xsd:element name="Settings" type="xsd:string"/>    
</xsd:schema>

@CharliePoole commented on Thu Oct 08 2015

As you see from the milestone assigned to it, we are not working on this right now, so I'll only give a few brief comments. I don't really want to try to create an XSD until after we settle on the actual file content. I think a "friendlier" format will be better for people to comment on.

The file content is being worked as issue #254 and the current documentation of the file comment is at https://github.com/nunit/nunit/wiki/Test-Result-XML-Format We are waiting for comments!!!

Since you have gone to this trouble, however, here are a few comments to your comments. :-)

Generating an XSD schema from the file: We are not doing that.
Description is not part of the file because it's a property rather than an attribute.
Use of hyphen in element names: The NUnit 2.x result format has used hyphens since the year 2000. It has never appeared to cause any problems. Is there a reason you say it is not a good practice? Changing all the elements would be a significant programming change that we would be unlikely to do at this late date.

As I explained above, issue #254 is a pre-requisite to this one. We do not want to have to work on the XSD more than once and we don't actually need it to release NUnit 3.0. That's why it's in the milestone of 3.0-Extras, that is, individual bits to be worked on after release.

@ravichandranjv commented on Thu Oct 08 2015

Hi Charlie, Thanks for spending some time on XML. :)

Use of hyphens as not a good practice is simply in the context of the XSD element naming conventions, which follows the upper case camel convention.

You have mentioned above, "some people like to validate against a XSD..", which is the reason why I posted the XSD.

By having the XSD, people can use tools to generate a schema diagram from the XSD file, which can also serve as documentation, along with comments next to the schema boxes in the diagram plus, validate the XML file, which may be important for people who may want to further work through the XML using Xpath or XSL.

Having a beta XSD, now, however broken or incomplete, will serve its purpose in gaining some insight into the type of XML that should be generated from code.

For instance, in issue #254, the issue you have highlighted under <environment> gets resolved if we approach the XML creation through XSD standards, which says that if you wish to have multiple values in a data set for the same attribute/property of an object then use elements, so the two different or more runtime versions can be accommodated under <environment> itself rather than at the assembly level, which should gain favor because it is an OO approach, as per your stated preference there -.to have a semi-colon separated set of values under a single field.

The reason I mention it here and not in the comments of #254 is because it is related more to XSD (as and when it may get created) because values, too, can be/get constrained in XSD and if ignored, can cause errors at the time of validation, post-XML generation causing problems for report generation of the test results.

So, even if it is not priority now, having a glimpse of the kind of XSD that may be needed later helps so as to reduce any chance of a (nunit/nunit#850), ie. a schema validation related XML error that has nothing to do with NUnit usage, causing tests to fail.

@CharliePoole commented on Thu Oct 08 2015

Thanks for your comments. We will keep them in mind when we work on the
XSD. NUnit itself does not require or make any use of an XSD, so it is an
extra that we will provide for those who need it only after the release.

We have been working on the new format for the result file for over a year
and it is already being created. The documentation I created reflects what
we are actually doing. It's important to us to have this reviewed. I
understand that you are suggesting a different process, but we have a
process already under way. We are stretched too thin to add any new tasks
right now. If you would like to help, please stick to the approach we are
taking and make comments either on issue #254 or through the developer list.

On Thu, Oct 8, 2015 at 6:32 PM, Ravichandran Jv [email protected]
wrote:

Hi Charlie, Thanks for spending some time on XML. :)

Use of hyphens as not a good practice is simply in the context of
the XSD element naming conventions, which follows the upper case camel
convention.

You have mentioned above, "some people like to validate against a XSD..",
which is the reason why I posted the XSD.

By having the XSD, people can use tools to generate a schema diagram from
the XSD file, which can also serve as documentation, along with comments
next to the schema boxes in the diagram plus, validate the XML file, which
may be important for people who may want to further work through the XML
using Xpath or XSL.

Having a beta XSD, now, however broken or incomplete, will serve its
purpose in gaining some insight into the type of XML that should be
generated from code.

For instance, in issue #254 nunit/nunit#254,
the issue you have highlighted under gets resolved if we
approach the XML creation through XSD standards, which says that if you
wish to have multiple values in a data set for the same attribute/property
of an object then use elements, so the two different or more runtime
versions can be accommodated under itself rather than at
the assembly level, which should gain favor because it is an OO approach,
as per your stated preference there -.to have a semi-colon separated set of
values under a single field.

The reason I mention it here and not in the comments of #254
nunit/nunit#254 is because it is related more
to XSD (as and when it may get created) because values, too, can be/get
constrained in XSD and if ignored, can cause errors at the time of
validation, post-XML generation causing problems for report generation of
the test results.

So, even if it is not priority now, having a glimpse of the kind of XSD
that may be needed later helps so as to reduce any chance of a nasty code
change just so that a schema validation error that has nothing to with the
NUnit user, causes the tests to fail as in issue 850
nunit/nunit#850.

—
Reply to this email directly or view it on GitHub
nunit/nunit#281 (comment).

@ravichandranjv commented on Fri Oct 09 2015

Hi Charlie,

I am not suggesting any process change nor proposing any new tasks just highlighted some points related to XSD.

I am happy to go by your judgement/direction.

@stashell commented on Wed Jul 20 2016

Hello everyone. We are writing tool which will parse also NUnit3 test results and thus I have a few questions regarding the XSD scheme and stability of the test-results XML format design.

Any news about the "existence" of XSD for NUnit3 test-results ?
How stable is the XML test-results design currently ?
Do you expect many changes in the further 3.x versions ?

Thank you.

Patrik

@rprouse commented on Wed Jul 20 2016

@stashell there is currently no XSD for the test-results format, but the design is fairly stable. The XML format is part of the API between the NUnit Engine and runners including 3rd Party runners, so we are careful to not change it unless there are bugs. Any changes tend to be minor and additive. For example, we might add a new attribute or occasionally a new child element, but that is rare.

To get a good example of all of the combinations, I would suggest creating a mock test suite that has a variety of passing, failing, ignored, explicit and errors. Errors are tests that throw non-NUnit assert exceptions. I would also use properties and categories on tests and include test output using the TestContext.Write methods.

@CharliePoole commented on Wed Jul 20 2016

@stashell In addition, read the docs! https://github.com/nunit/docs/wiki/Test-Result-XML-Format

@rprouse Maybe we should try to get this done. I don't think either of us is very enthusiatic about the job. :-) Any XML lovers out there to volunteer?

The text was updated successfully, but these errors were encountered:

jnm2 · 2018-07-13T03:15:08Z

Framework PR is up: nunit/nunit#2929

CharliePoole · 2022-01-03T18:44:45Z

This issue has been inactive for a long time. Closing it now. Comments may still be made and we will respond to them.

CharliePoole mentioned this issue Aug 15, 2016

XSD for Version 3 output format nunit/nunit#281

Closed

CharliePoole added Feature pri:normal labels Aug 19, 2016

mikkelbu mentioned this issue May 31, 2018

wiki: schema missing for NUnit 3.0 nunit/docs#270

Open

ChrisMaddock mentioned this issue Jul 11, 2018

wiki: invalid sample for NUnit 3.0 nunit/docs#269

Open

jnm2 mentioned this issue Jul 13, 2018

Added NUnit XML schemas nunit/nunit#2929

Merged

CharliePoole closed this as completed Jan 3, 2022

CharliePoole added this to the Closed Without Action milestone Jan 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

XSD for Version 3 output format #13

XSD for Version 3 output format #13

CharliePoole commented Aug 15, 2016

jnm2 commented Jul 13, 2018

CharliePoole commented Jan 3, 2022