Semantic markup and ICS table generation

[PaulMartinsen]

📑 Description

This PR provides document processing support to:

• extend semantic markup of requirements for generation of rich JSON extracts for the SDPi requirements core model,
• prototype semantic markup of use-cases to generate JSON extracts,
• insertion of automatically generated implementation conformance statement changes,
• copy links to SDPi requirements to the system clipboard clipboard from the published sdpi-profile web-document.
  

This slide show PDF provides a high-level overview.

This PR does not include the changes to the AsciiDoc source required to use the new processing features. Instead it is nearly 100% backwards compatible producing html output functionally identical as before.

Additional markup is needed in the source document, once this PR is done, to take advantage of these new features. I just don't want to make changes to the document source until everyone's had a chance to provide feedback at this stage. The goal of this PR is to get the processing and markup roughly right then I can start incorporating new features into the source files and get further feedback on how it works in practice.

Refer to the cookbook for documentation of the new markup and the test cases for examples.

Most of the document processing has been shifted from block processors to a tree processor. This mainly enables cross-referencing the bibliograph from sdpi_requirement_ref_standard requirements, and simplifies tagging use-case requirements with the related use-case. It may reduce the amount of work needed to extend the processor further (extracting transactions, for example).

Testing:

• unit-tests were added to the processing tool to check output generation for each requirement and use-case.
• html output documents generated from the master branch and this PR were compared to check the output was (materially) identical. See backwards compatiblity for differences.

☑ Mandatory Tasks

The following aspects have been respected by the pull request assignee and at least one reviewer:

• Changelog update (necessity checked and entry added or not added respectively)
  • Pull Request Assignee
  • Reviewer

[d-gregorczyk]

I must say - this is too many features in one branch.

• It contains ICS functionality and requirements interop in one PR, plus there are side features such as the "clipboard copy"
• I also saw macros dispatching to different output formats, which I do not understand why we need it
• Does it resonate with the requirements interoperability approach and has this approach been validated by anyone?

My gut feeling is that we are drowning in complexity here and would have needed to discuss the interop approach and clean up my tainted code in advance.

[d-gregorczyk]

Is this graphic freely available or drawn by yourself? We otherwise need to reference sources or include licenses in SDPi. Is this graphic really needed?

[PaulMartinsen]

It was part of the copy to clipboard fragment, which has a Mozzilla public license and the source is referenced in the fragment

[d-gregorczyk]

Please note that the image itself is listed under MIT license, and the license information gets lost once we export to PDF. That's just my concern.

[PaulMartinsen]

I must say - this is too many features in one branch.

• It contains ICS functionality and requirements interop in one PR, plus there are side features such as the "clipboard copy"
• I also saw macros dispatching to different output formats, which I do not understand why we need it
• Does it resonate with the requirements interoperability approach and has this approach been validated by anyone?

My gut feeling is that we are drowning in complexity here and would have needed to discuss the interop approach and clean up my tainted code in advance.

Yes, there is a lot to unpack here.

• By 'interop', i think you mean the JSON export for external tooling... otherwise i got the wrong idea
• I see ICS functionality as the short-term payoff for the SDPi document; exporting to JSON is useful for external tooling but as far as I know that tooling doesn't exist and isn't being worked on yet. So it may be useful to consider how interoperability support can benefit the document itself. ICS tables are a good example IMHO & one of the use cases @ToddCooper highlighted at the last SDPi workshop. Anyway, they are trivial after gathering all the requirements.
• It is intended to resonate with the requirements interoperability approach @ToddCooper shared at the last SDPi workshop. Or at least as far as I understood it. There's been a few practical difficulties with further discourse, to date. Github provides a way to do this collaboratively and asynchronously at least.
• I stuck with JSON as the interop medium, but refined the format to separate content after I found original format with normative statements, notes, examples etc all smashed together didn't really work for me. Is anyone using the JSON extracts currently? Could we get input from them?
• I don't think there was any tainted code to cleanup; without the prior version I wouldn't have had anything to build off and the barrier to learning Kotlin would have been too great.
• I'm not sure what you mean by macro dispatch? Guessing:
  ** in ascii doc source I exclude some javascript content when generating PDFs. Otherwise it just shows up as text in the PDF
  ** there are macro extension in the processor for generating reference and use-case cross references. This support using oids for permalinks if we want to do that in the future.
• Clipboard copy was to make this worthwhile for me; I find the SDPi document really hard to work as a user. For example, I regularly need to create references to requirements which is very difficult. I plan to make a suggestion around the "table of contents" if I get time to figure out how that works too.

As I understand it, interop is important but not urgent; we have the time to learn to swim here.

To cut through the complexity, a key outcome here tree-processor extensions lets us extract the information needed using roles. It isn't necessary to create block processors for interop features. And, this lets the tree processor work with the entire document; we aren't limited to source order.

[PaulMartinsen]

SDPi requirements export.pdf provides a higher-level overview .

[d-gregorczyk]

I must say - this is too many features in one branch.

• It contains ICS functionality and requirements interop in one PR, plus there are side features such as the "clipboard copy"
• I also saw macros dispatching to different output formats, which I do not understand why we need it
• Does it resonate with the requirements interoperability approach and has this approach been validated by anyone?

My gut feeling is that we are drowning in complexity here and would have needed to discuss the interop approach and clean up my tainted code in advance.

Yes, there is a lot to unpack here.

• By 'interop', i think you mean the JSON export for external tooling... otherwise i got the wrong idea
• I see ICS functionality as the short-term payoff for the SDPi document; exporting to JSON is useful for external tooling but as far as I know that tooling doesn't exist and isn't being worked on yet. So it may be useful to consider how interoperability support can benefit the document itself. ICS tables are a good example IMHO & one of the use cases @ToddCooper highlighted at the last SDPi workshop. Anyway, they are trivial after gathering all the requirements.
• It is intended to resonate with the requirements interoperability approach @ToddCooper shared at the last SDPi workshop. Or at least as far as I understood it. There's been a few practical difficulties with further discourse, to date. Github provides a way to do this collaboratively and asynchronously at least.
• I stuck with JSON as the interop medium, but refined the format to separate content after I found original format with normative statements, notes, examples etc all smashed together didn't really work for me. Is anyone using the JSON extracts currently? Could we get input from them?
• I don't think there was any tainted code to cleanup; without the prior version I wouldn't have had anything to build off and the barrier to learning Kotlin would have been too great.
• I'm not sure what you mean by macro dispatch? Guessing:
  ** in ascii doc source I exclude some javascript content when generating PDFs. Otherwise it just shows up as text in the PDF
  ** there are macro extension in the processor for generating reference and use-case cross references. This support using oids for permalinks if we want to do that in the future.
• Clipboard copy was to make this worthwhile for me; I find the SDPi document really hard to work as a user. For example, I regularly need to create references to requirements which is very difficult. I plan to make a suggestion around the "table of contents" if I get time to figure out how that works too.

As I understand it, interop is important but not urgent; we have the time to learn to swim here.

To cut through the complexity, a key outcome here tree-processor extensions lets us extract the information needed using roles. It isn't necessary to create block processors for interop features. And, this lets the tree processor work with the entire document; we aren't limited to source order.

Ok, I would like to see this approach being double-checked by Todd so he can drop his remarks from a high level design perspective (so he should not look at the Kotlin code :-) ).

From what I understand, RI and ICS tables do not have anything in common. RI being a means to relate requirements with other - partly external - requirements, and ICS tables being the means to provide conformity statements to sets of requirements.

Anyway, I still think the pull request is way too complex and should be decomposed and discussed. However, if reviewed and understood thoroughly by our stakeholders, I do not have any objection to merge it eventually.

And please do not get me wrong: I am totally advocating the idea of tweaking the SDPi input, I am just concerned that this might be overwhelming to the authors of the document (including me ;-) ).

[PaulMartinsen]

@ToddCooper , when we discussed gathering requirements for the profiles (such as SDPi-P), I had the idea that there would be a bunch of requirements defined in §1:2.3.10, for example. And we wanted to make a list of those. Looking closer, I see that section doesn't actually define requirements. Instead, it references transactions, use-cases and content modules.

So my new idea, is we want to gather all the requirements in referenced use-cases, transactions and content-modules to make a list of requirements for each profile. Is that correct?

Assuming it is correct, in which direction do you imagine these relationships are best defined in the AsciiDoc source? "Best" really means, what's most logical for how everyone thinks about the document or which is most natural easiest for editors? In other words, do you prefer:

1. when editing a use-case/transaction/content module we want to assign it to a profile, or
2. we want to select relevant use-cases, transactions, content-modules that belong to a profile in the profile section of the source document.

To me, the second option makes more sense. That is, there's a smorgasbord of use-cases/transactions/content-modules that profiles pick and choose from.

Of course, the document processor will be able to figure out the relationship in both directions with either option.

Finally, there is a third option. The processor could discover actors and may be able to build relationships based on where actors are used. For example, SDPi-P defines the SOMDS Provider. So "membership" in SDPi-P could include any use-case/transaction/content-module that refers to this actor. Its possible this will work magically, but it may also be frustrating to figure out why relationships are being created because there isn't really much in the way of diagnostic tooling. Could be fun to try though :)