diff --git a/README.md b/README.md index 0b352b1..b5e49fe 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ a preliminary test suite. ## Instructions It is highly recommended that you use [Docker](https://www.docker.com/) to run -these tests. This test kit requires at least 10 GB of memory are available to Docker. +these tests. This test kit requires at least 10 GB of memory to be available to Docker. - Clone this repo. - Run `setup.sh` in this repo. @@ -25,6 +25,15 @@ See the [Inferno Framework Documentation](https://inferno-framework.github.io/docs/getting-started-users.html) for more information on running Inferno. +## Description Authoring Guide + +An experimental [Description Authoring +Guide](docs/description_authoring_guide.md) is available to assist developers +and LLMs in writing thorough and up-to-date descriptions for test +implementations. This guide provides detailed guidance on analyzing test code +and authoring accurate descriptions at all levels of the test hierarchy (test +kit, suite, group, and individual test levels). + ## License Licensed under the Apache License, Version 2.0 (the "License"); you may not use @@ -41,4 +50,4 @@ specific language governing permissions and limitations under the License. ## Trademark Notice HL7, FHIR and the FHIR [FLAME DESIGN] are the registered trademarks of Health -Level Seven International and their use does not constitute endorsement by HL7. +Level Seven International and their use does not constitute endorsement by HL7. \ No newline at end of file diff --git a/docs/description_authoring_guide.md b/docs/description_authoring_guide.md new file mode 100644 index 0000000..f119d27 --- /dev/null +++ b/docs/description_authoring_guide.md @@ -0,0 +1,580 @@ +# Description Authoring Guide + +Guidance in this file can be used by developers and LLMs to support authoring +descriptions for new tests, or improving descriptions of existing tests. This +guidance is based on broad Inferno conventions, as well as specific practices +used in existing tests in the IPS test kit. Please note that all updates should +be manually reviewed for accuracy, as LLMs may not always produce perfect +results. + +This guide focuses specifically on writing effective descriptions at each level +of the test hierarchy: test kit, suite, group, and individual test levels. + +## Why Descriptions Matter + +Descriptions serve as the primary interface between test implementations and +users, providing essential information about what is being tested and why. While +the technical details of test validation logic are available in the source code, +natural language descriptions that appear in the user interface are critical for +helping implementers understand test purpose, scope, and requirements without +needing to parse complex code. These descriptions enable users to quickly assess +whether tests are relevant to their implementation, understand what capabilities +are being validated, and interpret test results in the context of specification +requirements. Maintaining accurate, up-to-date descriptions that stay in sync +with the underlying test logic is essential for the usability and effectiveness +of test kits, as they directly impact how implementers interact with and benefit +from the testing framework. + +## Table of Contents + +1. [Code Analysis Before Writing](#code-analysis-before-writing) +2. [Test Kit Level Descriptions](#test-kit-level-descriptions) +3. [Suite Level Descriptions](#suite-level-descriptions) +4. [Group Level Descriptions](#group-level-descriptions) +5. [Test Level Descriptions](#test-level-descriptions) +6. [Best Practices](#best-practices) + +--- + +## Code Analysis Before Writing + +**CRITICAL: Always analyze the actual test implementation before writing or updating descriptions.** + +### Why Code Analysis is Essential + +Descriptions must accurately reflect what tests actually do, not what they might do based on titles or assumptions. Writing descriptions without examining the implementation leads to: + +- Inaccurate descriptions that mislead users +- Claims about validation that don't match the actual test logic +- Descriptions that become outdated when code changes +- Loss of user trust when descriptions don't match test behavior + +### Required Analysis Steps + +Before writing any description, you must: + +1. **Read the test implementation code** - Examine the `run do` block to understand exactly what the test does +2. **Identify all assertions and validations** - Note every `assert`, `assert_valid_resource`, `assert_response_status`, etc. +3. **Understand test inputs and dependencies** - Check what inputs are required and what previous tests are referenced +4. **Note skip conditions** - Identify when tests will skip and why +5. **Check helper methods** - If tests call helper methods, examine those implementations too +6. **Verify specification claims** - Use web_fetch or other available tools (playwright MCP tool is particularly good if available) to access the relevant implementation guide or specification to verify any claims about requirements, optional vs required elements, or specification compliance rather than making assumptions + +### Code Analysis Examples + +**Example 1: Simple Validation Test** +```ruby +run do + skip_if bundle_content.blank?, 'No IPS Bundle provided' + resource_instance = FHIR.from_contents(bundle_content) + assert_resource_type(:bundle, resource: resource_instance) + assert_valid_resource(resource: resource_instance, profile_url: 'http://hl7.org/fhir/uv/ips/StructureDefinition/Bundle-uv-ips') +end +``` + +**Analysis Results:** +- Test skips if no bundle content provided +- Parses bundle content using FHIR.from_contents +- Verifies resource is Bundle type +- Validates against IPS Bundle profile +- Does NOT validate individual resources within the bundle +- Does NOT check bundle entry order or specific content + +**Example 2: Operation Test** +```ruby +run do + fhir_operation("Patient/#{patient_id}/$summary", name: :summary_operation, operation_method: :get) + assert_response_status(200) + assert_resource_type(:bundle) + assert_valid_resource(profile_url: self.class.profile_url) +end +``` + +**Analysis Results:** +- Executes GET operation on Patient/[id]/$summary endpoint +- Requires patient_id input +- Expects 200 response status +- Validates returned resource is Bundle type +- Validates Bundle against IPS Bundle profile +- Uses GET method only (not POST) +- Stores request for use by other tests + +### What to Document Based on Code Analysis + +**For Each Test, Document:** +- Exact validation steps performed +- Required inputs and their usage +- Skip conditions and when they apply +- Specific assertions made +- Profile URLs used for validation +- HTTP methods and endpoints called +- Dependencies on other tests or requests + +**Do NOT Document:** +- Assumptions about what the test "should" do +- Validation steps not actually performed +- General implementation guidance +- Features not actually tested + +### Common Code Analysis Patterns + +**Profile Validation Pattern:** +```ruby +assert_valid_resource(resource: resource_instance, profile_url: 'profile_url') +``` +→ Documents: "Validates conformance to [profile name] profile" + +**Resource Type Check:** +```ruby +assert_resource_type(:bundle, resource: resource_instance) +``` +→ Documents: "Verifies the resource is of type Bundle" + +**Response Status Check:** +```ruby +assert_response_status(200) +``` +→ Documents: "Expects successful HTTP 200 response" + +**Skip Conditions:** +```ruby +skip_if condition, 'message' +``` +→ Documents: "Test skips if [condition explanation]" + +**Resource Presence Check:** +```ruby +resources_present = resource.entry.any? { |r| r.resource.is_a?(FHIR::ResourceType) } +assert resources_present, 'error message' +``` +→ Documents: "Verifies Bundle contains at least one [ResourceType] resource" + +### Verification Process + +After writing descriptions based on code analysis: + +1. **Cross-check each claim** - Ensure every statement in the description corresponds to actual test code +2. **Verify assertion accuracy** - Confirm that described validations match the actual assertions +3. **Check completeness** - Ensure all significant test steps are documented +4. **Validate skip conditions** - Confirm skip condition descriptions match the code +5. **Review dependencies** - Verify that described dependencies are accurate + +### Red Flags - When Descriptions May Be Inaccurate + +- Descriptions that are too generic ("validates the resource") +- Claims about validation not supported by assertions in the code +- Missing mention of skip conditions that exist in the code +- Descriptions that don't match the complexity of the test implementation +- References to validation steps not present in the run block + +### Updating Descriptions When Code Changes + +When test code is modified: + +1. **Re-analyze the updated implementation** - Don't assume the description is still accurate +2. **Update descriptions to match new behavior** - Ensure every change in logic is reflected +3. **Check for new skip conditions or assertions** - Add documentation for new validation steps +4. **Verify dependency changes** - Update references to other tests if relationships change + +This rigorous approach to code analysis ensures that descriptions serve as accurate, trustworthy documentation of what tests actually validate. + + +## Test Kit Level Descriptions + +### Location and Format +Test kit descriptions are defined in the `metadata.rb` file using a heredoc format: + +```ruby +description <<~DESCRIPTION + [Content here] +DESCRIPTION +``` + +### Structure +Test kit descriptions should follow this specific structure: + +```ruby +description <<~DESCRIPTION + [Brief overview paragraph - what the test kit does] + + + [Detailed explanation of purpose and scope] + + ## Status + [Current implementation status and coverage] + + ## Repository + [Repository information and links] + + ## Providing Feedback + [How users can provide feedback] +DESCRIPTION +``` + +### Content Guidelines + +**Overview Paragraph (before ``)** +- Brief, high-level summary of what the test kit validates +- Target audience (implementers, vendors, etc.) +- Reference to the specification or implementation guide being tested +- Keep to 2-3 sentences maximum + +**Detailed Explanation** +- Comprehensive description of the test kit's purpose +- Scope of testing (what is and isn't covered) +- Key capabilities being validated +- Any important context about the specification + +**Status Section** +- Current maturity level and what that means +- Specific features/profiles being tested +- Known limitations or gaps +- Coverage information (e.g., "Tests 15 of 20 profiles") + +**Repository Section** +- Link to source code +- Installation/usage instructions +- License information if relevant + +**Feedback Section** +- How to report issues or bugs +- Types of feedback welcomed +- Contact information or preferred channels + +### Example +```ruby +description <<~DESCRIPTION + The International Patient Summary Test Kit provides an executable set of tests for the IPS Implementation Guide v1.1.0. + + + This test kit validates server implementations of the International Patient Summary (IPS) specification. It tests both the structural requirements of IPS documents and the operational capabilities required by IPS servers. + + The tests are organized into groups that validate: + - IPS Bundle structure and composition + - Required and optional resource profiles + - Support for the $summary operation + - Document reference capabilities + + ## Status + This test kit has **Low** maturity. It currently tests core IPS Bundle validation and basic profile conformance. Advanced workflow testing and comprehensive edge case coverage are planned for future releases. + + ## Repository + This test kit is open source and available at [GitHub Repository](https://github.com/inferno-framework/ips-test-kit). + + ## Providing Feedback + We welcome feedback including bug reports, feature requests, and suggestions for improving test coverage. Please submit issues through the GitHub repository or contact the Inferno team. +DESCRIPTION +``` + +--- + +## Suite Level Descriptions + +### Location and Format +Suite descriptions are defined in the suite class using a multi-line string format: + +```ruby +description %( + [Content here] +) +``` + +### Content Guidelines + +Suite descriptions should include: + +**Purpose Statement** +- What specific aspect of the specification this suite tests +- How it fits into the broader test kit + +**Key Capabilities** +- Bulleted list of main features being tested +- High-level test scenarios covered + +**Usage Context** +- When/why someone would run this suite +- Prerequisites or setup requirements +- Available presets or configurations + +**Scope Boundaries** +- What is explicitly tested +- What is not covered (if relevant) + +### Example +```ruby +description %( + This suite tests server support for the International Patient Summary (IPS) specification. + It validates both the structural requirements of IPS documents and the operational capabilities + required by IPS-compliant servers. + + Key capabilities tested: + - IPS Bundle validation and composition structure + - Required resource profile conformance (Patient, AllergyIntolerance, etc.) + - Optional resource profile support + - $summary operation implementation + - Document reference and retrieval workflows + + This suite can be run against any FHIR R4 server claiming IPS support. Test scenarios + include both pre-loaded IPS documents and dynamic document generation via the $summary operation. + + Two preset configurations are available: one for testing with example IPS bundles, + and another for testing against reference server implementations. +) +``` + +--- + +## Group Level Descriptions + +### Location and Format +Group descriptions are defined within group blocks: + +```ruby +group do + title 'Group Title' + description %( + [Content here] + ) +end +``` + +### Content Guidelines + +Group descriptions should be concise and focused: + +**Specific Purpose** +- What particular aspect or resource this group tests +- How it relates to the broader suite + +**Testing Approach** +- Brief explanation of the testing methodology +- Key validation points + +**Dependencies** +- Prerequisites from other groups +- Required inputs or setup + +### Example +```ruby +group do + title 'Patient Profile Validation' + description %( + This group tests server support for the IPS Patient profile. It validates that + Patient resources conform to the structural and terminology requirements defined + in the IPS specification. + + Tests verify required elements, cardinality constraints, and value set bindings + specific to the IPS Patient profile. The group assumes a valid Patient resource + ID is available from previous validation steps. + ) +end +``` + +--- + +## Test Level Descriptions + +### Location and Format +Test descriptions are defined within individual test blocks: + +```ruby +test do + title 'Test Title' + description %( + [Content here] + ) +end +``` + +### Content Guidelines + +Test descriptions should be precise and actionable: + +**What is Being Tested** +- Specific requirement or capability being validated +- Reference to specification section if applicable + +**Test Steps (if complex)** +- Brief outline of what the test does +- Key validation points + +**Success Criteria** +- What constitutes a passing test +- Important edge cases handled + +**Important Notes** +- Any assumptions or limitations +- Relationship to other tests + +### Key Principles +- **Be Specific**: Describe exactly what is required and tested +- **Avoid General Guidance**: Don't include implementation advice +- **Note Discrepancies**: If the test differs from the specification requirement, explain why +- **Use Active Voice**: "This test verifies..." rather than "The server should..." + +### Examples + +**Simple Test** +```ruby +test do + title 'Server returns Patient resource' + description %( + This test verifies that the server can return a Patient resource for the provided ID. + The test performs a FHIR read operation and validates that the response has a 200 status + and contains a Patient resource with the correct ID. + ) +end +``` + +**Complex Test** +```ruby +test do + title 'Patient resource conforms to IPS Patient profile' + description %( + This test validates that the Patient resource conforms to the IPS Patient profile + (http://hl7.org/fhir/uv/ips/StructureDefinition/Patient-uv-ips). + + The test verifies: + - Required elements are present (identifier, name, gender) + - Cardinality constraints are met + - Value set bindings are correct for coded elements + - Extensions are properly used according to the profile + + Note: This test uses the Patient resource retrieved in the previous test and will + skip if no valid Patient resource is available. + ) +end +``` + +**Test with Specification Discrepancy** +```ruby +test do + title 'Bundle contains required Composition resource' + description %( + This test verifies that the IPS Bundle contains exactly one Composition resource + as required by the IPS specification section 2.3.1. + + The test searches for Composition resources within the Bundle entries and validates + that exactly one is present. While the specification requires the Composition to be + the first entry, this test does not enforce entry order due to common implementation + variations that do not affect clinical meaning. + ) +end +``` + +--- + +## Best Practices + +### General Writing Guidelines + +1. **Clarity Over Brevity**: Be clear and specific rather than concise +2. **Active Voice**: Use active voice ("This test verifies..." not "The server should...") +3. **Present Tense**: Describe what the test does, not what it will do +4. **Avoid Jargon**: Write for implementers who may not be familiar with all terminology +5. **Be Precise**: Specify exact requirements and validation points + +### Maintaining Consistency Across Tests + +**Terminology Consistency** +- Use consistent terminology throughout the test kit (e.g., always use "IPS Bundle" not "International Patient Summary Bundle" in some places and "IPS Bundle" in others) +- Maintain a glossary of key terms and their preferred usage +- Use the same phrasing for similar concepts across different tests + +**Structural Consistency** +- Follow the same description structure for similar test types +- Use consistent formatting for specification references +- Apply the same level of detail for comparable tests + +**Cross-Reference Management** +- When tests depend on each other, use consistent language to describe those dependencies +- Reference other tests by their exact titles when mentioning relationships +- Keep dependency descriptions up-to-date when test titles or structures change + +### Handling Dynamic and Version-Specific Content + +**Version References** +- Always specify the exact version of specifications being tested (e.g., "IPS Implementation Guide v1.1.0") +- Update version references consistently across all levels when upgrading +- Include links to specific versions rather than "current" or "latest" versions + +**Dynamic Content Management** +- For test kits that support multiple versions, clearly indicate which version is being tested +- When test coverage changes, update the status sections to reflect current capabilities +- Use placeholder patterns for content that changes frequently (e.g., "Tests X of Y profiles" where X and Y are maintained programmatically) + +**Configuration-Dependent Descriptions** +- When tests behave differently based on configuration, describe the variations +- For optional tests or groups, clearly explain when they should be run +- Document preset-specific behavior in relevant descriptions + +### Description Evolution and Maintenance + +**Keeping Descriptions Current** +- Review descriptions whenever test logic changes +- Update examples and validation points when test implementations evolve +- Ensure specification references remain valid and current + +**Change Management** +- When modifying test behavior, update descriptions before or alongside code changes +- Consider the impact of description changes on user understanding +- Maintain backwards compatibility in description structure when possible + +**Quality Assurance** +- Regularly audit descriptions against actual test behavior +- Verify that all specification links are functional +- Check that examples accurately reflect current implementation patterns + +**Documentation Debt Prevention** +- Include description updates as part of the definition of done for test changes +- Use automated checks where possible to detect description-code mismatches +- Schedule periodic reviews of description accuracy and completeness + +### Level-Specific Guidelines + +**Test Kit Level** +- Write for decision-makers evaluating the test kit +- Include comprehensive scope and status information +- Provide clear next steps for users + +**Suite Level** +- Write for implementers planning their testing approach +- Focus on practical usage scenarios +- Highlight key capabilities and prerequisites + +**Group Level** +- Write for implementers working on specific resources/features +- Be concise but complete +- Clarify dependencies and relationships + +**Test Level** +- Write for implementers debugging specific failures +- Be extremely specific about what is tested +- Include relevant specification references +- Note any test limitations or assumptions + +### Common Mistakes to Avoid + +1. **Vague Language**: Avoid "may", "should", "generally" - be specific +2. **Implementation Guidance**: Don't include how to implement, only what is tested +3. **Outdated References**: Ensure specification links and version numbers are current +4. **Assumption of Knowledge**: Don't assume readers know the full context +5. **Missing Context**: Always explain why a test exists and what it validates +6. **Format Support Claims**: Only mention supported formats (e.g., JSON-only, not XML) + +### Markdown Formatting + +- Use `**bold**` for emphasis on key points +- Use `- bullet points` for lists of capabilities or requirements +- Use `[link text](URL)` for specification references +- Use `code blocks` for technical terms or resource names +- Use `` only in test kit descriptions to separate overview from details + +### Specification References + +- Always link to the specific version of the implementation guide +- Reference section numbers when applicable +- Explain any deviations from the specification +- Keep links current and verify they work + +This focused approach ensures that descriptions serve their primary purpose: helping implementers understand exactly what each test validates and why it matters. diff --git a/lib/ips/docref_operation.rb b/lib/ips/docref_operation.rb index 1e131d8..c612d4d 100644 --- a/lib/ips/docref_operation.rb +++ b/lib/ips/docref_operation.rb @@ -3,20 +3,37 @@ class DocRefOperation < Inferno::TestGroup title 'DocRef Operation Tests' description %( - Verify support for the $docref operation as as described in the [IPS - Guidance](http://hl7.org/fhir/uv/ips/STU1.1/ipsGeneration.html). + This group tests server support for the DocumentReference/$docref operation + as referenced in the IPS Implementation Guide v1.1.0. The tests validate + both the operational capability declaration and basic operation functionality. - Note that this currently does not request an IPS bundle specifically - therefore does not validate the content. + Key capabilities tested: + - CapabilityStatement declares $docref operation support + - $docref operation responds successfully to requests + + Note: These tests currently validate basic operation functionality but do not + specifically request IPS bundles or validate IPS-specific content. Future + versions may include more comprehensive IPS-specific validation. ) id :ips_docref_operation run_as_group optional test do - title 'IPS Server declares support for $docref operation in CapabilityStatement' + title 'Server declares support for $docref operation in CapabilityStatement' description %( - The IPS Server declares support for DocumentReference/$docref operation in its server CapabilityStatement + This test verifies that the server declares support for the DocumentReference/$docref + operation in its CapabilityStatement. The test retrieves the server's CapabilityStatement + and searches for the $docref operation definition in the DocumentReference resource operations. + + The test validates: + - HTTP 200 response when retrieving CapabilityStatement + - CapabilityStatement is a valid CapabilityStatement resource + - DocumentReference resource declares $docref operation support + + The test checks for either: + - An operation with the definition URL http://hl7.org/fhir/uv/ipa/OperationDefinition/docref + - An operation with the name "docref" (case-insensitive) ) # link 'http://build.fhir.org/composition-operation-document.html' @@ -40,11 +57,20 @@ class DocRefOperation < Inferno::TestGroup end test do - title 'Server responds successfully to a $docref operation' + title 'Server responds successfully to $docref operation request' description %( - This test creates a $docref operation request for a patient. Note that this - currently does not request an IPS bundle specifically therefore does not validate - the content. + This test verifies that the server responds successfully to a DocumentReference/$docref + operation request. The test constructs a FHIR Parameters resource with a patient + parameter and sends it to the $docref operation endpoint. + + The test validates: + - HTTP 200 response status from the $docref operation + + The test requires a valid patient_id input and creates a docref_operation request + that can be referenced by subsequent tests. + + Note: This test currently validates basic operation functionality but does not + specifically request IPS bundles or validate IPS-specific content in the response. ) input :patient_id diff --git a/lib/ips/ips_resource_validation.rb b/lib/ips/ips_resource_validation.rb index 646d98b..f7c1362 100644 --- a/lib/ips/ips_resource_validation.rb +++ b/lib/ips/ips_resource_validation.rb @@ -3,25 +3,41 @@ class IPSResourceValidation < Inferno::TestGroup title 'IPS Resource Validation Tests' id :ips_resource_validation description %( - This group performs content validation, which is useful for evaluating - conformance of systems that produce IPS bundles but do not generate - content using a FHIR interface as described in the [IPS - Guidance](http://hl7.org/fhir/uv/ips/STU1.1/ipsGeneration.html). - - There is currently a single validation test for IPS bundles without any - other context-specific constraints. Therefore, this test measures a - systems ability to produce a single, valid IPS bundle. Future enhancements - to these tests should require systems to demonstrate more than just a - single valid IPS bundle. + This group validates pre-existing IPS bundles for conformance to the IPS Bundle profile + without requiring a live FHIR server. It is designed for testing static IPS documents + or validating bundle structure and profile conformance offline. + + The validation tests accept IPS bundle content in JSON format and verify + conformance to the IPS Bundle profile (http://hl7.org/fhir/uv/ips/StructureDefinition/Bundle-uv-ips) + as defined in the IPS Implementation Guide v1.1.0. + + This testing approach is ideal for: + - Systems that generate IPS bundles through non-FHIR interfaces + - Offline validation of existing IPS documents + - Initial conformance testing during development + - Validating IPS bundles before deployment + + The group currently provides basic IPS Bundle profile validation. Additional context-specific + validation tests may be added to verify specific clinical scenarios or content requirements. ) group do - title 'IPS Bundle with no other constraints' + title 'Basic IPS Bundle Validation' test do title 'IPS Bundle meets constraints provided in the IPS Bundle profile' description %( - This test will validate the content of an IPS bundle to ensure it is valid. + This test validates that the provided bundle content is a valid FHIR Bundle resource + that conforms to the IPS Bundle profile (http://hl7.org/fhir/uv/ips/StructureDefinition/Bundle-uv-ips). + + The test performs the following validations: + - Parses the provided bundle content (JSON format) + - Verifies the resource is of type Bundle + - Validates conformance to the IPS Bundle profile constraints + + The test will skip if no bundle content is provided. Profile validation includes + verification of required elements, cardinality constraints, terminology bindings, + and structural requirements as defined in the IPS Implementation Guide v1.1.0. ) input :bundle_content, title: 'IPS Bundle', type: 'textarea', optional: true, description: 'Validate a single IPS bundle (optional)' diff --git a/lib/ips/metadata.rb b/lib/ips/metadata.rb index 89acf69..2cb2d6c 100644 --- a/lib/ips/metadata.rb +++ b/lib/ips/metadata.rb @@ -5,41 +5,65 @@ class Metadata < Inferno::TestKit id :ips_test_kit title 'International Patient Summary Test Kit' description <<~DESCRIPTION - The International Patient Summary Test Kit provides an - executable set of tests for the [International Patient Summary (IPS) - Implementation Guide v1.1.0](https://hl7.org/fhir/uv/ips/STU1.1/). This test kit - simulates requests performed by a realistic IPS Requestor and - validating responses based on requirements specified within the IPS IG and the base FHIR specification. + The International Patient Summary Test Kit provides an executable set of + tests for IPS server implementers to validate their implementations + against the [International Patient Summary (IPS) Implementation Guide + v1.1.0](https://hl7.org/fhir/uv/ips/STU1.1/). - This test kit is [open source](https://github.com/inferno-framework/ips-test-kit#license) and freely available for use or - adoption by the health IT community including EHR vendors, health app - developers, and testing labs. It is built using the [Inferno Framework](https://inferno-framework.github.io/inferno-core/). The Inferno Framework is - designed for reuse and aims to make it easier to build test kits for any - FHIR-based data exchange. + This test kit validates server implementations of the International + Patient Summary (IPS) specification. It tests both the structural + requirements of IPS documents and the operational capabilities required by + IPS servers. + + The tests are organized into three main approaches: + - **IPS Resource Validation Tests**: For validating standalone IPS bundles without requiring a server API + - **IPS Operation Tests**: For testing server support of the $summary operation and document reference capabilities + - **IPS Read Tests**: For validating individual resource profile conformance via FHIR read operations + + Key capabilities tested include: + - IPS Bundle structure and composition validation + - Profile conformance for IPS resource profiles + - Support for the $summary operation + - Document reference and retrieval workflows + - Terminology validation and value set bindings ## Status - These tests are intended to allow IPS server implementers to perform checks of their server against IPS requrirements. Future versions of these tests may validate other requirements and may change how these are tested. + This test kit has **Low** maturity. It provides comprehensive coverage of + IPS profile validation and core operational requirements. The test kit + currently validates resource profiles defined in the IPS + Implementation Guide v1.1.0, including all required profiles for Patient, + AllergyIntolerance, Condition, Medication resources, and comprehensive + coverage of observation profiles for laboratory, pathology, and radiology + results. - The test kit currently tests the following requirements: - - IPS Bundle validation - - Support for IPS operations - - Profile Validation + Two preset configurations are available to facilitate testing: one using + example IPS bundles from the specification, and another configured for + testing against reference server implementations. - See the test descriptions within the test kit for detail on the specific validations performed as part of testing these requirements. + Future versions may expand operational workflow testing and add + comprehensive edge case coverage. ## Repository - The IPS Test Kit GitHub repository can be [found here](https://github.com/inferno-framework/ips-test-kit). + This test kit is [open + source](https://github.com/inferno-framework/ips-test-kit#license) and + freely available for use or adoption by the health IT community including + EHR vendors, health app developers, and testing labs. It is built using + the [Inferno + Framework](https://inferno-framework.github.io/). + + The IPS Test Kit GitHub repository can be [found + here](https://github.com/inferno-framework/ips-test-kit). - ## Providing Feedback and Reporting Issues + ## Providing Feedback We welcome feedback on the tests, including but not limited to the following areas: - - Validation logic, such as potential bugs, lax checks, and unexpected failures. - - Requirements coverage, such as requirements that have been missed, tests that necessitate features that the IG does not require, or other issues with the interpretation of the IG's requirements. - - User experience, such as confusing or missing information in the test UI. + - Validation logic, such as potential bugs, lax checks, and unexpected failures + - Requirements coverage, such as requirements that have been missed, tests that necessitate features that the IG does not require, or other issues with the interpretation of the IG's requirements + - User experience, such as confusing or missing information in the test UI Please report any issues with this set of tests in the [issues section](https://github.com/inferno-framework/ips-test-kit/issues) of the repository. DESCRIPTION diff --git a/lib/ips/summary_operation.rb b/lib/ips/summary_operation.rb index db74012..441ed6f 100644 --- a/lib/ips/summary_operation.rb +++ b/lib/ips/summary_operation.rb @@ -5,8 +5,18 @@ module IPS class SummaryOperation < Inferno::TestGroup title 'Summary Operation Tests' description %( - Verify support for the $summary operation as as described in the [IPS - Guidance](http://hl7.org/fhir/uv/ips/STU1.1/ipsGeneration.html). + This group tests server support for the Patient/$summary operation as defined in the + IPS Implementation Guide v1.1.0. The tests validate both the operational capability + declaration and the structure and content of returned IPS Bundle resources. + + Key capabilities tested: + - CapabilityStatement declares $summary operation support + - $summary operation returns valid IPS Bundle + - Bundle contains required Composition as first entry + - Bundle contains valid IPS resource profiles for core clinical data + + Tests use GET requests for the $summary operation and validate returned Bundle + entries against their respective IPS profiles. ) id :ips_summary_operation run_as_group @@ -18,9 +28,16 @@ class SummaryOperation < Inferno::TestGroup test from: :ips_summary_operation_valid_composition test do - title 'IPS Server returns Bundle resource containing valid IPS MedicationStatement entry' + title 'Bundle contains valid IPS MedicationStatement resources' description %( - IPS Server return valid IPS MedicationStatement resource in the Bundle as first entry + This test verifies that the Bundle returned by the $summary operation contains + at least one MedicationStatement resource and that all MedicationStatement + resources conform to the IPS MedicationStatement profile. + + The test skips if no Bundle was returned from the $summary operation. It checks + for the presence of MedicationStatement resources within the Bundle entries and + validates each MedicationStatement against the IPS MedicationStatement profile + (http://hl7.org/fhir/uv/ips/StructureDefinition/MedicationStatement-uv-ips). ) # link 'http://hl7.org/fhir/uv/ips/StructureDefinition-MedicationStatement-uv-ips.html' uses_request :summary_operation @@ -41,9 +58,16 @@ class SummaryOperation < Inferno::TestGroup end test do - title 'IPS Server returns Bundle resource containing valid IPS AllergyIntolerance entry' + title 'Bundle contains valid IPS AllergyIntolerance resources' description %( - IPS Server return valid IPS AllergyIntolerance resource in the Bundle as first entry + This test verifies that the Bundle returned by the $summary operation contains + at least one AllergyIntolerance resource and that all AllergyIntolerance + resources conform to the IPS AllergyIntolerance profile. + + The test skips if no Bundle was returned from the $summary operation. It checks + for the presence of AllergyIntolerance resources within the Bundle entries and + validates each AllergyIntolerance against the IPS AllergyIntolerance profile + (http://hl7.org/fhir/uv/ips/StructureDefinition/AllergyIntolerance-uv-ips). ) # link 'http://hl7.org/fhir/uv/ips/StructureDefinition-AllergyIntolerance-uv-ips.html' uses_request :summary_operation @@ -64,9 +88,16 @@ class SummaryOperation < Inferno::TestGroup end test do - title 'IPS Server returns Bundle resource containing valid IPS Condition entry' + title 'Bundle contains valid IPS Condition resources' description %( - IPS Server return valid IPS Condition resource in the Bundle as first entry + This test verifies that the Bundle returned by the $summary operation contains + at least one Condition resource and that all Condition resources conform to + the IPS Condition profile. + + The test skips if no Bundle was returned from the $summary operation. It checks + for the presence of Condition resources within the Bundle entries and validates + each Condition against the IPS Condition profile + (http://hl7.org/fhir/uv/ips/StructureDefinition/Condition-uv-ips). ) # link 'http://hl7.org/fhir/uv/ips/StructureDefinition-Condition-uv-ips.html' uses_request :summary_operation diff --git a/lib/ips/summary_operation_return_bundle.rb b/lib/ips/summary_operation_return_bundle.rb index e3725e3..7d70e18 100644 --- a/lib/ips/summary_operation_return_bundle.rb +++ b/lib/ips/summary_operation_return_bundle.rb @@ -2,19 +2,24 @@ module IPS class SummaryOperationReturnBundle < Inferno::Test - title 'IPS Server returns Bundle resource for Patient/[id]/$summary GET operation' + title 'Server returns valid IPS Bundle for Patient/$summary operation' description %( - IPS Server returns a valid IPS Bundle resource as successful result of - $summary operation. + This test verifies that the server returns a valid IPS Bundle resource when + the Patient/$summary operation is invoked. The test performs a GET request + to the Patient/[id]/$summary endpoint and validates the response. - This test currently only issues a GET request for the summary due to a - limitation in Inferno in issuing POST requests that omit a Content-Type - header when the body is empty. Inferno currently adds a `Content-Type: - application/x-www-form-urlencoded` header when issuing a POST with no - body, which causes issues in known reference implementations. + The test validates: + - HTTP 200 response status + - Response contains a Bundle resource + - Bundle conforms to the IPS Bundle profile (http://hl7.org/fhir/uv/ips/StructureDefinition/Bundle-uv-ips) - A future update to this test suite should include a required POST - request as well as an optional GET request for this content. + Note: This test currently uses only GET requests due to a limitation in Inferno + where POST requests with empty bodies include an unwanted Content-Type header + that causes issues with some reference implementations. Future versions may + include both GET and POST request testing. + + This test requires a valid patient_id input and creates a summary_operation + request that can be referenced by subsequent tests. ) id :ips_summary_operation_return_bundle diff --git a/lib/ips/summary_operation_support.rb b/lib/ips/summary_operation_support.rb index 8b3fa4a..1832949 100644 --- a/lib/ips/summary_operation_support.rb +++ b/lib/ips/summary_operation_support.rb @@ -2,9 +2,17 @@ module IPS class SummaryOperationSupport < Inferno::Test - title 'IPS Server declares support for $summary operation in CapabilityStatement' + title 'Server declares support for $summary operation in CapabilityStatement' description %( - The IPS Server declares support for Patient/[id]/$summary operation in its server CapabilityStatement + This test verifies that the server declares support for the Patient/$summary operation + in its CapabilityStatement. The test retrieves the server's CapabilityStatement and + searches for the $summary operation definition in the Patient resource operations. + + The test checks for either: + - An operation with the definition URL http://hl7.org/fhir/uv/ips/OperationDefinition/summary + - An operation with the name "summary" or "patient-summary" (case-insensitive) + + This test expects a successful HTTP 200 response when retrieving the CapabilityStatement. ) id :ips_summary_operation_support diff --git a/lib/ips/summary_operation_valid_composition.rb b/lib/ips/summary_operation_valid_composition.rb index 01b26a3..c37ae09 100644 --- a/lib/ips/summary_operation_valid_composition.rb +++ b/lib/ips/summary_operation_valid_composition.rb @@ -2,8 +2,20 @@ module IPS class SummaryOperationValidComposition < Inferno::Test - title 'IPS Server returns Bundle resource containing valid IPS Composition entry' - description 'IPS Server return valid IPS Composition resource in the Bundle as first entry' + title 'Bundle contains valid IPS Composition as first entry' + description %( + This test verifies that the Bundle returned by the $summary operation contains + a valid IPS Composition resource as the first entry, as required by the IPS + specification. + + The test validates: + - Bundle contains at least one entry + - First entry in the Bundle is a Composition resource + - Composition conforms to the IPS Composition profile (http://hl7.org/fhir/uv/ips/StructureDefinition/Composition-uv-ips) + + The test skips if no Bundle was returned from the $summary operation. This test + uses the Bundle resource from the previous summary_operation request. + ) id :ips_summary_operation_valid_composition uses_request :summary_operation diff --git a/lib/ips_test_kit.rb b/lib/ips_test_kit.rb index f19180a..78cbd13 100644 --- a/lib/ips_test_kit.rb +++ b/lib/ips_test_kit.rb @@ -6,29 +6,70 @@ class Suite < Inferno::TestSuite title 'International Patient Summary (IPS) v1.1.0' short_title 'IPS v1.1.0' description %( - This test suite evaluates the ability of a system to provide patient - summary data expressed using HL7® FHIR® in accordance with the - [International Patient Summary Implementation Guide (IPS - IG) v1.1.0](https://www.hl7.org/fhir/uv/ips/STU1.1). - - Because IPS bundles can be generated and transmitted in many different - ways beyond a traditional FHIR RESTful server, this test suite allows you - to optionally evaluate a single bundle that is not being provided by a server in the - 'IPS Resource Validation Tests'. - - For systems that support a standard API for generating and communicating - these bundles in accordance with the guidance provided in the IG, use the - 'IPS Operation Tests'. - - For systems that also provide a FHIR API access to the components resources - of the IPS bundle, use the 'IPS Read Tests'. - - This suite provides two presets: - * HL7.org IPS Server: Hosted reference IPS Server. This is suitable for running - the 'Operation' and 'Read' tests. Resource IDs may not remain valid as this is an - open server. - * IPS Example Summary Bundle: Populates the 'IPS Resource Validation Test' with an - example provided in the IG. + This test suite evaluates the ability of systems to provide patient summary data + expressed using HL7® FHIR® R4 in accordance with the [International Patient Summary + Implementation Guide (IPS IG) v1.1.0](https://www.hl7.org/fhir/uv/ips/STU1.1). + + The suite provides three distinct testing approaches to accommodate different + implementation scenarios and system capabilities: + + **Testing Approaches:** + + - **IPS Resource Validation Tests**: For validating pre-existing IPS bundles without + requiring a live FHIR server. Use this approach when you have static IPS documents + to validate or when testing bundle structure and profile conformance offline. + + - **IPS Operation Tests**: For testing systems that implement the $summary operation + and document reference capabilities as defined in the IPS IG. Use this approach + when your system generates IPS bundles dynamically via FHIR operations. + + - **IPS Read Tests**: For validating individual IPS resource profiles via standard + FHIR read operations. Use this approach when your system provides RESTful access + to individual IPS resources and you want to test profile conformance at the + resource level. + + **Key Capabilities Tested:** + + *Structural Validation:* + - IPS Bundle structure and composition requirements + - Resource entry organization and references + - Required vs. optional section validation + + *Profile Conformance:* + - All required IPS profiles (Patient, AllergyIntolerance, Condition, Medication, etc.) + - Optional IPS profiles for comprehensive coverage + - Cardinality constraints and element requirements + + *Operational Capabilities:* + - $summary operation implementation and response validation + - Document reference creation and retrieval workflows + - Bundle generation from patient data + + *Terminology Validation:* + - Value set bindings for coded elements + - Required terminology systems and codes + - IPS-specific code system usage + + **Prerequisites:** + - FHIR R4 server implementation (for Operation and Read tests) + - Valid patient data conforming to IPS profiles + - For Operation tests: Implementation of the $summary operation + - For Read tests: RESTful FHIR API with read capabilities + + **Available Presets:** + + - **HL7.org IPS Server**: Pre-configured for testing against the hosted reference + IPS server. Suitable for Operation and Read tests. Note that resource IDs may + change as this is a shared testing environment. + + - **IPS Example Summary Bundle**: Pre-loads the Resource Validation tests with + example IPS bundles from the specification. Ideal for initial validation and + understanding IPS structure requirements. + + **Scope and Limitations:** + This suite focuses on IPS document structure, profile conformance, and core operational + requirements. It does not test advanced workflow scenarios, security implementations, + or integration with external systems beyond the IPS specification scope. ) id 'ips' @@ -92,6 +133,34 @@ class Suite < Inferno::TestSuite group do title 'IPS Server Read and Validate Profiles Tests' short_title 'IPS Read Tests' + description %( + This group tests server support for individual IPS resource profiles via standard + FHIR read operations. It validates that servers can provide RESTful access to IPS + resources and that these resources conform to their respective IPS profile requirements. + + Each resource type test group performs two key validations: + - **Read Operation Test**: Executes a FHIR read operation using the provided resource ID, + verifies successful HTTP 200 response, confirms correct resource type, and validates + that the returned resource has the expected ID + - **Profile Conformance Test**: Validates that the retrieved resource conforms to its + specific IPS profile requirements including structural constraints, cardinality rules, + and terminology bindings + + This testing approach is suitable for servers that store IPS resources individually + and provide standard FHIR RESTful API access. It requires valid resource IDs for + each resource type being tested. + + **Prerequisites**: + - FHIR R4 server with RESTful read capabilities + - Valid resource IDs for the IPS resources to be tested + - Resources that conform to IPS profile requirements + + **Coverage**: Tests all IPS resource profiles including: + - Core structural profiles (Bundle, Composition, Patient) - always required + - Required section profiles (AllergyIntolerance, Condition, MedicationStatement/MedicationRequest/Medication) + - Recommended section profiles (Immunization, Procedure, DiagnosticReport, Device profiles) + - Optional section profiles (Observation variants for vital signs, pregnancy, social history, etc.) + ) optional input :url, title: 'IPS FHIR Server Base URL'