Skip to content

README.md

Latest commit

 

History

History
126 lines (88 loc) · 6.54 KB

README.md

File metadata and controls

126 lines (88 loc) · 6.54 KB
Raw

LionWeb integration testing

Automated tests that check that the other repos within the LionWeb GitHub (organization) integrate well and consistently with each other.

The build

The definition of “well” in the previous sentences is not very broad — effectively it means: the build runs green. With “build” we mean the GitHub Action, from here on.

Note that this repository serves as a “canary in a coal mine”: if its build turns red, it means there’s a mismatch(/incompatibility) between either an implementation and the specification (and/or between implementations). That also means that modifications and additions to this repository – in the form of PRs – might turn the build red when run on the PR’s branch. That is a good thing: it means that the modification/addition spots an incompatibility that wasn’t spotted before.

Whenever the build of this repository fails, the steps to take are as follows:

  1. find out what the incompatibility is,
  2. find out where it originates from — typically an implementation, sometimes the specification,
  3. submit an issue on the corresponding repository — which is most likely not this repository,
  4. fix that issue (there),
  5. trigger the build here to verify it runs green again.

The build – in the form of a GitHub Action name “LionWeb integration tests” – does the following:

  • validate the delta payloads under delta/ against the Delta JSON Schema for those,
  • validate the valid serialization chunks (recognizable as such by the presence of a valid fragment in their paths) under testchanges/ and testset/ against the Serialization chunk JSON Schema for those,
  • run the integration test suite implemented in TypeScript, by running scripts/run-ts-tests.sh.

The automated tests in the cloned repositories are not run.

And remember: a green build is not a guarantee for the absence of bugs (or presence of quality), but a red build is definitely a guarantee that you need to look at something.

Note also that the build sometimes fails during the “Test all C# projects” part, with something like the following in the output:

  Error Message:
   System.Net.HttpListenerException : Address already in use
  Stack Trace:
     at System.Net.HttpEndPointManager.GetEPListener(String host, Int32 port, HttpListener listener, Boolean secure)
   at System.Net.HttpEndPointManager.AddPrefixInternal(String p, HttpListener listener)
   at System.Net.HttpEndPointManager.AddListener(HttpListener listener)
   at System.Net.HttpListener.Start()
   at LionWeb.WebSocket.WebSocketServer.StartServer(String ipAddress, Int32 port)

If this happens, then just rerun the job: usually(/often), the job succeeds the next time. Why this happens is unclear to us, as we do take pains to assign each C# WebSocket server its own, unique port number.

JSON validation

The schema validation that the build performs can’t be run locally directly, but the following should be equivalent:

$ ./scripts/validate-jsons.sh

The validate-specific-message-json.ts script can be used to troubleshoot message JSONs that don’t validate.

$ ./scripts/download-json-schemas.sh   # (download full JSON Schema for delta protocol messages)

$ ./ts/scripts/validate-specific-message-json.ts
Usage: execute
    node ts/scripts/validate-specific-message-json.ts <path to JSON with message> [message kind]
to validate that JSON as a message of the indicated kind — hopefully producing more understandable errors.
If the message kind is not given, we try to derive that from the file name, although that might fail.
In addition, a JSON schema that only pertains to that message kind is saved to a file with path "schemas/<message kind>.specific-schema.json".

$ ./ts/scripts/validate-specific-message-json.ts delta/event/ErrorEvent.delta.json
Derived message kind from path of JSON file as: ErrorEvent
JSON file with path "delta/event/ErrorEvent.delta.json" contains a valid message of kind ErrorEvent.

(All JSON files within the schemas/ dir. are Git-ignored.)

Installation requirements

  • Node.js (including NPM), currently (at least) version 24.y.z — for running Node.js scripts such as ts/scripts/validate-all-jsons.ts.
  • Java 11 for the lionweb-jvm repo.

Test language

The testLanguage directory contains (artifacts relating to) the TestLanguage language.

By running generate-test-language.sh, the TestLanguage language is:

Test data for serialization format validators

The testset/ directory contains test data that can be used to test validators that validate whether serialization chunks (JSON files) conform to the serialization format specification. The subdirectory withoutLanguage/ pertains to serialization chunks that are to be validated without a registered language. The subdirectory withLanguage/ pertains to serialization chunks that are to be validated against a registered language — m.n. the one serialized as myLang.language.json. Either subdirectories contain two subdirectories named valid and invalid. Each (nested) subdirectory under invalid contains a __TestExpectation.json file which specifies which chunk fails validation, and if so, with which error type. The format of the __TestExpectation.json files is as follows:

{
  "errors": [
    {
      "file": "<file next to __TestExpectation.json>",
      "error": "<error type>"
    },
    ...
  ]
}

Consider the following example of a member of the errors array:

{
  "file": "empty.json",
  "error": "PropertyValueIncorrect"
}

This means that validating the empty.json file is expected to produce an error of type PropertyValueIncorrect (and that it’s the first one produced).