Cucumber Messages

Cucumber Messages is a message protocol for representing results and other information from Cucumber. The protocol aims to decouple various components of the Cucumber platform, with the following advantages:

Each component only needs to know about a subset of messages
Gherkin is decoupled from the Cucumber execution component
Enables the future support other formats such as Markdown and Excel

Message Overview

Notes:

The image sketches out the general concept, but is incomplete. See relations.md for a complete visualisation of the relationships between messages.
Markdown and Excel formats are currently not supported and mentioned here as potential future alternative languages to express BDD scenarios.

JSON Schema

The jsonschema directory contains JSON Schema definitions for each message type.

See messages.md for a detailed description of each message type.

Encoding

When Cucumber Messages are stored in a file or sent over a network, they are encoded as NDJSON. We call this a message stream.

Each message in a message stream is of type Envelope.

Language implementations

Each subdirectory defines language-specific implementations of these messages, generated from the JSON schemas. The current implementations are:

Languages automatically released for every version

The release process for the below languages uses polyglot-release

.NET
Java
Javascript
Perl
PHP
Python
Ruby

Go

The Go implementation is also cut for every version but is just maintained as a GitHub tag

C++

This is currently tested but not automatically released. We are seeking help for this implementation

Elixir

There is also a subdirectory for Elixir which contains the legacy implementation which was largely driven by protobuf. This has recently been re-integrated into our CI and will be ported over to the JSON schema protocol in due course, for now it remains written in protobuf

Message emitters

Cucumber Messages are currently sent by the following versions of Cucumber (using the message formatter):

Cucumber-JVM 6.0.0 and later
Cucumber-Ruby 4.0.0 and later
Cucumber.js 7.0.0 and later

Messages emitters for the other Cucumber languages (e.g. Perl) are not yet implemented.

Why Cucumber Messages

Cucumber needs to produce results in a machine-readable format so that other tools can generate reports.

Historically, Cucumber did this with the json and junit formatters. These formats however, have several shortcomings that are now addressed by using Cucumber Messages.

The json formatter is now in maintenance mode for these implementations, and Messages is the preferred standard. See utilities for a list of tools that may help with backward and forward compatibility with the json format.

High memory footprint

JSON and XML production/consumption is done by serialising/deserialising an object graph. For "big" Cucumber runs this graph may consume a considerable amount of RAM, in particular if several large attachments (screenshots) are attached.

This can cause out of memory errors, aborting Cucumber runs or reporting jobs. It also means that no results can be consumed by downstream processors until the last scenario has finished.

With Cucumber Messages, several messages containing smaller pieces of information are emitted continuously to a stream (NDJSON, see above), avoiding high memory consumption and enabling real-time processing of results.

Lack of a schema

The JSON reporter does not have a formal schema. This has led to slightly inconsistent implementations of the JSON formatter in various Cucumber implementations. Consumers of the JSON format have to anticipate and detect these inconsistencies and try to cope with them.

Limited information

The junit XML format can only contain very limited information such as test case name and status. While there isn't an official schema for JUnit XML, there are a few defacto ones around which are very limited.

The json format represents the following information:

Gherkin source (as a rough and lossy representation of a Gherkin document's abstract syntax tree)
Attachments (formerly called embeddings)
Path and line for step definitions
Results for hooks

However, it does not contain the following information (but Cucumber Messages does):

Original source code of the Gherkin document
Gherkin document abstract syntax tree
Step definitions
Parameter types

This kind of information is required to produce rich reports and analytics, and is used in @cucumber/react-components and Cucumber Reports.

Examples

You will find examples of Cucumber Messages in the compatibility-kit project.

Utilities

@cucumber/react - React component that renders Cucumber Messages nicely
- Used internally by Cucumber Reports.

Uh oh!

License

cucumber/messages

About

Topics

Resources

License

Code of conduct

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases 26

Sponsor this project

Uh oh!

Used by 28.2k

Contributors 37

Languages