Let a retry policy decide will_be_retried in the runner by Enceradeira · Pull Request #345 · cucumber/cucumber-ruby-core

Enceradeira · 2026-09-05T13:21:03Z

Description

Alternative to #344, fixing cucumber/cucumber-ruby#1905 together with cucumber/cucumber-ruby#1910.

Since 19.0.0 the Runner emits the TestCaseFinished envelope and decides its will_be_retried field from a maximum number of attempts. The runner cannot know that on its own: whether a test case runs again is decided by whoever runs it again, which in cucumber-ruby is the retry filter honouring both --retry and --retry-total. #344 copies the --retry-total counter into the runner, which makes the reported symptom go away but keeps the retry decision in two places, and leaves two further defects in the message stream:

cucumber-ruby passes the number of retries where the runner expects the number of attempts. With --retry 1 the flag is never true, so a retried attempt is reported as a final result and the HTML report shows the scenario twice. With --retry 2 the second attempt is reported as final although a third one follows.
The runner broadcasts the test_case_finished event before emitting the TestCaseFinished envelope. The retry filter re-runs the test case from inside that event, so the envelope of the first attempt is emitted after the retry and carries the test_case_started_id of the retry. On main, a retried scenario produces two TestCaseFinished envelopes for attempt 2 and none for attempt 1.

The compatibility kit does not catch any of this because it only compares message keys, not values.

This PR:

replaces the max_attempts constructor argument with a retry_policy, an object answering will_be_retried?(test_case, result). With no policy, no test case is reported as retried.
emits the TestCaseFinished envelope before broadcasting the test_case_finished event, so a retry started from that event no longer reorders the stream.
adds specs for the envelope values and for the re-entrant retry case, which previously had no coverage.

Type of change

Bug fix
Breaking change (constructor argument of Cucumber::Core::Test::Runner), see upgrading_notes/20.0.0.md

Checklist:

Tests have been added for any changes to behaviour of the code
New and existing tests are passing locally and on CI
bundle exec rubocop reports no offenses
RDoc comments have been updated
CHANGELOG.md has been updated

The runner emits the TestCaseFinished envelope after broadcasting the test_case_finished event. The retry filter in cucumber-ruby re-runs the test case from inside that event, so the envelope of the first attempt is emitted after the whole retry and carries the test_case_started id of the retry instead of its own. The runner also derives will_be_retried from a maximum number of attempts, which it cannot know: whether a test case runs again is decided by whoever runs it again, and the retry filter stops retrying once --retry-total is exhausted. The runner has to ask a retry policy. These specs fail until the runner emits the envelope before the event and asks a retry policy. Related to cucumber/cucumber-ruby#1905.

The runner computed the will_be_retried field of the TestCaseFinished envelope from a maximum number of attempts. It cannot know that on its own: whether a test case runs again is decided by whoever runs it again, which in cucumber-ruby is the retry filter honouring both --retry and --retry-total. The runner now asks a retry policy instead. The envelope is also emitted before the test_case_finished event. The retry filter re-runs the test case from inside that event, so the envelope of the first attempt used to be emitted after the retry, with the test_case_started id of the retry. Related to cucumber/cucumber-ruby#1905.

luke-hill · 2026-09-08T09:30:58Z

Heya. Just posting this once @Enceradeira as I know you've done work across both repos.

I've been away for a bit and will likely not be involved much with this in the coming week or so.

Thankyou for your contribution. We have a call on thursdays (Although I'm not there this thursday), so I will try to get to this ASAP (But it could be a while). Rest assured it's not forgotten

The retry policy was the fourth positional argument of the runner, so passing one meant spelling out the id generator and backtrace filter defaults as well. id_generator, backtrace_filter and retry_policy are now keyword arguments; Runner.new(event_bus) keeps working as before. Related to cucumber/cucumber-ruby#1910 (comment)

luke-hill

Added some thoughts whilst I'm around for now. Will be away for another week or so.

luke-hill · 2026-09-15T15:58:36Z

              test_case_started_id: @current_test_case_started_id,
              timestamp: time_to_timestamp(Time.now),
-              will_be_retried: result.failed? && (@attempt < @max_attempts)
+              will_be_retried: @retry_policy&.will_be_retried?(test_case, result) || false


Instead of having this nil safe check and an alternation, can we instead have the retry policy always instantiated and return a value?

Done in f398b79: the runner now defaults to a NoRetries policy, so the nil check is gone.

luke-hill · 2026-09-15T15:59:37Z

+### With cucumber-core 20.0.0
+
+```ruby
+class RetryPolicy


Lets namespace this just to cover ourselves as it's completely new

Maybe with a comment for the file path ref that is the default one

Done in 6944446: the example is now MyProject::RetryPolicy, with a comment pointing to the cucumber-ruby implementation.

luke-hill · 2026-09-15T16:02:26Z

-        def initialize(event_bus, id_generator = Cucumber::Messages::Helpers::IdGenerator::UUID.new, backtrace_filter = nil, max_attempts = 1)
+        # @param retry_policy [#will_be_retried?, nil] asked, once a test case has finished, whether it is going to be
+        #   run again. It receives the test case and its result. When nil, no test case is ever reported as retried.
+        def initialize(event_bus, id_generator: Cucumber::Messages::Helpers::IdGenerator::UUID.new, backtrace_filter: nil, retry_policy: nil)


I'd rather have the retry policy not be something passed in as an extraneous class. But instead something that is forcibly set based on configuration.

So I envisage something like

@configuration.retry_policy.will_be_retried?

Thanks! Just to make sure I understand you correctly: core has no configuration object, Configuration lives in cucumber-ruby. Did you mean the policy built there (Configuration#retry_policy, as in your comment on cucumber/cucumber-ruby#1910) and handed to the runner, as it is now? Or should I introduce a configuration in core that the runner reads the policy from?

The runner guarded the retry policy against nil. It now defaults to NoRetries, so it always has a policy to ask. Related to #345 (comment)

Related to #345 (comment)

luke-hill

Reviewed 3/5 (mini bits), will get to the rest soon.

Again thankyou for your work, sorry for delays

luke-hill

One remaining request for an additional test. Then this is good and RtM

…y policy

luke-hill · 2026-10-05T11:02:58Z

Thankyou for all your hard work @Enceradeira This is RtM. I'll work on cutting a release for core soon. Probably in a couple of days

Enceradeira mentioned this pull request Sep 5, 2026

Report will_be_retried from the retry filter's decision cucumber/cucumber-ruby#1910

Merged

5 tasks done

Enceradeira force-pushed the fix/cucumber-ruby-1905-will-be-retried branch 2 times, most recently from 19900a2 to d98745a Compare September 5, 2026 14:51

Enceradeira force-pushed the fix/cucumber-ruby-1905-will-be-retried branch from d98745a to 3419704 Compare September 5, 2026 14:55

Enceradeira force-pushed the fix/cucumber-ruby-1905-will-be-retried branch from 3419704 to 52656e0 Compare September 5, 2026 14:57

This was referenced Sep 5, 2026

retry-total conflicts with html-formatter when exhaustion limit hit cucumber/cucumber-ruby#1905

Closed

Handle the retry_total_tests configuration option in the runner #344

Closed

Enceradeira marked this pull request as ready for review September 5, 2026 15:51

Enceradeira requested review from brasmusson and luke-hill September 5, 2026 15:52

Enceradeira force-pushed the fix/cucumber-ruby-1905-will-be-retried branch from 2d7df3f to 9ab0226 Compare September 14, 2026 12:46

luke-hill reviewed Sep 15, 2026

View reviewed changes

Enceradeira added 2 commits September 25, 2026 18:40

Default the runner to a retry policy that never retries

f398b79

The runner guarded the retry policy against nil. It now defaults to NoRetries, so it always has a policy to ask. Related to #345 (comment)

Namespace the retry policy example in the upgrading notes

6944446

Related to #345 (comment)

luke-hill reviewed Sep 29, 2026

View reviewed changes

Comment thread lib/cucumber/core/test/no_retries.rb

luke-hill reviewed Sep 29, 2026

View reviewed changes

luke-hill reviewed Oct 5, 2026

View reviewed changes

Comment thread spec/cucumber/core/test/runner_spec.rb

luke-hill reviewed Oct 5, 2026

View reviewed changes

Specify the runner asks NoRetries by default and honours a given retr…

ecd3dd9

…y policy

luke-hill approved these changes Oct 5, 2026

View reviewed changes

luke-hill merged commit 8272f24 into main Oct 5, 2026
10 checks passed

luke-hill deleted the fix/cucumber-ruby-1905-will-be-retried branch October 5, 2026 11:03

Navigation Menu

Repository navigation

Let a retry policy decide will_be_retried in the runner - #345

Let a retry policy decide will_be_retried in the runner#345
luke-hill merged 6 commits into
mainfrom
fix/cucumber-ruby-1905-will-be-retried

Enceradeira commented Sep 5, 2026 •

edited

Loading

Uh oh!

luke-hill commented Sep 8, 2026

Uh oh!

luke-hill left a comment

Uh oh!

luke-hill Sep 15, 2026

Uh oh!

Enceradeira Sep 25, 2026

Uh oh!

luke-hill Sep 15, 2026

Uh oh!

Enceradeira Sep 25, 2026

Uh oh!

luke-hill Sep 15, 2026

Uh oh!

Enceradeira Sep 25, 2026

Uh oh!

Uh oh!

luke-hill left a comment

Uh oh!

Uh oh!

luke-hill left a comment

Uh oh!

luke-hill commented Oct 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Enceradeira commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Checklist:

Uh oh!

luke-hill commented Sep 8, 2026

Uh oh!

luke-hill left a comment

Choose a reason for hiding this comment

Uh oh!

luke-hill Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

Enceradeira Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

luke-hill Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

Enceradeira Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

luke-hill Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

Enceradeira Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

luke-hill left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

luke-hill left a comment

Choose a reason for hiding this comment

Uh oh!

luke-hill commented Oct 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Enceradeira commented Sep 5, 2026 •

edited

Loading