Let a retry policy decide will_be_retried in the runner

CHANGELOG.md

@@ -9,6 +9,14 @@ This document is formatted according to the principles of [Keep A CHANGELOG](htt
 Please visit [cucumber/CONTRIBUTING.md](https://github.com/cucumber/cucumber/blob/master/CONTRIBUTING.md) for more info on how to contribute to Cucumber.
 
 ## [Unreleased]
+### Changed
+- **BREAKING CHANGE:** `Cucumber::Core::Test::Runner` now takes a retry policy instead of a maximum number of attempts, and
+asks it whether a finished test case will be retried. Its optional constructor arguments became keyword arguments. See upgrading notes for
+[20.0.0.md](upgrading_notes/20.0.0.md#upgrading-to-cucumber-core-2000), for full changes
+
+### Fixed
+- The `TestCaseFinished` envelope is now emitted before the `test_case_finished` event, so a test case retried from
+that event no longer has its envelope emitted out of order and with the started id of the retry
 
 ## [19.0.0] - 2026-08-19
 ### Added

lib/cucumber/core/test/no_retries.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+module Cucumber
+  module Core
+    module Test
+      class NoRetries
+        def will_be_retried?(_test_case, _result) = false
+      end
+    end
+  end
+end

lib/cucumber/core/test/runner.rb

@@ -2,6 +2,7 @@
 
 require 'cucumber/messages/helpers/test_step_result_comparator'
 
+require_relative 'no_retries'
 require_relative 'timer'
 require 'cucumber/messages'
 
@@ -14,11 +15,13 @@ class Runner
         attr_reader :event_bus, :running_test_case, :running_test_step, :id_generator
         private :event_bus, :running_test_case, :running_test_step, :id_generator
 
-        def initialize(event_bus, id_generator = Cucumber::Messages::Helpers::IdGenerator::UUID.new, backtrace_filter = nil, max_attempts = 1)
+        # @param retry_policy [#will_be_retried?] asked, once a test case has finished, whether it is going to be
+        #   run again. It receives the test case and its result. By default, no test case is ever reported as retried.
+        def initialize(event_bus, id_generator: Cucumber::Messages::Helpers::IdGenerator::UUID.new, backtrace_filter: nil, retry_policy: NoRetries.new)
           @event_bus = event_bus
           @id_generator = id_generator
           @backtrace_filter = backtrace_filter
-          @max_attempts = max_attempts
+          @retry_policy = retry_policy
           @current_test_case = nil
         end
 
@@ -34,8 +37,8 @@ def test_case(test_case, &descend)
           descend.call(self)
 
           result = calculate_test_case_result(test_case)
+          event_bus.envelope(to_test_case_finished_envelope(test_case, result))
           event_bus.test_case_finished(test_case, result)
-          event_bus.envelope(to_test_case_finished_envelope(result))
           self
         end
 
@@ -82,12 +85,12 @@ def to_test_case_started_envelope(test_case)
           )
         end
 
-        def to_test_case_finished_envelope(result)
+        def to_test_case_finished_envelope(test_case, result)
           Cucumber::Messages::Envelope.new(
             test_case_finished: Cucumber::Messages::TestCaseFinished.new(
               test_case_started_id: @current_test_case_started_id,
               timestamp: time_to_timestamp(Time.now),
-              will_be_retried: result.failed? && (@attempt < @max_attempts)
+              will_be_retried: @retry_policy.will_be_retried?(test_case, result)
             )
           )
         end

spec/cucumber/core/test/runner_spec.rb

@@ -4,6 +4,7 @@
 require 'cucumber/core/test/hook_step'
 require 'cucumber/core/test/case'
 require 'cucumber/core/test/step'
+require 'cucumber/core/event_bus'
 
 require 'support/duration_matcher'
 
@@ -520,4 +521,72 @@
       test_case.describe_to(runner)
     end
   end
+
+  context 'when emitting envelopes' do
+    let(:event_bus) { Cucumber::Core::EventBus.new }
+    let(:test_steps) { [failing_step] }
+    let(:envelopes) { [] }
+
+    before { event_bus.on(:envelope) { |event| envelopes << event.envelope } }
+
+    context 'when the test case is run again from the `test_case_finished` event' do
+      before do
+        retried = false
+        event_bus.on(:test_case_finished) do
+          next if retried
+
+          retried = true
+          test_case.describe_to(runner)
+        end
+      end
+
+      it 'references the `test_case_started` id of the same attempt in each `test_case_finished` envelope' do
+        test_case.describe_to(runner)
+
+        expect(envelopes.filter_map(&:test_case_finished).map(&:test_case_started_id)).to eq(envelopes.filter_map(&:test_case_started).map(&:id))
+      end
+    end
+
+    context 'when no retry policy is given' do
+      it 'reports a failed test case will not be retried' do
+        test_case.describe_to(runner)
+
+        expect(envelopes.filter_map(&:test_case_finished).map(&:will_be_retried)).to eq([false])
+      end
+
+      it 'asks a `NoRetries` instance whether the test case will be retried' do
+        no_retries = Cucumber::Core::Test::NoRetries.new
+        allow(Cucumber::Core::Test::NoRetries).to receive(:new).and_return(no_retries)
+        expect(no_retries).to receive(:will_be_retried?).with(test_case, an_instance_of(Cucumber::Core::Test::Result::Failed)).and_call_original
+
+        test_case.describe_to(runner)
+      end
+    end
+
+    context 'when a failed test case is not run again although attempts remain' do
+      let(:retry_policy) { double }
+      let(:runner) { described_class.new(event_bus, retry_policy: retry_policy) }
+
+      before { allow(retry_policy).to receive(:will_be_retried?).with(test_case, an_instance_of(Cucumber::Core::Test::Result::Failed)).and_return(false) }
+
+      it 'reports the test case will not be retried' do
+        test_case.describe_to(runner)
+
+        expect(envelopes.filter_map(&:test_case_finished).map(&:will_be_retried)).to eq([false])
+      end
+    end
+
+    context 'when the retry policy reports the failed test case will be retried' do
+      let(:retry_policy) { double }
+      let(:runner) { described_class.new(event_bus, retry_policy: retry_policy) }
+
+      before { allow(retry_policy).to receive(:will_be_retried?).with(test_case, an_instance_of(Cucumber::Core::Test::Result::Failed)).and_return(true) }
+
+      it 'reports the test case will be retried' do
+        test_case.describe_to(runner)
+
+        expect(envelopes.filter_map(&:test_case_finished).map(&:will_be_retried)).to eq([true])
+      end
+    end
+  end
 end

upgrading_notes/20.0.0.md

@@ -0,0 +1,47 @@
+# Upgrading to cucumber-core 20.0.0
+
+## Retry policy for the Test Runner
+
+In cucumber-core 19.0.0 the `Cucumber::Core::Test::Runner` started to emit the `TestCaseFinished` envelope itself, and
+decided the value of its `will_be_retried` field from a maximum number of attempts passed to its constructor. The
+runner cannot know that on its own: whether a test case is retried is decided by whoever runs it again (in
+cucumber-ruby, the retry filter, which also honours `--retry-total`), so the two got out of sync.
+
+The runner now takes a retry policy instead, and its optional constructor arguments (`id_generator`, `backtrace_filter`
+and `retry_policy`) became keyword arguments. The policy is asked, every time a test case finishes, whether that test
+case will be run again. It has to answer without side effects, as it is asked before the
+`test_case_finished` event is broadcast.
+
+### Before cucumber-core 20.0.0
+
+```ruby
+runner = Cucumber::Core::Test::Runner.new(event_bus, id_generator, backtrace_filter, max_attempts)
+```
+
+### With cucumber-core 20.0.0
+
+```ruby
+module MyProject
+  # For the policy cucumber-ruby uses, see lib/cucumber/retry_policy.rb in cucumber-ruby
+  class RetryPolicy
+    def will_be_retried?(test_case, result)
+      result.failed? && attempts_so_far(test_case) < max_attempts
+    end
+  end
+end
+
+runner = Cucumber::Core::Test::Runner.new(
+  event_bus,
+  id_generator: id_generator,
+  backtrace_filter: backtrace_filter,
+  retry_policy: MyProject::RetryPolicy.new
+)
+```
+
+When no retry policy is given, the runner uses `Cucumber::Core::Test::NoRetries`, which never reports a test case as
+retried. `Cucumber::Core::Test::Runner.new(event_bus)` keeps working as before.
+
+## Envelope ordering
+
+The runner now emits the `TestCaseFinished` envelope before it broadcasts the `test_case_finished` event. Handlers of
+that event that run the test case again therefore see the envelope of the previous attempt already emitted.