Skip to content

GD-655: Provide capability for visual regression testing #1065

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
MikeSchulze wants to merge 3 commits into
base: master
Choose a base branch
from

GD-655: Provide capability for visual regression testing

49b9a91
Select commit
Loading
Failed to load commit list.
Sign in for the full log view
Draft

GD-655: Provide capability for visual regression testing #1065

GD-655: Provide capability for visual regression testing
49b9a91
Select commit
Loading
Failed to load commit list.
GitHub Actions / report_gdUnit4_Godot4.5-stable succeeded Jan 25, 2026 in 0s

1443 passed, 4 failed and 28 skipped

Tests failed

Report Passed Failed Skipped Time
./home/runner/work/gdUnit4/gdUnit4/reports/report_1/results.xml 1443 ✅ 4 ❌ 28 ⚪ 0ms

❌ ./home/runner/work/gdUnit4/gdUnit4/reports/report_1/results.xml

1475 tests were completed in 0ms with 1443 passed, 4 failed and 28 skipped.

Test suite Passed Failed Skipped Time
AnyArgumentMatcherTest 3 ✅ 23ms
AnyBuildInTypeArgumentMatcherTest 13 ✅ 106ms
AnyClazzArgumentMatcherTest 4 ✅ 36ms
BaseTest 1 ✅ 9ms
BaseTest 1 ✅ 9ms
CallBackValueProviderTest 2 ✅ 12ms
ChainedArgumentMatcherTest 3 ✅ 21ms
CmdArgumentParserTest 5 ✅ 39ms
CmdCommandHandlerTest 13 ✅ 124ms
CmdCommandTest 2 ✅ 15ms
CmdOptionsTest 4 ✅ 27ms
CmdOptionTest 8 ✅ 52ms
CustomArgumentMatcherTest 1 ✅ 38ms
DiscoverExampleTestSuite 12 ✅ 108ms
ExtendedTest 1 ✅ 17ms
ExtendedTest 1 ✅ 17ms
ExtendsExtendedTest 1 ✅ 24ms
ExtendsExtendedTest 1 ✅ 24ms
flaky_test 10 ⚪ 0ms
GdArrayToolsTest 77 ✅ 709ms
GdDefaultValueDecoderTest 84 ✅ 2s
GdDiffToolTest 4 ✅ 62ms
GdFunctionArgumentTest 6 ✅ 49ms
GdFunctionDescriptorTest 11 ✅ 140ms
GdMarkDownReaderTest 9 ✅ 2s
GdMockFunctionDoublerTest 7 ✅ 69ms
GdObjectsTest 33 ✅ 881ms
GdScriptParserTest 36 ✅ 312ms
GdScriptParsingFunctionDescriptors 18 ✅ 218ms
GdSpyFunctionDoublerTest 12 ✅ 114ms
GdUnit4CSharpApiLoaderTest 8 ⚪ 0ms
GdUnit4VersionTest 7 ✅ 5s
GdUnitArgumentMatchersTest 1 ✅ 8ms
GdUnitArrayAssertImplTest 42 ✅ 269ms
GdUnitAssertImplTest 11 ✅ 187ms
GdUnitAwaiterTest 4 ✅ 516ms
GdUnitBoolAssertImplTest 11 ✅ 66ms
GdUnitCommandHandlerTest 1 ✅ 53ms
GdUnitConsoleTestReporterTest 3 ✅ 31ms
GdUnitCsiMessageWriterTest 4 ✅ 34ms
GdUnitDictionaryAssertImplTest 22 ✅ 120ms
GdUnitEventTest 1 ✅ 10ms
GdUnitExecutionContextTest 7 ✅ 201ms
GdUnitExpressionRunnerTest 8 ✅ 135ms
GdUnitFailureAssertImplTest 17 ✅ 103ms
GdUnitFileAccessTest 12 ✅ 135ms
GdUnitFloatAssertImplTest 22 ✅ 103ms
GdUnitFuncAssertImplTest 15 ✅ 5s
GdUnitFunctionDoublerBuilderTest 5 ✅ 50ms
GdUnitFuncValueExtractorTest 5 ✅ 37ms
GdUnitFuzzerValueInjectionTest 12 ✅ 6s
GdUnitGodotErrorAssertImplTest 6 ✅ 3 ⚪ 64ms
GdUnitGodotErrorWithAssertTest 2 ⚪ 0ms
GdUnitGuidTest 4 ✅ 396ms
GdUnitIntAssertImplTest 22 ✅ 2s
GdUnitMockBuilderTest 4 ✅ 339ms
GdUnitMockerTest 68 ✅ 16s
GdUnitObjectAssertImplTest 14 ✅ 108ms
GdUnitPackedArrayAssertTest 131 ✅ 2s
GdUnitPatcherTest 9 ✅ 45ms
GdUnitResultAssertImplTest 11 ✅ 54ms
GdUnitResultTest 4 ✅ 37ms
GdUnitRunnerConfigTest 6 ✅ 56ms
GdUnitScanTestSuitePerformanceTest 1 ✅ 19ms
GdUnitScanTestSuiteTest 4 ✅ 24ms
GdUnitSceneRunnerInputEventTest 38 ✅ 12s
GdUnitSceneRunnerTest 30 ✅ 1 ❌ 21s
GdUnitSettingsTest 5 ✅ 91ms
GdUnitShortcutTest 1 ✅ 10ms
GdUnitSignalAssertImplTest 23 ✅ 2s
GdUnitSignalAwaiterTest 3 ✅ 2s
GdUnitSignalMonitorTest 3 ✅ 79ms
GdUnitSingletonTest 2 ✅ 17ms
GdUnitSpyBuilderTest 2 ✅ 176ms
GdUnitSpyCallableTest 22 ✅ 817ms
GdUnitSpyTest 35 ✅ 2s
GdUnitStringAssertImplTest 27 ✅ 160ms
GdUnitTcpServerIntegrationTest 2 ✅ 461ms
GdUnitTestCaseTest 1 ✅ 10ms
GdUnitTestCaseTimeoutTest 6 ✅ 19s
GdUnitTestCaseTimingTest 3 ✅ 26s
GdUnitTestCIRunnerTest 6 ✅ 116ms
GdUnitTestDiscovererTest 6 ✅ 1 ❌ 1 ⚪ 115ms
GdUnitTestDiscoverGuardTest 7 ✅ 1 ⚪ 180ms
GdUnitTestDiscoverSinkTest 1 ✅ 11ms
GdUnitTestEventSerdeTest 6 ✅ 48ms
GdUnitTestParameterSetResolverTest 23 ✅ 267ms
GdUnitTestResourceLoaderTest 1 ✅ 55ms
GdUnitTestSessionHookServiceTest 9 ✅ 99ms
GdUnitTestSuiteBuilderTest 2 ✅ 54ms
GdUnitTestSuiteExecutorTest 19 ✅ 4s
GdUnitTestSuitePerformanceTest 1 ✅ 1s
GdUnitTestSuiteScannerTest 18 ✅ 272ms
GdUnitTestSuiteTemplateTest 5 ✅ 51ms
GdUnitTestSuiteTest 3 ✅ 1 ❌ 1 ⚪ 33ms
GdUnitToolsTest 2 ✅ 18ms
GdUnitTupleTest 3 ✅ 20ms
GdUnitUiAssertImplTest 2 ✅ 1 ❌ 833ms
GdUnitUiToolsTest 4 ✅ 27ms
GdUnitUpdateTest 1 ✅ 8ms
GdUnitVariantAssertThatTest 4 ✅ 29ms
GdUnitVectorAssertImplTest 106 ✅ 2s
GeneratedPersonTest 1 ✅ 7ms
GodotGdErrorMonitorTest 2 ✅ 1 ⚪ 30ms
InspectorProgressBarTest 7 ✅ 264ms
InspectorTreeMainPanelPerformanceTest 40 ✅ 237ms
InspectorTreeMainPanelTest 22 ✅ 1 ⚪ 2s
JUnitXmlReportWriterTest 2 ✅ 20ms
LocalTimeTest 6 ✅ 292ms
ParameterizedTestCaseTest 49 ✅ 662ms
StringFuzzerTest 5 ✅ 287ms
TestSuiteTemplateTest 3 ✅ 242ms
XmlElementTest 7 ✅ 71ms

❌ GdUnitSceneRunnerTest

❌ test_is_equal_screenshot
	FAILURE: FAILED: res://addons/gdUnit4/test/core/GdUnitSceneRunnerTest.gd:495

❌ GdUnitTestDiscovererTest

❌ test_scan_test_directories
	FAILURE: FAILED: res://addons/gdUnit4/test/core/discovery/GdUnitTestDiscovererTest.gd:124

❌ GdUnitTestSuiteTest

❌ test_flaky_success
	FAILURE: FAILED: res://addons/gdUnit4/test/GdUnitTestSuiteTest.gd:96

❌ GdUnitUiAssertImplTest

❌ test_is_equal_screenshot_strict_with_alpha
	FAILURE: FAILED: res://addons/gdUnit4/test/asserts/GdUnitUiAssertImplTest.gd:26

Annotations

Check failure on line 0 in ./home/runner/work/gdUnit4/gdUnit4/reports/report_1/results.xml

See this annotation in the file changed.

@github-actions github-actions / report_gdUnit4_Godot4.5-stable

GdUnitSceneRunnerTest ► test_is_equal_screenshot 

Failed test found in:
  ./home/runner/work/gdUnit4/gdUnit4/reports/report_1/results.xml
Error:
  FAILURE: FAILED: res://addons/gdUnit4/test/core/GdUnitSceneRunnerTest.gd:495
Raw output 


		Expecting same component size:
		 (256, 256)
		 but was
		 (105, 105)

Check failure on line 0 in ./home/runner/work/gdUnit4/gdUnit4/reports/report_1/results.xml

See this annotation in the file changed.

@github-actions github-actions / report_gdUnit4_Godot4.5-stable

GdUnitTestDiscovererTest ► test_scan_test_directories 

Failed test found in:
  ./home/runner/work/gdUnit4/gdUnit4/reports/report_1/results.xml
Error:
  FAILURE: FAILED: res://addons/gdUnit4/test/core/discovery/GdUnitTestDiscovererTest.gd:124
Raw output 

Expecting contains exactly elements:
 'PackedStringArray(["res://addons", "res://assets", "res://test_img"])'
 do contains (in same order)
 '["res://addons", "res://assets"]'
but some elements where not expected:
 '["res://test_img"]'

Check failure on line 0 in ./home/runner/work/gdUnit4/gdUnit4/reports/report_1/results.xml

See this annotation in the file changed.

@github-actions github-actions / report_gdUnit4_Godot4.5-stable

GdUnitTestSuiteTest ► test_flaky_success 

Failed test found in:
  ./home/runner/work/gdUnit4/gdUnit4/reports/report_1/results.xml
Error:
  FAILURE: FAILED: res://addons/gdUnit4/test/GdUnitTestSuiteTest.gd:96
Raw output 

failure 1: at retry 1

Check failure on line 0 in ./home/runner/work/gdUnit4/gdUnit4/reports/report_1/results.xml

See this annotation in the file changed.

@github-actions github-actions / report_gdUnit4_Godot4.5-stable

GdUnitUiAssertImplTest ► test_is_equal_screenshot_strict_with_alpha 

Failed test found in:
  ./home/runner/work/gdUnit4/gdUnit4/reports/report_1/results.xml
Error:
  FAILURE: FAILED: res://addons/gdUnit4/test/asserts/GdUnitUiAssertImplTest.gd:26
Raw output 

Visual regression detected in 'Control864':
		
		[font_size=16]Metrics:[/font_size]
			Component: <Control864>
			Node Path: /root/GdUnitUiAssertImplTest/@Control@864
			Resolution: (512, 512) pixels
			Config: strict
				- pixel tolerance: 0.0%
				- difference threshold: 0.0%
				- ignore alpha: true
			- 89828.0 pixels of total 262144 differs (34.2% of total)
			- Avg color distance: 1.81 (threshold: 0.0)
			
		[font_size=16]Comparision:[/font_size]
			
			[center]Expected[/center][center]Actual[/center][center]Difference[/center]
			[img]res://reports/2920957654_test_is_equal_screenshot_strict_with_alpha/visual_regression/Control864-3795274702834/Control864_expected.png[/img][img]res://reports/2920957654_test_is_equal_screenshot_strict_with_alpha/visual_regression/Control864-3795274702834/Control864_current.png[/img][img]res://reports/2920957654_test_is_equal_screenshot_strict_with_alpha/visual_regression/Control864-3795274702834/Control864_diff.png[/img]
View more details on GitHub Actions