-
Notifications
You must be signed in to change notification settings - Fork 0
feat: forward t.Logf() output to testing.T on property test failures #45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: forward t.Logf() output to testing.T on property test failures #45
Conversation
Code ReviewI've reviewed this PR and the implementation looks solid overall. Here are my findings: ✅ Positives
🔍 Code Qualityproperty.go:20 - The logs []string // Captured log messages (unused when logPassthrough is true)Since you're always using passthrough mode for the final replay and discarding logs otherwise, this field serves no purpose. property.go:64-65 - Type assertion could use a helper variable for clarity: if helper, ok := t.testingT.(interface{ Helper() }); ok {
helper.Helper()
}Minor: the interface could be extracted to a type for reusability (it's used in 3 places), but not critical. property.go:274-276 - The warning for non-deterministic tests is good defensive programming. Consider if there are legitimate cases where this could trigger false positives (e.g., time-based generators without seeds). 🤔 Potential Issuesgenerator.go:29 - Adding Test determinism: property_logf_test.go uses fixed seeds which is good for reproducibility, but confirm the shrinking behavior is deterministic enough that tests won't flake. Performance & SecurityNo concerns. The change is minimal overhead (one boolean flag, one conditional check per log call). SummaryThis is well-implemented functionality that provides genuine debugging value. The only actionable feedback is removing the unused Recommendation: Approve with minor cleanup suggested. |
Code ReviewOverall this is a well-implemented feature with clean design. I have a few observations: Issuesproperty.go:20 - The logs []string // Captured log messages (unused when logPassthrough is true)Since you're always using passthrough mode for the final replay and discarding logs otherwise, this field serves no purpose. Observationsproperty.go:64-65 - The type assertion for Helper() could be cleaner with an extracted helper variable, though this is used in 3 places so extracting it to a named type might help with consistency. Not critical. generator.go:29 - Adding property.go:274-276 - The non-deterministic warning is good defensive programming. Consider whether legitimate cases might trigger false positives (time-based generators without explicit seeds). Test CoverageThe test coverage in SummaryClean implementation that provides genuine value. Only actionable item is removing the unused |
Additional Review NotesTwo previous reviews already covered the main points well. One actionable item to address: property.go:20 - Remove the unused The rest looks good - the implementation is clean, test coverage is solid, and the passthrough design avoids memory overhead from capturing/discarding logs on successful iterations. |
Review SummaryThe previous reviews have correctly identified the main issue - the unused Beyond that, the implementation is solid:
Recommendation: Remove the unused field and this is ready to merge. |
ReviewThe existing review comments have thoroughly covered this PR. I'll only add: Action Required:
Everything Else Looks Good:
The implementation is solid and ready to merge once the unused field is removed. |
Property tests now capture logs from t.Logf() and include them in the error output when a test fails. This gives users debugging capability while avoiding log spam from successful iterations. Changes: - Add testingT field and logs slice to specta.T struct - Modify T.Logf() to capture messages instead of being a no-op - Pass outer testing.T when creating T instances in Property() - Forward captured logs in reportFailure() for shrunk counterexample - Add t.Helper() to Draw() function for better stack traces - Add comprehensive tests for log forwarding behavior Logs are only shown for the final shrunk failing iteration, which provides useful debugging context without cluttering test output.
Changed log forwarding to call t.Logf() directly during the final replay instead of capturing logs and printing them in the error message. This provides better integration with Go's testing infrastructure and correct file:line attribution using the Helper() mechanism. Changes: - Add logPassthrough flag to T struct - Update T.Logf() to forward directly to testingT.Logf() when enabled - Call testingT.Helper() before forwarding to get correct line attribution - Enable logPassthrough during final replay in reportFailure() - Add sanity check for replay failures (non-deterministic tests) - Update tests to check spy.Logs instead of error message contents Benefits: - Logs appear as regular log lines with correct file:line (user code, not property.go) - More idiomatic Go test output - Better IDE and test runner integration - No misleading line numbers
Simplified the test to use a single DeepEqual assertion that checks both the exact logs and their order, rather than multiple individual assertions.
Updated tests to explicitly check that logged values are from the shrunk iteration, not the original failure: - First test checks x = 5 (minimal failing value for x >= 5) - Shrinking test checks x = -1 (minimal negative value in range) - Renamed tests to clarify they verify shrinking behavior - Added Range(-1000, 1000) to ensure predictable shrinking to -1
Removed the first test which was redundant with the shrinking test. Now we have three focused tests: - logs not shown on success - logs show shrunk values (verifies shrinking behavior) - multiple logs captured in order
Since TestingT interface already requires Helper(), the type assertion checks are redundant. Simplified to direct calls: - T.Logf(): t.testingT.Helper() - Property(): t.Helper() - reportFailure(): t.Helper() All tests pass with the simplified code.
38526a4 to
9e098a9
Compare
Review SummaryMultiple reviews have already identified the key issue: Action Required:
Otherwise looks good:
Ready to merge after removing the unused field. |
- Remove logs []string field from T struct (no longer captured, only forwarded) - Regenerate API documentation to reflect Logf() behavior change
Code ReviewThis PR has been significantly improved since the earlier reviews. The unused ✅ StrengthsClean passthrough design (property.go:20, 56-67): The Correct line attribution (property.go:63, generator.go:29): Using Comprehensive test coverage (property_logf_test.go): The test suite covers the key scenarios - success (no logs), failure with shrinking (logs show shrunk values), and multiple logs in order. Well-designed tests. Defensive check (property.go:265-267): The sanity check for non-deterministic behavior is good defensive programming that could help catch subtle bugs. Minor ObservationsType assertion pattern (property.go:63): The inline type assertion for Draw Helper usage (generator.go:29): Adding SummaryThis is a well-implemented feature that provides genuine debugging value. The code is clean, well-tested, and follows Go best practices. The earlier feedback has been addressed. Recommendation: Approve and merge. |
Added two Example functions showing how pt.Logf() works in property tests: - Example_propertyLogging: shows logs appearing for shrunk failures - Example_propertyLoggingMultipleValues: shows execution trace with multiple logs The examples demonstrate: - Logs only appear for failing tests (no spam) - Logs show values from the shrunk counterexample - Multiple log calls appear in order
Code ReviewThe implementation looks clean and well-tested. The unused Key Strengths
Minor NoteThe defensive check at property.go:265-267 for non-deterministic replay is good practice. Just be aware it could theoretically trigger false positives with time-based generators without explicit seeds, though unlikely in practice. Recommendation: Approve and merge. The feature provides genuine debugging value without introducing technical debt. |
Property tests now capture logs from t.Logf() and include them in the
error output when a test fails. This gives users debugging capability
while avoiding log spam from successful iterations.
Changes:
Logs are only shown for the final shrunk failing iteration, which
provides useful debugging context without cluttering test output.