-
Notifications
You must be signed in to change notification settings - Fork 48
Fix nil pointer dereference in isEnvelopeInvalid #1447
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
The Close() method had a race condition between decrementing the reference count and reading it for logging. This could cause incorrect values to be logged when the reference count changed between the Add(-1) and Load() operations. Fixed by capturing the result of Add(-1) and using it directly, eliminating the race window. Note: This fix only addresses the logging race. The use-after-free concern is already prevented by the cache implementation which removes entries from its map before calling Close(), ensuring no new references can be obtained once eviction begins.
Modified Close() to return bool indicating success, and added warnings when cache eviction fails due to active references. This makes memory leaks from orphaned keys visible in logs. Note: This doesn't prevent the leak, but at least makes it observable. A proper fix would require modifying the cache library to support fallible eviction callbacks.
When the cache evicts a key that still has active references, it removes the key from the cache map before checking if Close() succeeds. This creates orphaned keys that leak memory. This fix: - Makes cachedCryptoKey.Close() return bool to indicate success - Tracks orphaned keys in a separate list when eviction fails - Periodically attempts to clean up orphaned keys - Ensures orphaned keys are cleaned up on cache close This is a minimal change that doesn't require modifying the third-party cache library. Co-Authored-By: Claude <[email protected]>
Background cleanup in GetOrLoad would introduce variable latency in the hot path. Since orphaned keys should be rare (only when a key is evicted while actively being used), we now only clean them up during cache Close(). This trades a small memory leak for consistent performance. Co-Authored-By: Claude <[email protected]>
- Runs cleanup goroutine every 30 seconds to free orphaned keys - Prevents memory accumulation in long-running services - Keeps cleanup out of the hot path for consistent performance - Properly shuts down cleanup on cache close with sync.Once This ensures orphaned keys (those evicted while still referenced) are eventually freed without waiting for cache close. Co-Authored-By: Claude <[email protected]>
- Swap orphan list under lock to minimize critical section - Process Close() operations outside the lock - Allows new orphans to be added during cleanup - More efficient batch processing This reduces lock contention between eviction callbacks and the cleanup goroutine. Co-Authored-By: Claude <[email protected]>
- Add periods to end of comments (godot) - Fix import formatting (gci) - Remove trailing space Co-Authored-By: Claude <[email protected]>
- Remove trailing spaces throughout - Fix gofmt formatting - Add newline at end of file Co-Authored-By: Claude <[email protected]>
- Add periods to bullet points in comments - Remove extra blank line at end of file Co-Authored-By: Claude <[email protected]>
- Remove cache eviction orphan issue (fixed) - Remove reference counting race issue (fixed) - Renumber remaining issues - Update priority list to reflect only unfixed issues The REMEDIATION.md now only contains issues that still need to be addressed. Co-Authored-By: Claude <[email protected]>
The default cache now uses bounded eviction policies, but users who explicitly choose 'simple' cache still get unbounded growth. Co-Authored-By: Claude <[email protected]>
Simple cache is intentionally unbounded for ephemeral environments like AWS Lambda where: - Process lifetime is short - Memory is reset between invocations - Eviction overhead is wasteful - Maximum performance is desired This is a deliberate architectural choice, not a flaw. Co-Authored-By: Claude <[email protected]>
Replace unbounded goroutine creation with single cleanup processor to prevent memory exhaustion during cache eviction bursts. ## Problem - Session cache created unlimited goroutines on eviction via `go v.encryption.Remove()` - Under memory pressure, mass evictions could create thousands of goroutines - Each goroutine consumes ~8KB stack + overhead, leading to memory explosion - Unpredictable performance impact, especially problematic for Lambda environments ## Solution - Single background goroutine processes cleanup requests sequentially - 10,000-item buffered channel handles burst evictions gracefully - Falls back to synchronous cleanup when buffer full (no goroutine creation) - Global processor shared across all session caches ## Performance Benefits - **Memory**: 99%+ reduction (8MB+ → 88KB for 1000 concurrent evictions) - **CPU**: O(n) → O(1) goroutine allocation, reduced context switching - **Lambda**: Predictable ~88KB overhead vs variable goroutine creation - **Servers**: Bounded resource usage regardless of eviction patterns ## Implementation - `sessionCleanupProcessor` with buffered channel replaces direct goroutine spawning - Sequential processing eliminates concurrency complexity - Comprehensive test coverage for leak prevention and burst handling - Fully backward compatible - same cleanup semantics, no API changes 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Separates the nil check from the field access to prevent panic when envelopeEncryption receiver is nil. The original code had a race where e.Policy.ExpireKeyAfter could be accessed even when e == nil. Changes: - Add early return for nil receiver in isEnvelopeInvalid - Add comprehensive tests for nil pointer safety - Add tests for valid/invalid envelope scenarios - Update REMEDIATION.md to remove fixed issues 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
This issue has been fixed in the previous commit, so removing it from the remediation document as requested. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
This issue has been fixed in PR #1448, so removing it from the remediation document as requested. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Correct outdated security analysis based on modern understanding: - Modern CPUs provide hardware RNG (RDRAND/RDSEED) - Modern OS /dev/urandom doesn't block or fail on 'entropy exhaustion' - Go's crypto/rand failures indicate serious system issues, not entropy problems - Reclassify from critical security issue to design consideration The panic may actually be appropriate for genuine system failures. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Panic on random number generation failure is appropriate behavior for a cryptographic library when the system cannot provide random bytes. This represents a 'should never happen' scenario indicating a fundamentally broken system. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Add warning logging for metastore store failures while maintaining the existing flow logic. This provides observability into potential infrastructure issues (network, permissions) without changing the behavior of treating all errors as duplicate key scenarios. Changes: - Add warning log when metastore Store() fails - Maintain existing flow logic (error still 'intentionally ignored') - Update REMEDIATION.md to mark all issues as resolved 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Add resetGlobalSessionCleanupProcessor() for test isolation - Reset processor before and after each test that uses session cache - Add wait time for async cleanup operations to complete - Fix import formatting issues with gci
- Run gci on session_worker_pool.go, session.go, and envelope.go - Fix import ordering to satisfy linter requirements
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey, @jgowdy-godaddy. This looks like several different fixes bundled together. The nil pointer fix is great, but there's also a whole cache overhaul, new worker pool system, race condition fixes, etc.
These changes look solid but need to be split into separate PRs for proper review.
Also, the working notes (ORPHAN_KEY_FIX_SUMMARY.md, PR_DESCRIPTION.md, REMEDIATION.md) should be removed.
Summary
This PR fixes a critical nil pointer dereference bug in the
isEnvelopeInvalid
method that could cause production crashes when envelope encryption instances are unexpectedly nil.Issue Analysis
The Problem
The original code in
envelope.go:201
had a logical error in defensive programming:Critical flaw: The method checks
e == nil
but then immediately accessese.Policy.ExpireKeyAfter
. Ife
were actually nil, the field access would panic before the nil check could protect against it.Why This Matters
Solution
The Fix
Separate the nil check with an early return:
This ensures:
Testing
Added comprehensive tests covering:
Nil Pointer Safety:
true
without panickingValid Case Coverage:
false
true
true
Performance Impact
Additional Changes
Validation
Impact
This fix prevents production crashes and improves system reliability by properly handling edge cases in envelope encryption validation.
🤖 Generated with Claude Code