Skip to content

Add rate limiting functionality to Cache Warmer #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hpowernl merged 3 commits into from
Feb 15, 2026
Merged

Add rate limiting functionality to Cache Warmer #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2443bbb
Add rate limiting functionality to Cache Warmer
hpowernl Feb 15, 2026
7e2ebd7
Add max retries for HTTP 429 responses in Cache Warmer
hpowernl Feb 15, 2026
280f72a
Enhance README with 429 rate limit handling details
hpowernl Feb 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ A fast, native cache warmer written in Go for automatically warming caches via s
- 🗺️ **Sitemap Support**: Including nested sitemaps and .gz compression
- ⚙️ **Configurable**: TOML configuration file
- 📈 **Cache Flush Tracking**: Mark cache flushes for re-warming
- 🛡️ **429 Rate Limit Handling**: Adaptive concurrency reduction on HTTP 429, applies to both sitemap fetching and URL warming

## 📦 Installation

Expand Down Expand Up @@ -88,6 +89,11 @@ min_delay_ms = 50
retries = 2
retry_backoff_seconds = 1.0

# 429 rate limit handling (adaptive concurrency)
rate_limit_cooldown_seconds = 120
rate_limit_recover_after = 50
rate_limit_max_429_retries = 10

[load]
max_load = 2.0
check_interval_seconds = 2
Expand Down Expand Up @@ -189,6 +195,9 @@ All commands accept the `--config path/to/config.toml` flag.
- `min_delay_ms`: Minimum delay between requests (rate limiting)
- `retries`: Number of retry attempts on failures
- `retry_backoff_seconds`: Backoff multiplier for retries
- `rate_limit_cooldown_seconds`: Cooldown duration after 429 (default: 120)
- `rate_limit_recover_after`: Consecutive successes needed before increasing concurrency again (default: 50)
- `rate_limit_max_429_retries`: Max retries per URL on 429 before giving up (default: 10)

### [load]
- `max_load`: Maximum 1-minute load average (CPU protection)
Expand Down Expand Up @@ -246,6 +255,7 @@ CGO_ENABLED=1 go build -o cache-warmer cache-warmer.go
- Check firewall/IP whitelist
- Increase timeout in config
- Check server logs for rate limiting
- If you see many 429 errors: the rate limiter will automatically reduce concurrency; increase `rate_limit_max_429_retries` if URLs are being marked failed too quickly

### Warmer is slow

Expand Down
Loading
Loading