Reddit allows roughly one request per minute per IP on unauthenticated
.rss, but the app fired six separate subreddit requests, retried 429s
every couple of minutes, and only paced hosts within a single process.
Concurrent page/API/cron requests stampeded providers.
- Combine all due subreddits into one /r/a+b+c/.rss request and attribute
items back to each subreddit by link; fall back to individual fetches
when a banned/private subreddit poisons the combined feed.
- Single-flight refresh via data/refresh.lock so concurrent callers serve
the cache instead of refetching.
- Enforce throttled-host spacing across processes (feeds.hostLastAttempt)
in addition to the per-refresh cap; add www.youtube.com to throttled hosts.
- Honor 429 Retry-After / x-ratelimit-reset and back off exponentially
(fail_count doubling, capped at error_retry_cap, plus jitter), reset on
success. New feeds columns fail_count and next_retry_at.
- Jitter the healthy TTL so feeds don't all expire at once.
- Detect add-feed 404 by status instead of matching the error string.
- Update README (semantics, data model, config) and default user agent.
Harwinder Sekhon
397ee1a413