Eliminate Reddit/YouTube 429s with efficient request handling

Reddit allows roughly one request per minute per IP on unauthenticated
.rss, but the app fired six separate subreddit requests, retried 429s
every couple of minutes, and only paced hosts within a single process.
Concurrent page/API/cron requests stampeded providers.

- Combine all due subreddits into one /r/a+b+c/.rss request and attribute
  items back to each subreddit by link; fall back to individual fetches
  when a banned/private subreddit poisons the combined feed.
- Single-flight refresh via data/refresh.lock so concurrent callers serve
  the cache instead of refetching.
- Enforce throttled-host spacing across processes (feeds.hostLastAttempt)
  in addition to the per-refresh cap; add www.youtube.com to throttled hosts.
- Honor 429 Retry-After / x-ratelimit-reset and back off exponentially
  (fail_count doubling, capped at error_retry_cap, plus jitter), reset on
  success. New feeds columns fail_count and next_retry_at.
- Jitter the healthy TTL so feeds don't all expire at once.
- Detect add-feed 404 by status instead of matching the error string.
- Update README (semantics, data model, config) and default user agent.
This commit is contained in:
Harwinder Sekhon
2026-10-07 20:34:16 -04:00
parent a6c27404e5
commit 397ee1a413
11 changed files with 513 additions and 80 deletions
+21 -6
View File
@@ -8,13 +8,27 @@ return [
'cache_ttl' => 1800,
// Shorter retry interval for feeds whose last fetch failed (e.g. rate limited).
'error_retry_ttl' => 120,
// Backoff grows from here (doubling per consecutive failure) and is capped
// by error_retry_cap; a server Retry-After hint always wins.
'error_retry_ttl' => 300,
// Minimum milliseconds between requests to the same host.
// Maximum retry backoff for a failing feed (2 hours).
'error_retry_cap' => 7200,
// Random extra seconds added to the healthy TTL so feeds do not all expire
// (and get fetched) at the same moment.
'ttl_jitter' => 300,
// Minimum milliseconds between requests to the same host within one refresh.
'host_interval_ms' => 1000,
// Hosts with aggressive rate limits: only fetch this many of their feeds per refresh.
'throttled_hosts' => ['www.reddit.com'],
// Hosts with aggressive rate limits: minimum seconds between requests to
// that host (enforced across processes via the cache DB), and at most
// throttled_host_max_per_refresh feeds of it per refresh.
'throttled_hosts' => [
'www.reddit.com' => 60,
'www.youtube.com' => 15,
],
'throttled_host_max_per_refresh' => 1,
// Number of items per page.
@@ -33,8 +47,9 @@ return [
'request_timeout' => 15,
'connect_timeout' => 10,
// Identify the client to feed providers (important for Reddit).
'user_agent' => 'RSSFeed/1.0 (+https://localhost; contact: admin@localhost)',
// Identify the client to feed providers (important for Reddit). Reddit asks
// for a descriptive, unique agent — set a real contact before deploying.
'user_agent' => 'RSSFeed/1.0 (self-hosted dashboard; contact: admin@example.com)',
// Password for manage.php (manual subreddit/channel configuration).
// Accepts a plain password or a password_hash() value; empty disables the