Reddit allows roughly one request per minute per IP on unauthenticated
.rss, but the app fired six separate subreddit requests, retried 429s
every couple of minutes, and only paced hosts within a single process.
Concurrent page/API/cron requests stampeded providers.
- Combine all due subreddits into one /r/a+b+c/.rss request and attribute
items back to each subreddit by link; fall back to individual fetches
when a banned/private subreddit poisons the combined feed.
- Single-flight refresh via data/refresh.lock so concurrent callers serve
the cache instead of refetching.
- Enforce throttled-host spacing across processes (feeds.hostLastAttempt)
in addition to the per-refresh cap; add www.youtube.com to throttled hosts.
- Honor 429 Retry-After / x-ratelimit-reset and back off exponentially
(fail_count doubling, capped at error_retry_cap, plus jitter), reset on
success. New feeds columns fail_count and next_retry_at.
- Jitter the healthy TTL so feeds don't all expire at once.
- Detect add-feed 404 by status instead of matching the error string.
- Update README (semantics, data model, config) and default user agent.
Browser-based feed selection layered on top of config/feeds.php:
- manage.php (password + CSRF gated, session based): add subreddits
(name / r/name / URL) and YouTube channels (@handle / UC id / URL,
handle resolved to channel_id once, youtube.com only); enable/disable
toggles; remove manual feeds; config feeds are hidden, not deleted.
- New feeds are validated with one immediate fetch (bypassing the
per-refresh host cap): HTTP 404 rejects the entry, 429 keeps it for
a later refresh. YouTube labels adopt the real channel title.
- feeds table gains origin (config|manual) and enabled columns, with
a guarded ALTER migration for existing databases.
- Only enabled feeds are refreshed and served: getItems/countItems
filter by feed_url in enabled feeds, so deselected feeds' cached
stories are hidden, not deleted; re-adding a URL re-enables it.
- Dashboard header gets a Manage link; manage page styled with the
existing design tokens; README documents the feature.
Harwinder Sekhon