Eliminate Reddit/YouTube 429s with efficient request handling
Reddit allows roughly one request per minute per IP on unauthenticated .rss, but the app fired six separate subreddit requests, retried 429s every couple of minutes, and only paced hosts within a single process. Concurrent page/API/cron requests stampeded providers. - Combine all due subreddits into one /r/a+b+c/.rss request and attribute items back to each subreddit by link; fall back to individual fetches when a banned/private subreddit poisons the combined feed. - Single-flight refresh via data/refresh.lock so concurrent callers serve the cache instead of refetching. - Enforce throttled-host spacing across processes (feeds.hostLastAttempt) in addition to the per-refresh cap; add www.youtube.com to throttled hosts. - Honor 429 Retry-After / x-ratelimit-reset and back off exponentially (fail_count doubling, capped at error_retry_cap, plus jitter), reset on success. New feeds columns fail_count and next_retry_at. - Jitter the healthy TTL so feeds don't all expire at once. - Detect add-feed 404 by status instead of matching the error string. - Update README (semantics, data model, config) and default user agent.
This commit is contained in:
Harwinder Sekhon
@@ -203,11 +203,16 @@ index.php api/items.php
|
||||
FeedService::refreshStale() ← runs on both entry points, respects TTL
|
||||
│
|
||||
├─ FeedRepository::syncFeeds() ensure every configured feed has a row
|
||||
├─ FeedRepository::staleFeeds() feeds older than TTL (or error cooldown)
|
||||
└─ for each stale feed:
|
||||
├─ flock data/refresh.lock single-flight: concurrent callers skip
|
||||
├─ FeedRepository::staleFeeds() due feeds, minus retry backoff, jittered TTL
|
||||
├─ host pacing per-refresh cap + cross-process host interval
|
||||
├─ all stale Reddit feeds → ONE combined request (/r/a+b+c/.rss),
|
||||
│ items attributed back to each subreddit by link
|
||||
└─ for each other stale feed:
|
||||
FeedFetcher::fetch() Guzzle GET → laminas parse → normalize
|
||||
FeedRepository::upsertItems() INSERT OR IGNORE (dedup by sha256(link))
|
||||
FeedRepository::recordAttempt() success/error + timestamps
|
||||
FeedRepository::recordAttempt() success → clears error/fail_count;
|
||||
failure → fail_count++, next_retry_at
|
||||
|
||||
FeedService::page(source, page, pageSize, since) ← read path
|
||||
└─ FeedRepository::getItems()/countItems() SQL over the SQLite cache
|
||||
@@ -230,14 +235,27 @@ button when `has_more` is false.
|
||||
- `items` and `feeds` live in `data/feeds.sqlite` (WAL mode, outside the web root).
|
||||
- Only **enabled** feeds are refreshed and served — disabling one from the
|
||||
manage page stops both without deleting its cached items.
|
||||
- A feed is **stale** when `now - last_attempt_at >= cache_ttl`, or, if its last
|
||||
attempt errored, when `now - last_attempt_at >= error_retry_ttl + jitter`
|
||||
(jitter derived from `crc32(url)` so rate-limited feeds don't all retry at once).
|
||||
- Fetch failures are recorded (`last_error`) and **never delete cached items** —
|
||||
the dashboard keeps serving the last good data.
|
||||
- Reddit rate-limits hard. `throttled_hosts` caps how many feeds of a given host
|
||||
are fetched per refresh (`throttled_host_max_per_refresh`), and requests to the
|
||||
same host are spaced by `host_interval_ms`. Only 5xx responses retry in-request.
|
||||
- A feed is **due** when its retry backoff has elapsed (`next_retry_at`) and its
|
||||
jittered TTL has passed: `now - last_attempt_at >= cache_ttl + crc32(url) % ttl_jitter`
|
||||
(jitter spreads expiry so feeds don't all refresh at once).
|
||||
- **One refresh at a time.** `refreshStale()` takes a non-blocking lock on
|
||||
`data/refresh.lock`; concurrent page/API/cron requests serve the cache and
|
||||
return immediately instead of stampeding providers.
|
||||
- **Rate limits are respected.** Throttled hosts (`throttled_hosts`) are limited
|
||||
to `throttled_host_max_per_refresh` fetches per refresh *and* to a per-host
|
||||
minimum interval enforced across processes. The gap between requests to the
|
||||
same host is also paced in-request (`host_interval_ms`).
|
||||
- **Reddit is fetched as one request.** All due subreddits are combined into a
|
||||
single `reddit.com/r/a+b+c/.rss` call and the items are attributed back to each
|
||||
subreddit by their links. If that fails with 404/403 (a private or banned
|
||||
subreddit poisons the combined feed), the app falls back to individual fetches
|
||||
so the culprit is isolated.
|
||||
- **429 handling.** A `429` carries the server's own window (`Retry-After`, or
|
||||
Reddit's `x-ratelimit-reset`); the feed is parked until then. Other failures
|
||||
back off exponentially (`error_retry_ttl` doubling per consecutive failure, up
|
||||
to `error_retry_cap`, plus jitter). Success resets the counters.
|
||||
- Fetch failures are recorded (`last_error`, `fail_count`, `next_retry_at`) and
|
||||
**never delete cached items** — the dashboard keeps serving the last good data.
|
||||
|
||||
---
|
||||
|
||||
@@ -255,6 +273,9 @@ button when `has_more` is false.
|
||||
│ ├── bootstrap.php Wires config + classes together, returns the container
|
||||
│ ├── Database.php PDO SQLite connection + schema migration
|
||||
│ ├── FeedFetcher.php HTTP + parse + normalize a single feed
|
||||
│ ├── FeedFetchException.php Fetch failure carrying its HTTP status
|
||||
│ ├── RateLimitException.php HTTP 429, carries the server's retry window
|
||||
│ ├── Backoff.php Retry policy: server hint, else capped exponential + jitter
|
||||
│ ├── FeedRepository.php All SQL: upsert/query/count, staleness, feed management
|
||||
│ ├── FeedService.php Orchestration: refreshStale(), page(), sources()
|
||||
│ ├── FeedManager.php Manual selection: addReddit()/addYouTube()/remove()/setEnabled()
|
||||
@@ -279,8 +300,9 @@ button when `has_more` is false.
|
||||
|---|---|---|
|
||||
| `App\Database` | Creates the SQLite file/dir, applies schema, sets WAL + busy timeout | `connect()` |
|
||||
| `App\FeedFetcher` | Fetch one feed (with host throttle + 5xx retry), parse with laminas, extract media fields, normalize to a plain array | `fetch()` |
|
||||
| `App\FeedRepository` | Every SQL statement; returns arrays/DTO-less rows | `syncFeeds()`, `enabledFeeds()`, `staleFeeds()`, `upsertItems()`, `recordAttempt()`, `getItems()`, `countItems()`, `addFeed()`, `setFeedEnabled()`, `deleteFeed()`, `feedsStatus()` |
|
||||
| `App\FeedService` | Business flow: refresh policy + paginated reads | `refreshStale()`, `page()`, `sources()`, `feedsStatus()` |
|
||||
| `App\Backoff` | Turns a failure (fail count + optional server hint) into the next retry time | `nextRetryAt()` |
|
||||
| `App\FeedRepository` | Every SQL statement; returns arrays/DTO-less rows | `syncFeeds()`, `enabledFeeds()`, `staleFeeds()`, `hostLastAttempt()`, `upsertItems()`, `recordAttempt()`, `getItems()`, `countItems()`, `addFeed()`, `setFeedEnabled()`, `deleteFeed()`, `feedsStatus()` |
|
||||
| `App\FeedService` | Business flow: single-flight refresh (throttled hosts, combined Reddit fetch) + paginated reads | `refreshStale()`, `page()`, `sources()`, `feedsStatus()` |
|
||||
| `App\FeedManager` | Manual selection changes; validates new feeds with one immediate fetch (404 rejects the entry) | `addReddit()`, `addYouTube()`, `remove()`, `setEnabled()` |
|
||||
| `App\ChannelResolver` | YouTube input (`@handle`, `UC…` ID or channel URL) → Atom feed URL; resolves handles once via the channel page (youtube.com only) | `resolve()` |
|
||||
| `App\ManageAuth` | Session gate for `manage.php`: password login, CSRF tokens, flash messages | `login()`, `authed()`, `csrfToken()`, `checkCsrf()` |
|
||||
@@ -324,6 +346,8 @@ Indexes: `idx_items_published(published_at DESC)`,
|
||||
| `last_error` | last failure message, `NULL` on success |
|
||||
| `origin` | `config` (from `config/feeds.php`) or `manual` (added via manage.php) |
|
||||
| `enabled` | `1` = selected; disabled feeds are neither refreshed nor shown |
|
||||
| `fail_count` | consecutive failures; resets on success (drives exponential backoff) |
|
||||
| `next_retry_at` | ISO-8601 UTC; the feed is skipped until this time (rate-limit / backoff) |
|
||||
|
||||
Only items whose `feed_url` belongs to an enabled feed are served — disabling or
|
||||
removing a feed hides its cached stories without deleting them.
|
||||
@@ -338,10 +362,12 @@ removing a feed hides its cached stories without deleting them.
|
||||
|---|---|---|
|
||||
| `db_path` | `data/feeds.sqlite` | SQLite file location |
|
||||
| `cache_ttl` | `1800` | seconds before a healthy feed is re-fetched |
|
||||
| `error_retry_ttl` | `120` | base retry delay for failed feeds (plus jitter) |
|
||||
| `host_interval_ms` | `1000` | min gap between requests to the same host |
|
||||
| `throttled_hosts` | `['www.reddit.com']` | hosts fetched sparingly |
|
||||
| `throttled_host_max_per_refresh` | `1` | max feeds of a throttled host per refresh |
|
||||
| `error_retry_ttl` | `300` | base backoff for failed feeds (doubles per consecutive failure, plus jitter) |
|
||||
| `error_retry_cap` | `7200` | maximum retry backoff for a failing feed |
|
||||
| `ttl_jitter` | `300` | random seconds added to `cache_ttl` so feeds don't all expire at once |
|
||||
| `host_interval_ms` | `1000` | min gap between requests to the same host within one refresh |
|
||||
| `throttled_hosts` | `['www.reddit.com' => 60, 'www.youtube.com' => 15]` | host => min seconds between requests (enforced across processes) |
|
||||
| `throttled_host_max_per_refresh` | `1` | max feeds of a throttled host fetched per refresh (combined Reddit = 1) |
|
||||
| `page_size` | `20` | items per page |
|
||||
| `ranges` | `1h,2h,6h,24h,7d` | time-filter windows (key → seconds) |
|
||||
| `request_timeout` / `connect_timeout` | `15` / `10` | Guzzle timeouts |
|
||||
@@ -472,7 +498,7 @@ The source tab and filter pill are generated automatically from `sources()`.
|
||||
| Symptom | Cause / fix |
|
||||
|---|---|
|
||||
| `cURL error 7 ... Could not connect` | SELinux: enable `httpd_can_network_connect` (see above). |
|
||||
| Reddit feeds empty / `HTTP 429` | Reddit's unauthenticated limit (~1 req/window). It self-heals via `error_retry_ttl` + jitter and the per-host cap; cached items still show. |
|
||||
| Reddit feeds empty / `HTTP 429` | Reddit's unauthenticated limit is tight (~1 request/min/IP). The app now combines all subreddits into one request, honors `x-ratelimit-reset`, backs off, and serves cached items meanwhile. If it persists, set a real `user_agent` contact and/or lower the Reddit refresh rate (`throttled_hosts`). |
|
||||
| `403` on `/src/...`, `/config/...` | Expected — `.htaccess` denies those paths. |
|
||||
| `unable to open database file` | `data/` not writable by the web user, or wrong SELinux type on `data/`. |
|
||||
| Blank page / 500 | Check the PHP error log; run `php -l` on changed files. |
|
||||
|
||||
Reference in New Issue
Block a user