Harwinder Sekhon
397ee1a413
Reddit allows roughly one request per minute per IP on unauthenticated .rss, but the app fired six separate subreddit requests, retried 429s every couple of minutes, and only paced hosts within a single process. Concurrent page/API/cron requests stampeded providers. - Combine all due subreddits into one /r/a+b+c/.rss request and attribute items back to each subreddit by link; fall back to individual fetches when a banned/private subreddit poisons the combined feed. - Single-flight refresh via data/refresh.lock so concurrent callers serve the cache instead of refetching. - Enforce throttled-host spacing across processes (feeds.hostLastAttempt) in addition to the per-refresh cap; add www.youtube.com to throttled hosts. - Honor 429 Retry-After / x-ratelimit-reset and back off exponentially (fail_count doubling, capped at error_retry_cap, plus jitter), reset on success. New feeds columns fail_count and next_retry_at. - Jitter the healthy TTL so feeds don't all expire at once. - Detect add-feed 404 by status instead of matching the error string. - Update README (semantics, data model, config) and default user agent.
59 lines
2.1 KiB
PHP
59 lines
2.1 KiB
PHP
<?php
|
|
|
|
return [
|
|
// SQLite database location (kept outside the web root).
|
|
'db_path' => dirname(__DIR__) . '/data/feeds.sqlite',
|
|
|
|
// Seconds before a feed is considered stale and re-fetched on demand.
|
|
'cache_ttl' => 1800,
|
|
|
|
// Shorter retry interval for feeds whose last fetch failed (e.g. rate limited).
|
|
// Backoff grows from here (doubling per consecutive failure) and is capped
|
|
// by error_retry_cap; a server Retry-After hint always wins.
|
|
'error_retry_ttl' => 300,
|
|
|
|
// Maximum retry backoff for a failing feed (2 hours).
|
|
'error_retry_cap' => 7200,
|
|
|
|
// Random extra seconds added to the healthy TTL so feeds do not all expire
|
|
// (and get fetched) at the same moment.
|
|
'ttl_jitter' => 300,
|
|
|
|
// Minimum milliseconds between requests to the same host within one refresh.
|
|
'host_interval_ms' => 1000,
|
|
|
|
// Hosts with aggressive rate limits: minimum seconds between requests to
|
|
// that host (enforced across processes via the cache DB), and at most
|
|
// throttled_host_max_per_refresh feeds of it per refresh.
|
|
'throttled_hosts' => [
|
|
'www.reddit.com' => 60,
|
|
'www.youtube.com' => 15,
|
|
],
|
|
'throttled_host_max_per_refresh' => 1,
|
|
|
|
// Number of items per page.
|
|
'page_size' => 20,
|
|
|
|
// Time-range filters (key => window length in seconds), shown newest-first.
|
|
'ranges' => [
|
|
'1h' => 3600,
|
|
'2h' => 7200,
|
|
'6h' => 21600,
|
|
'24h' => 86400,
|
|
'7d' => 604800,
|
|
],
|
|
|
|
// HTTP client settings.
|
|
'request_timeout' => 15,
|
|
'connect_timeout' => 10,
|
|
|
|
// Identify the client to feed providers (important for Reddit). Reddit asks
|
|
// for a descriptive, unique agent — set a real contact before deploying.
|
|
'user_agent' => 'RSSFeed/1.0 (self-hosted dashboard; contact: admin@example.com)',
|
|
|
|
// Password for manage.php (manual subreddit/channel configuration).
|
|
// Accepts a plain password or a password_hash() value; empty disables the
|
|
// manage page entirely. Change this before exposing the app publicly.
|
|
'manage_password' => 'rssfeed-admin',
|
|
];
|