RSS-Feed/README.md

RSS Feed Dashboard

A small PHP dashboard that aggregates RSS/Atom feeds (YouTube channels, Reddit subreddits, RT news) into a single stream, newest first, with source and time-range filters and AJAX pagination.

No framework, no build step. PHP + SQLite for caching, Bootstrap 5 for the UI.

---

What it does

• Pulls YouTube (Atom), Reddit (RSS 2.0) and RT (RSS) feeds.
• Merges every item into one timeline ordered by published_at (fallback fetched_at), newest first.
• Filters by source (youtube / reddit / rt) and time range (all time, 1h, 2h, 6h, 24h, 7d).
• Lets you manually configure the subreddit and YouTube channel selection from the browser (manage.php, password protected): add channels by @handle, disable or remove feeds — no config editing needed.
• Paginates with a "Load more stories" button (AJAX, 20 per page).
• Caches everything in SQLite so page loads never wait on the network; feeds are refreshed on demand when stale (default TTL 30 min).

---

Tech stack

Layer	Choice
Runtime	PHP 8.1+ (ext-pdo, ext-curl)
Parsing	laminas/laminas-feed
HTTP	guzzlehttp/guzzle
Storage	SQLite via PDO (WAL mode)
Front end	Bootstrap 5 (CDN) + vanilla JS, custom assets/style.css
Autoload	Composer PSR-4, App\ → src/

---

Installation

Requirements

• PHP 8.1+ with pdo_sqlite, curl, dom, simplexml, mbstring
• Composer (or the bundled composer.phar)
• A web server with PHP (Apache/nginx + php-fpm) — or PHP's built-in server for a quick look
• No database server: SQLite is embedded

Verify PHP:

php -v
php -m | grep -E 'pdo_sqlite|curl|dom|simplexml|mbstring'

1. Get the code

git clone ssh://git@192.168.0.11:222/harry/RSS-Feed.git
cd RSS-Feed

2. Install dependencies

composer.phar is committed with the repo, so no global Composer is required:

php composer.phar install --no-dev --optimize-autoloader

This populates vendor/ (both vendor/ and the SQLite DB are gitignored, so they are created here). A system Composer works the same: composer install --no-dev --optimize-autoloader.

If composer.phar is ever missing, fetch it once:

php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
php composer-setup.php --install-dir=. --filename=composer.phar
rm composer-setup.php

3. Configure

• Feeds — edit config/feeds.php (see Configuration for the format and the YouTube @handle → channel_id note), or skip this entirely and manage the selection from the browser on manage.php (see Managing feeds; set its password in config/app.php).
• Runtime — optionally tune config/app.php (page size, TTLs, ranges, timeouts, user agent).

4. Create the cache directory

mkdir -p data

The PHP/web user must be able to write data/feeds.sqlite (SQLite also creates -wal/-shm files there). Make it writable by the web user:

chmod 775 data
# or, when the web user differs from the owner:
chmod 1777 data

Cleaner still, let the web server's user own it:

sudo chown -R www-data:www-data data   # Debian/Ubuntu
# sudo chown -R apache:apache data     # RHEL/Fedora

5. Point a web server at the project

The project root is the web root. Apache example:

<VirtualHost *:80>
    ServerName localhost
    DocumentRoot "/path/to/RSS-Feed"

    <Directory "/path/to/RSS-Feed">
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>
</VirtualHost>

Ensure index.php is a directory index (DirectoryIndex index.php) and that .htaccess overrides are allowed (AllowOverride All) so the bundled protection files take effect. For nginx: set root /path/to/RSS-Feed;, route requests to index.php / api/items.php, and deny src/, config/, vendor/, templates/, data/.

The .htaccess files block HTTP access to src/, config/, vendor/, templates/, data/ and composer.*. They do not affect PHP's filesystem reads, so the app keeps working normally.

6. SELinux (Fedora / RHEL only)

# Let php-fpm write the SQLite cache
chcon -R -t httpd_sys_rw_content_t data

# Let php-fpm make outbound HTTP requests (Reddit/YouTube/RT)
sudo setsebool -P httpd_can_network_connect on

Skipping the second command still renders cached data, but every fetch fails with cURL error 7 ... Could not connect to server. See Deployment notes.

7. First run

Open the site (http://localhost/, or http://localhost/RSS-Feed/ if it lives under a parent document root). The first request fetches all feeds and builds data/feeds.sqlite — this can take several seconds. Reddit may return 429 at first; its items appear on subsequent refreshes.

Quick start without a web server

php -S 127.0.0.1:8000 -t .
# http://127.0.0.1:8000/

The built-in server does not read .htaccess, so use it for local evaluation only.

Optional: scheduled refresh (no root)

If you can't change SELinux (or prefer not to fetch during web requests), run the refresher from cron as your own user — CLI processes are unconfined and have network access:

cd /path/to/RSS-Feed
php -r '$app = require "src/bootstrap.php"; $app["service"]->refreshStale();'

Add to crontab -e to refresh every 30 minutes:

*/30 * * * * cd /path/to/RSS-Feed && php -r '$app = require "src/bootstrap.php"; $app["service"]->refreshStale();' >/dev/null 2>&1

When the cron user differs from the web user, make sure both can write data/ (see step 4).

---

Architecture

Browser
  │  GET /RSSFeed/ ?source=&range=                GET /RSSFeed/api/items.php ?source=&range=&page=
  ▼                                                         ▼
index.php                                            api/items.php
  └── require src/bootstrap.php  ─────────────────────────┘
          │  builds: PDO, FeedRepository, FeedFetcher, FeedService, config, feeds
          ▼
   FeedService::refreshStale()          ← runs on both entry points, respects TTL
          │
          ├─ FeedRepository::syncFeeds()      ensure every configured feed has a row
          ├─ flock data/refresh.lock          single-flight: concurrent callers skip
          ├─ FeedRepository::staleFeeds()     due feeds, minus retry backoff, jittered TTL
          ├─ host pacing                      per-refresh cap + cross-process host interval
          ├─ all stale Reddit feeds → ONE combined request (/r/a+b+c/.rss),
          │     items attributed back to each subreddit by link
          └─ for each other stale feed:
                 FeedFetcher::fetch()         Guzzle GET → laminas parse → normalize
                 FeedRepository::upsertItems() INSERT OR IGNORE (dedup by sha256(link))
                 FeedRepository::recordAttempt()  success → clears error/fail_count;
                                                  failure → fail_count++, next_retry_at

   FeedService::page(source, page, pageSize, since)   ← read path
          └─ FeedRepository::getItems()/countItems()  SQL over the SQLite cache
                 │
                 └─ View::render('layout' | 'cards')  server-rendered HTML
                    (api/items.php returns that HTML inside a JSON envelope)

Request lifecycle (page load): bootstrap → refresh stale feeds → validate source/range → query one page → render templates/layout.php which includes templates/cards.php.

Request lifecycle (load more): assets/app.js calls api/items.php with the next page (plus current source and range), receives { html, page, total, has_more, ... }, appends html to #feed, and hides the button when has_more is false.

Caching & refresh semantics

• items and feeds live in data/feeds.sqlite (WAL mode, outside the web root).
• Only enabled feeds are refreshed and served — disabling one from the manage page stops both without deleting its cached items.
• A feed is due when its retry backoff has elapsed (next_retry_at) and its jittered TTL has passed: now - last_attempt_at >= cache_ttl + crc32(url) % ttl_jitter (jitter spreads expiry so feeds don't all refresh at once).
• One refresh at a time. refreshStale() takes a non-blocking lock on data/refresh.lock; concurrent page/API/cron requests serve the cache and return immediately instead of stampeding providers.
• Rate limits are respected. Throttled hosts (throttled_hosts) are limited to throttled_host_max_per_refresh fetches per refresh and to a per-host minimum interval enforced across processes. The gap between requests to the same host is also paced in-request (host_interval_ms).
• Reddit is fetched as one request. All due subreddits are combined into a single reddit.com/r/a+b+c/.rss call and the items are attributed back to each subreddit by their links. If that fails with 404/403 (a private or banned subreddit poisons the combined feed), the app falls back to individual fetches so the culprit is isolated.
• 429 handling. A 429 carries the server's own window (Retry-After, or Reddit's x-ratelimit-reset); the feed is parked until then. Other failures back off exponentially (error_retry_ttl doubling per consecutive failure, up to error_retry_cap, plus jitter). Success resets the counters.
• Fetch failures are recorded (last_error, fail_count, next_retry_at) and never delete cached items — the dashboard keeps serving the last good data.

---

Files

.
├── index.php                 Page entry point: refresh, filter, render page 1
├── manage.php                 Manual feed configuration (password protected)
├── api/items.php             JSON endpoint for "Load more" (returns rendered HTML)
├── config/
│   ├── app.php               Runtime settings (see Configuration)
│   └── feeds.php             The default feed list: [{source, label, url}, ...]
├── src/
│   ├── bootstrap.php         Wires config + classes together, returns the container
│   ├── Database.php          PDO SQLite connection + schema migration
│   ├── FeedFetcher.php       HTTP + parse + normalize a single feed
│   ├── FeedFetchException.php  Fetch failure carrying its HTTP status
│   ├── RateLimitException.php  HTTP 429, carries the server's retry window
│   ├── Backoff.php           Retry policy: server hint, else capped exponential + jitter
│   ├── FeedRepository.php    All SQL: upsert/query/count, staleness, feed management
│   ├── FeedService.php       Orchestration: refreshStale(), page(), sources()
│   ├── FeedManager.php       Manual selection: addReddit()/addYouTube()/remove()/setEnabled()
│   ├── ChannelResolver.php   YouTube @handle / channel URL → channel_id → Atom feed URL
│   ├── ManageAuth.php        manage.php gate: password, CSRF, flash messages
│   └── View.php              Escaping + render() + labels/colors/time formatting
├── templates/
│   ├── layout.php            Full page shell, filters, grid, empty state, Load more
│   ├── manage.php            Feed management page (login, add forms, feed list)
│   └── cards.php             One feed card partial (used by page + API)
├── assets/
│   ├── app.js                Load-more, relative timestamps, filter propagation
│   ├── manage.js             Confirm dialogs on the manage page
│   └── style.css             Design tokens + card/pill/header/manage styling
├── data/                     SQLite database (gitignored, created at runtime)
└── vendor/                   Composer deps (gitignored)

Class reference

Class / file	Responsibility	Key methods
App\Database	Creates the SQLite file/dir, applies schema, sets WAL + busy timeout	connect()
App\FeedFetcher	Fetch one feed (with host throttle + 5xx retry), parse with laminas, extract media fields, normalize to a plain array	fetch()
App\Backoff	Turns a failure (fail count + optional server hint) into the next retry time	nextRetryAt()
App\FeedRepository	Every SQL statement; returns arrays/DTO-less rows	syncFeeds(), enabledFeeds(), staleFeeds(), hostLastAttempt(), upsertItems(), recordAttempt(), getItems(), countItems(), addFeed(), setFeedEnabled(), deleteFeed(), feedsStatus()
App\FeedService	Business flow: single-flight refresh (throttled hosts, combined Reddit fetch) + paginated reads	refreshStale(), page(), sources(), feedsStatus()
App\FeedManager	Manual selection changes; validates new feeds with one immediate fetch (404 rejects the entry)	addReddit(), addYouTube(), remove(), setEnabled()
App\ChannelResolver	YouTube input (@handle, UC… ID or channel URL) → Atom feed URL; resolves handles once via the channel page (youtube.com only)	resolve()
App\ManageAuth	Session gate for manage.php: password login, CSRF tokens, flash messages	login(), authed(), csrfToken(), checkCsrf()
App\View	Output escaping, template rendering, source labels/colors, ranges, timeAgo()	e(), render(), sourceLabel(), sourceColor(), rangeLabel(), timeAgo()
src/bootstrap.php	Builds and returns ['config','feeds','pdo','repository','fetcher','resolver','manager','service']	—

FeedFetcher notes: laminas does not expose the YouTube media: extension, so extractExtras() does a second DOM/XPath pass over the raw XML to pull media:thumbnail and media:description. Thumbnail resolution order is media:thumbnail → image <enclosure> → YouTube i.ytimg.com fallback. Descriptions are stripped to plain text and truncated (XSS-safe; the template escapes again on output).

---

Data model

items — one row per article/video/post:

column	notes
id	sha256(link), primary key (dedup)
source	youtube / reddit / rt
feed_label, feed_url	which feed it came from
title, link, author
published_at	ISO-8601 UTC (nullable)
description	plain text, truncated
thumbnail	image URL (nullable)
fetched_at	ISO-8601 UTC

Indexes: idx_items_published(published_at DESC), idx_items_source(source, published_at DESC).

feeds — one row per feed (config or manually added) + selection/health state:

column	notes
url	primary key
source, label	from config/feeds.php or the manage page
last_attempt_at, last_success_at	ISO-8601 UTC
last_error	last failure message, NULL on success
origin	config (from config/feeds.php) or manual (added via manage.php)
enabled	1 = selected; disabled feeds are neither refreshed nor shown
fail_count	consecutive failures; resets on success (drives exponential backoff)
next_retry_at	ISO-8601 UTC; the feed is skipped until this time (rate-limit / backoff)

Only items whose feed_url belongs to an enabled feed are served — disabling or removing a feed hides its cached stories without deleting them.

---

Configuration

config/app.php returns an array:

key	default	meaning
db_path	data/feeds.sqlite	SQLite file location
cache_ttl	1800	seconds before a healthy feed is re-fetched
error_retry_ttl	300	base backoff for failed feeds (doubles per consecutive failure, plus jitter)
error_retry_cap	7200	maximum retry backoff for a failing feed
ttl_jitter	300	random seconds added to cache_ttl so feeds don't all expire at once
host_interval_ms	1000	min gap between requests to the same host within one refresh
throttled_hosts	['www.reddit.com' => 60, 'www.youtube.com' => 15]	host => min seconds between requests (enforced across processes)
throttled_host_max_per_refresh	1	max feeds of a throttled host fetched per refresh (combined Reddit = 1)
page_size	20	items per page
ranges	1h,2h,6h,24h,7d	time-filter windows (key → seconds)
request_timeout / connect_timeout	15 / 10	Guzzle timeouts
user_agent	RSSFeed/1.0 (...)	sent to providers (matters for Reddit)
manage_password	rssfeed-admin	password for manage.php; plain or password_hash() value, empty = management disabled

config/feeds.php returns a list of feeds:

[
    'source' => 'reddit',                    // groups the item + drives its color
    'label'  => 'r/homelab',                 // shown under the badge
    'url'    => 'https://www.reddit.com/r/homelab/.rss',
]

YouTube note: YouTube serves Atom only by channel_id, not @handle. Resolve a handle once (fetch https://www.youtube.com/@handle and read <meta itemprop="identifier"> or the canonical channel link) and store https://www.youtube.com/feeds/videos.xml?channel_id=....

---

Managing feeds (manual configuration)

manage.php (linked as Manage in the dashboard header) configures the subreddit/YouTube selection from the browser. It is password protected:

• Set manage_password in config/app.php. It accepts a plain password (fine for a LAN/dev box) or a password_hash() value for better hygiene:

  php -r 'echo password_hash("your secret", PASSWORD_DEFAULT), PHP_EOL;'
  

• An empty value disables the page (it shows a setup notice instead).

What you can do there:

Action	Input	Behavior
Add subreddit	homelab, r/homelab, or any reddit.com/r/… URL	Normalized to https://www.reddit.com/r/<name>/.rss, labeled r/<name>
Add YouTube channel	@handle, UC… channel ID, or any channel URL	Handle resolved to channel_id once, stored as the Atom feed URL; the label becomes the real channel title
Disable / enable	toggle per feed	Disabled feeds are neither refreshed nor shown (their cached stories are kept and reappear when re-enabled)
Remove	manual feeds	Deleted from the selection; re-add anytime
Remove	config feeds	Hidden (enabled = 0) — config/feeds.php re-registers them on every sync, so they cannot be deleted from the UI

Semantics worth knowing:

• Adding a feed fetches it immediately (bypassing the per-refresh Reddit host cap): an HTTP 404 rejects the entry ("not found, nothing was added"), while a rate-limited (429) add is kept and fills in on a later refresh.
• Re-adding an existing feed URL re-enables it.
• Handle resolution only ever requests youtube.com pages — user input can never point the resolver at arbitrary URLs.
• All POST actions are CSRF-protected (session token) and use POST-then-redirect.
• config/feeds.php remains the default set and keeps working as before; the manage page layers manual feeds and on/off state on top of it (persisted in the feeds table, so they survive restarts).

---

Routes & query parameters

Route	Params	Returns
index.php (/RSSFeed/)	source (optional), range (optional)	full HTML page, page 1
api/items.php	page (default 1), source, range	JSON { html, page, page_size, count, total, has_more, range, refresh }
manage.php	— (POST: login, logout, add_reddit, add_youtube, toggle, remove)	management page (password gated)

• source must match a source in config/feeds.php, else ignored.
• range must be a key in ranges, else ignored.
• Both filters are preserved across navigation and across "Load more".

---

Adding feeds and sources

New feed (existing source): either add it on the manage page (no code editing, see Managing feeds) or add an entry to config/feeds.php. syncFeeds() registers config entries automatically on the next request; they will be fetched on the next refresh.

New source type (e.g. hackernews): add feeds with 'source' => 'hackernews', then teach the UI its identity in src/View.php:

sourceLabel('hackernews')  // tab / badge text
sourceColor('hackernews')  // hex used for the card spine, chip and active pill

The source tab and filter pill are generated automatically from sources().

---

Deployment notes (Apache + php-fpm on SELinux, e.g. Fedora)

1. Document root = project root. index.php is served for /RSSFeed/; the built-in files under api/ and assets/ are served directly.

2. Protect internals. .htaccess files deny HTTP access to src/, config/, vendor/, templates/, data/, and composer.*. These only affect HTTP; PHP still reads the files from disk. Directory listing is disabled.

3. Let the web user write the cache. data/ must be writable by the php-fpm user (e.g. apache), and on SELinux it needs the httpd_sys_rw_content_t type:

   chmod 1777 data
   chcon -R -t httpd_sys_rw_content_t data
   

4. Allow outbound HTTP. On SELinux, php-fpm runs in httpd_t and outbound connections are denied by default (symptom: cURL error 7 ... Could not connect to server). Enable it once:

   sudo setsebool -P httpd_can_network_connect on
   

   Without this, the app still renders the cached data but cannot refresh. As an alternative (no root), run the fetcher from a user cron/CLI process instead — CLI processes run unconfined and do have network access.

---

Troubleshooting

Symptom	Cause / fix
cURL error 7 ... Could not connect	SELinux: enable httpd_can_network_connect (see above).
Reddit feeds empty / HTTP 429	Reddit's unauthenticated limit is tight (~1 request/min/IP). The app now combines all subreddits into one request, honors x-ratelimit-reset, backs off, and serves cached items meanwhile. If it persists, set a real user_agent contact and/or lower the Reddit refresh rate (throttled_hosts).
403 on /src/..., /config/...	Expected — .htaccess denies those paths.
unable to open database file	data/ not writable by the web user, or wrong SELinux type on data/.
Blank page / 500	Check the PHP error log; run php -l on changed files.
Stale data	Feeds refresh only when older than cache_ttl; delete data/feeds.sqlite to force a rebuild, or lower cache_ttl.
New source has no color	Add a case to View::sourceLabel() / View::sourceColor().

---

Conventions

• Strict types (declare(strict_types=1)) in all PHP files; PSR-12 style.
• All SQL is parameterized; all output is escaped via App\View::e().
• No comments in code unless they explain non-obvious behavior.
• Tests: none yet — verify manually via the two entry points (page + API) and php -l.