From 60bcf226928c09991207c04fce0ba143e261aa6c Mon Sep 17 00:00:00 2001 From: Harwinder Sekhon Date: Mon, 5 Oct 2026 17:11:23 -0400 Subject: [PATCH] Add developer README with architecture and installation instructions --- README.md | 416 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 416 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..f7f7c11 --- /dev/null +++ b/README.md @@ -0,0 +1,416 @@ +# RSS Feed Dashboard + +A small PHP dashboard that aggregates RSS/Atom feeds (YouTube channels, Reddit +subreddits, RT news) into a single stream, newest first, with source and +time-range filters and AJAX pagination. + +No framework, no build step. PHP + SQLite for caching, Bootstrap 5 for the UI. + +--- + +## What it does + +- Pulls **YouTube** (Atom), **Reddit** (RSS 2.0) and **RT** (RSS) feeds. +- Merges every item into one timeline ordered by `published_at` (fallback + `fetched_at`), newest first. +- Filters by **source** (`youtube` / `reddit` / `rt`) and **time range** + (all time, 1h, 2h, 6h, 24h, 7d). +- Paginates with a **"Load more stories"** button (AJAX, 20 per page). +- Caches everything in **SQLite** so page loads never wait on the network; feeds + are refreshed on demand when stale (default TTL 30 min). + +--- + +## Tech stack + +| Layer | Choice | +|------------|---------------------------------------------------------------| +| Runtime | PHP 8.1+ (`ext-pdo`, `ext-curl`) | +| Parsing | [`laminas/laminas-feed`](https://github.com/laminas/laminas-feed) | +| HTTP | [`guzzlehttp/guzzle`](https://github.com/guzzle/guzzle) | +| Storage | SQLite via PDO (WAL mode) | +| Front end | Bootstrap 5 (CDN) + vanilla JS, custom `assets/style.css` | +| Autoload | Composer PSR-4, `App\` → `src/` | + +--- + +## Installation + +### Requirements + +- PHP **8.1+** with `pdo_sqlite`, `curl`, `dom`, `simplexml`, `mbstring` +- Composer (or the bundled `composer.phar`) +- A web server with PHP (Apache/nginx + php-fpm) — or PHP's built-in server for a quick look +- No database server: SQLite is embedded + +Verify PHP: + +```bash +php -v +php -m | grep -E 'pdo_sqlite|curl|dom|simplexml|mbstring' +``` + +### 1. Get the code + +```bash +git clone ssh://git@192.168.0.11:222/harry/RSS-Feed.git +cd RSS-Feed +``` + +### 2. Install dependencies + +```bash +php composer.phar install +``` + +This populates `vendor/`. If Composer is installed globally, `composer install` +works the same. (`vendor/` and the SQLite DB are gitignored.) + +### 3. Configure + +- **Feeds** — edit `config/feeds.php` (see *Configuration* for the format and the + YouTube `@handle` → `channel_id` note). +- **Runtime** — optionally tune `config/app.php` (page size, TTLs, ranges, + timeouts, user agent). + +### 4. Create the cache directory + +```bash +mkdir -p data +``` + +The PHP/web user must be able to write `data/feeds.sqlite` (SQLite also creates +`-wal`/`-shm` files there). Make it writable by the web user: + +```bash +chmod 775 data +# or, when the web user differs from the owner: +chmod 1777 data +``` + +Cleaner still, let the web server's user own it: + +```bash +sudo chown -R www-data:www-data data # Debian/Ubuntu +# sudo chown -R apache:apache data # RHEL/Fedora +``` + +### 5. Point a web server at the project + +The **project root is the web root**. Apache example: + +```apache + + ServerName localhost + DocumentRoot "/path/to/RSS-Feed" + + + Options Indexes FollowSymLinks + AllowOverride All + Require all granted + + +``` + +Ensure `index.php` is a directory index (`DirectoryIndex index.php`) and that +`.htaccess` overrides are allowed (`AllowOverride All`) so the bundled protection +files take effect. For nginx: set `root /path/to/RSS-Feed;`, route requests to +`index.php` / `api/items.php`, and deny `src/`, `config/`, `vendor/`, +`templates/`, `data/`. + +> The `.htaccess` files block HTTP access to `src/`, `config/`, `vendor/`, +> `templates/`, `data/` and `composer.*`. They do **not** affect PHP's filesystem +> reads, so the app keeps working normally. + +### 6. SELinux (Fedora / RHEL only) + +```bash +# Let php-fpm write the SQLite cache +chcon -R -t httpd_sys_rw_content_t data + +# Let php-fpm make outbound HTTP requests (Reddit/YouTube/RT) +sudo setsebool -P httpd_can_network_connect on +``` + +Skipping the second command still renders cached data, but every fetch fails with +`cURL error 7 ... Could not connect to server`. See *Deployment notes*. + +### 7. First run + +Open the site (`http://localhost/`, or `http://localhost/RSS-Feed/` if it lives +under a parent document root). The first request fetches all feeds and builds +`data/feeds.sqlite` — this can take several seconds. Reddit may return `429` at +first; its items appear on subsequent refreshes. + +### Quick start without a web server + +```bash +php -S 127.0.0.1:8000 -t . +# http://127.0.0.1:8000/ +``` + +The built-in server does not read `.htaccess`, so use it for local evaluation +only. + +### Optional: scheduled refresh (no root) + +If you can't change SELinux (or prefer not to fetch during web requests), run the +refresher from cron as your own user — CLI processes are unconfined and have +network access: + +```bash +cd /path/to/RSS-Feed +php -r '$app = require "src/bootstrap.php"; $app["service"]->refreshStale();' +``` + +Add to `crontab -e` to refresh every 30 minutes: + +``` +*/30 * * * * cd /path/to/RSS-Feed && php -r '$app = require "src/bootstrap.php"; $app["service"]->refreshStale();' >/dev/null 2>&1 +``` + +When the cron user differs from the web user, make sure both can write `data/` +(see step 4). + +--- + +## Architecture + +``` +Browser + │ GET /RSSFeed/ ?source=&range= GET /RSSFeed/api/items.php ?source=&range=&page= + ▼ ▼ +index.php api/items.php + └── require src/bootstrap.php ─────────────────────────┘ + │ builds: PDO, FeedRepository, FeedFetcher, FeedService, config, feeds + ▼ + FeedService::refreshStale() ← runs on both entry points, respects TTL + │ + ├─ FeedRepository::syncFeeds() ensure every configured feed has a row + ├─ FeedRepository::staleFeeds() feeds older than TTL (or error cooldown) + └─ for each stale feed: + FeedFetcher::fetch() Guzzle GET → laminas parse → normalize + FeedRepository::upsertItems() INSERT OR IGNORE (dedup by sha256(link)) + FeedRepository::recordAttempt() success/error + timestamps + + FeedService::page(source, page, pageSize, since) ← read path + └─ FeedRepository::getItems()/countItems() SQL over the SQLite cache + │ + └─ View::render('layout' | 'cards') server-rendered HTML + (api/items.php returns that HTML inside a JSON envelope) +``` + +**Request lifecycle (page load):** bootstrap → refresh stale feeds → validate +`source`/`range` → query one page → render `templates/layout.php` which includes +`templates/cards.php`. + +**Request lifecycle (load more):** `assets/app.js` calls `api/items.php` with the +next `page` (plus current `source` and `range`), receives +`{ html, page, total, has_more, ... }`, appends `html` to `#feed`, and hides the +button when `has_more` is false. + +### Caching & refresh semantics + +- `items` and `feeds` live in `data/feeds.sqlite` (WAL mode, outside the web root). +- A feed is **stale** when `now - last_attempt_at >= cache_ttl`, or, if its last + attempt errored, when `now - last_attempt_at >= error_retry_ttl + jitter` + (jitter derived from `crc32(url)` so rate-limited feeds don't all retry at once). +- Fetch failures are recorded (`last_error`) and **never delete cached items** — + the dashboard keeps serving the last good data. +- Reddit rate-limits hard. `throttled_hosts` caps how many feeds of a given host + are fetched per refresh (`throttled_host_max_per_refresh`), and requests to the + same host are spaced by `host_interval_ms`. Only 5xx responses retry in-request. + +--- + +## Files + +``` +. +├── index.php Page entry point: refresh, filter, render page 1 +├── api/items.php JSON endpoint for "Load more" (returns rendered HTML) +├── config/ +│ ├── app.php Runtime settings (see Configuration) +│ └── feeds.php The feed list: [{source, label, url}, ...] +├── src/ +│ ├── bootstrap.php Wires config + classes together, returns the container +│ ├── Database.php PDO SQLite connection + schema migration +│ ├── FeedFetcher.php HTTP + parse + normalize a single feed +│ ├── FeedRepository.php All SQL: upsert/query/count, staleness, attempt state +│ ├── FeedService.php Orchestration: refreshStale(), page(), sources() +│ └── View.php Escaping + render() + labels/colors/time formatting +├── templates/ +│ ├── layout.php Full page shell, filters, grid, empty state, Load more +│ └── cards.php One feed card partial (used by page + API) +├── assets/ +│ ├── app.js Load-more, relative timestamps, filter propagation +│ └── style.css Design tokens + card/pill/header styling +├── data/ SQLite database (gitignored, created at runtime) +└── vendor/ Composer deps (gitignored) +``` + +### Class reference + +| Class / file | Responsibility | Key methods | +|---|---|---| +| `App\Database` | Creates the SQLite file/dir, applies schema, sets WAL + busy timeout | `connect()` | +| `App\FeedFetcher` | Fetch one feed (with host throttle + 5xx retry), parse with laminas, extract media fields, normalize to a plain array | `fetch()` | +| `App\FeedRepository` | Every SQL statement; returns arrays/DTO-less rows | `syncFeeds()`, `staleFeeds()`, `upsertItems()`, `recordAttempt()`, `getItems()`, `countItems()`, `feedsStatus()` | +| `App\FeedService` | Business flow: refresh policy + paginated reads | `refreshStale()`, `page()`, `sources()`, `feedsStatus()` | +| `App\View` | Output escaping, template rendering, source labels/colors, ranges, `timeAgo()` | `e()`, `render()`, `sourceLabel()`, `sourceColor()`, `rangeLabel()`, `timeAgo()` | +| `src/bootstrap.php` | Builds and returns `['config','feeds','pdo','repository','fetcher','service']` | — | + +`FeedFetcher` notes: laminas does not expose the YouTube `media:` extension, so +`extractExtras()` does a second DOM/XPath pass over the raw XML to pull +`media:thumbnail` and `media:description`. Thumbnail resolution order is +`media:thumbnail` → image `` → YouTube `i.ytimg.com` fallback. +Descriptions are stripped to plain text and truncated (XSS-safe; the template +escapes again on output). + +--- + +## Data model + +`items` — one row per article/video/post: + +| column | notes | +|---|---| +| `id` | `sha256(link)`, primary key (dedup) | +| `source` | `youtube` / `reddit` / `rt` | +| `feed_label`, `feed_url` | which feed it came from | +| `title`, `link`, `author` | | +| `published_at` | ISO-8601 UTC (nullable) | +| `description` | plain text, truncated | +| `thumbnail` | image URL (nullable) | +| `fetched_at` | ISO-8601 UTC | + +Indexes: `idx_items_published(published_at DESC)`, +`idx_items_source(source, published_at DESC)`. + +`feeds` — one row per configured feed + health state: + +| column | notes | +|---|---| +| `url` | primary key | +| `source`, `label` | from `config/feeds.php` | +| `last_attempt_at`, `last_success_at` | ISO-8601 UTC | +| `last_error` | last failure message, `NULL` on success | + +--- + +## Configuration + +`config/app.php` returns an array: + +| key | default | meaning | +|---|---|---| +| `db_path` | `data/feeds.sqlite` | SQLite file location | +| `cache_ttl` | `1800` | seconds before a healthy feed is re-fetched | +| `error_retry_ttl` | `120` | base retry delay for failed feeds (plus jitter) | +| `host_interval_ms` | `1000` | min gap between requests to the same host | +| `throttled_hosts` | `['www.reddit.com']` | hosts fetched sparingly | +| `throttled_host_max_per_refresh` | `1` | max feeds of a throttled host per refresh | +| `page_size` | `20` | items per page | +| `ranges` | `1h,2h,6h,24h,7d` | time-filter windows (key → seconds) | +| `request_timeout` / `connect_timeout` | `15` / `10` | Guzzle timeouts | +| `user_agent` | `RSSFeed/1.0 (...)` | sent to providers (matters for Reddit) | + +`config/feeds.php` returns a list of feeds: + +```php +[ + 'source' => 'reddit', // groups the item + drives its color + 'label' => 'r/homelab', // shown under the badge + 'url' => 'https://www.reddit.com/r/homelab/.rss', +] +``` + +**YouTube note:** YouTube serves Atom only by `channel_id`, not `@handle`. +Resolve a handle once (fetch `https://www.youtube.com/@handle` and read +`` or the canonical channel link) and store +`https://www.youtube.com/feeds/videos.xml?channel_id=...`. + +--- + +## Routes & query parameters + +| Route | Params | Returns | +|---|---|---| +| `index.php` (`/RSSFeed/`) | `source` (optional), `range` (optional) | full HTML page, page 1 | +| `api/items.php` | `page` (default 1), `source`, `range` | JSON `{ html, page, page_size, count, total, has_more, range, refresh }` | + +- `source` must match a `source` in `config/feeds.php`, else ignored. +- `range` must be a key in `ranges`, else ignored. +- Both filters are preserved across navigation and across "Load more". + +--- + +## Adding feeds and sources + +**New feed (existing source):** add an entry to `config/feeds.php`. `syncFeeds()` +registers it automatically on the next request; it will be fetched on the next +refresh. + +**New source type** (e.g. `hackernews`): add feeds with `'source' => 'hackernews'`, +then teach the UI its identity in `src/View.php`: + +```php +sourceLabel('hackernews') // tab / badge text +sourceColor('hackernews') // hex used for the card spine, chip and active pill +``` + +The source tab and filter pill are generated automatically from `sources()`. + +--- + +## Deployment notes (Apache + php-fpm on SELinux, e.g. Fedora) + +1. **Document root = project root.** `index.php` is served for `/RSSFeed/`; the + built-in files under `api/` and `assets/` are served directly. +2. **Protect internals.** `.htaccess` files deny HTTP access to `src/`, `config/`, + `vendor/`, `templates/`, `data/`, and `composer.*`. These only affect HTTP; + PHP still reads the files from disk. Directory listing is disabled. +3. **Let the web user write the cache.** `data/` must be writable by the php-fpm + user (e.g. `apache`), and on SELinux it needs the `httpd_sys_rw_content_t` + type: + + ```bash + chmod 1777 data + chcon -R -t httpd_sys_rw_content_t data + ``` + +4. **Allow outbound HTTP.** On SELinux, php-fpm runs in `httpd_t` and outbound + connections are denied by default (symptom: `cURL error 7 ... Could not + connect to server`). Enable it once: + + ```bash + sudo setsebool -P httpd_can_network_connect on + ``` + + Without this, the app still renders the cached data but cannot refresh. As an + alternative (no root), run the fetcher from a user cron/CLI process instead — + CLI processes run unconfined and do have network access. + +--- + +## Troubleshooting + +| Symptom | Cause / fix | +|---|---| +| `cURL error 7 ... Could not connect` | SELinux: enable `httpd_can_network_connect` (see above). | +| Reddit feeds empty / `HTTP 429` | Reddit's unauthenticated limit (~1 req/window). It self-heals via `error_retry_ttl` + jitter and the per-host cap; cached items still show. | +| `403` on `/src/...`, `/config/...` | Expected — `.htaccess` denies those paths. | +| `unable to open database file` | `data/` not writable by the web user, or wrong SELinux type on `data/`. | +| Blank page / 500 | Check the PHP error log; run `php -l` on changed files. | +| Stale data | Feeds refresh only when older than `cache_ttl`; delete `data/feeds.sqlite` to force a rebuild, or lower `cache_ttl`. | +| New source has no color | Add a case to `View::sourceLabel()` / `View::sourceColor()`. | + +--- + +## Conventions + +- Strict types (`declare(strict_types=1)`) in all PHP files; PSR-12 style. +- All SQL is parameterized; all output is escaped via `App\View::e()`. +- No comments in code unless they explain non-obvious behavior. +- Tests: none yet — verify manually via the two entry points (page + API) and + `php -l`.