Files

T

Harwinder Sekhon 60bcf22692 Add developer README with architecture and installation instructions

2026-10-05 17:11:23 -04:00

16 KiB

Raw Blame History

RSS Feed Dashboard

A small PHP dashboard that aggregates RSS/Atom feeds (YouTube channels, Reddit subreddits, RT news) into a single stream, newest first, with source and time-range filters and AJAX pagination.

No framework, no build step. PHP + SQLite for caching, Bootstrap 5 for the UI.

What it does

Pulls YouTube (Atom), Reddit (RSS 2.0) and RT (RSS) feeds.
Merges every item into one timeline ordered by published_at (fallback fetched_at), newest first.
Filters by source (youtube / reddit / rt) and time range (all time, 1h, 2h, 6h, 24h, 7d).
Paginates with a "Load more stories" button (AJAX, 20 per page).
Caches everything in SQLite so page loads never wait on the network; feeds are refreshed on demand when stale (default TTL 30 min).

Tech stack

Layer	Choice
Runtime	PHP 8.1+ (`ext-pdo`, `ext-curl`)
Parsing	`laminas/laminas-feed`
HTTP	`guzzlehttp/guzzle`
Storage	SQLite via PDO (WAL mode)
Front end	Bootstrap 5 (CDN) + vanilla JS, custom `assets/style.css`
Autoload	Composer PSR-4, `App\` → `src/`

Installation

Requirements

PHP 8.1+ with pdo_sqlite, curl, dom, simplexml, mbstring
Composer (or the bundled composer.phar)
A web server with PHP (Apache/nginx + php-fpm) — or PHP's built-in server for a quick look
No database server: SQLite is embedded

Verify PHP:

php -v
php -m | grep -E 'pdo_sqlite|curl|dom|simplexml|mbstring'

1. Get the code

git clone ssh://git@192.168.0.11:222/harry/RSS-Feed.git
cd RSS-Feed

2. Install dependencies

php composer.phar install

This populates vendor/. If Composer is installed globally, composer install works the same. (vendor/ and the SQLite DB are gitignored.)

3. Configure

Feeds — edit config/feeds.php (see Configuration for the format and the YouTube @handle → channel_id note).
Runtime — optionally tune config/app.php (page size, TTLs, ranges, timeouts, user agent).

4. Create the cache directory

mkdir -p data

The PHP/web user must be able to write data/feeds.sqlite (SQLite also creates -wal/-shm files there). Make it writable by the web user:

chmod 775 data
# or, when the web user differs from the owner:
chmod 1777 data

Cleaner still, let the web server's user own it:

sudo chown -R www-data:www-data data   # Debian/Ubuntu
# sudo chown -R apache:apache data     # RHEL/Fedora

5. Point a web server at the project

The project root is the web root. Apache example:

<VirtualHost *:80>
    ServerName localhost
    DocumentRoot "/path/to/RSS-Feed"

    <Directory "/path/to/RSS-Feed">
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>
</VirtualHost>

Ensure index.php is a directory index (DirectoryIndex index.php) and that .htaccess overrides are allowed (AllowOverride All) so the bundled protection files take effect. For nginx: set root /path/to/RSS-Feed;, route requests to index.php / api/items.php, and deny src/, config/, vendor/, templates/, data/.

The .htaccess files block HTTP access to src/, config/, vendor/, templates/, data/ and composer.*. They do not affect PHP's filesystem reads, so the app keeps working normally.

6. SELinux (Fedora / RHEL only)

# Let php-fpm write the SQLite cache
chcon -R -t httpd_sys_rw_content_t data

# Let php-fpm make outbound HTTP requests (Reddit/YouTube/RT)
sudo setsebool -P httpd_can_network_connect on

Skipping the second command still renders cached data, but every fetch fails with cURL error 7 ... Could not connect to server. See Deployment notes.

7. First run

Open the site (http://localhost/, or http://localhost/RSS-Feed/ if it lives under a parent document root). The first request fetches all feeds and builds data/feeds.sqlite — this can take several seconds. Reddit may return 429 at first; its items appear on subsequent refreshes.

Quick start without a web server

php -S 127.0.0.1:8000 -t .
# http://127.0.0.1:8000/

The built-in server does not read .htaccess, so use it for local evaluation only.

Optional: scheduled refresh (no root)

If you can't change SELinux (or prefer not to fetch during web requests), run the refresher from cron as your own user — CLI processes are unconfined and have network access:

cd /path/to/RSS-Feed
php -r '$app = require "src/bootstrap.php"; $app["service"]->refreshStale();'

Add to crontab -e to refresh every 30 minutes:

*/30 * * * * cd /path/to/RSS-Feed && php -r '$app = require "src/bootstrap.php"; $app["service"]->refreshStale();' >/dev/null 2>&1

When the cron user differs from the web user, make sure both can write data/ (see step 4).

Architecture

Browser
  │  GET /RSSFeed/ ?source=&range=                GET /RSSFeed/api/items.php ?source=&range=&page=
  ▼                                                         ▼
index.php                                            api/items.php
  └── require src/bootstrap.php  ─────────────────────────┘
          │  builds: PDO, FeedRepository, FeedFetcher, FeedService, config, feeds
          ▼
   FeedService::refreshStale()          ← runs on both entry points, respects TTL
          │
          ├─ FeedRepository::syncFeeds()      ensure every configured feed has a row
          ├─ FeedRepository::staleFeeds()     feeds older than TTL (or error cooldown)
          └─ for each stale feed:
                 FeedFetcher::fetch()         Guzzle GET → laminas parse → normalize
                 FeedRepository::upsertItems() INSERT OR IGNORE (dedup by sha256(link))
                 FeedRepository::recordAttempt()  success/error + timestamps

   FeedService::page(source, page, pageSize, since)   ← read path
          └─ FeedRepository::getItems()/countItems()  SQL over the SQLite cache
                 │
                 └─ View::render('layout' | 'cards')  server-rendered HTML
                    (api/items.php returns that HTML inside a JSON envelope)

Request lifecycle (page load): bootstrap → refresh stale feeds → validate source/range → query one page → render templates/layout.php which includes templates/cards.php.

Request lifecycle (load more): assets/app.js calls api/items.php with the next page (plus current source and range), receives { html, page, total, has_more, ... }, appends html to #feed, and hides the button when has_more is false.

Caching & refresh semantics

items and feeds live in data/feeds.sqlite (WAL mode, outside the web root).
A feed is stale when now - last_attempt_at >= cache_ttl, or, if its last attempt errored, when now - last_attempt_at >= error_retry_ttl + jitter (jitter derived from crc32(url) so rate-limited feeds don't all retry at once).
Fetch failures are recorded (last_error) and never delete cached items — the dashboard keeps serving the last good data.
Reddit rate-limits hard. throttled_hosts caps how many feeds of a given host are fetched per refresh (throttled_host_max_per_refresh), and requests to the same host are spaced by host_interval_ms. Only 5xx responses retry in-request.

Files

.
├── index.php                 Page entry point: refresh, filter, render page 1
├── api/items.php             JSON endpoint for "Load more" (returns rendered HTML)
├── config/
│   ├── app.php               Runtime settings (see Configuration)
│   └── feeds.php             The feed list: [{source, label, url}, ...]
├── src/
│   ├── bootstrap.php         Wires config + classes together, returns the container
│   ├── Database.php          PDO SQLite connection + schema migration
│   ├── FeedFetcher.php       HTTP + parse + normalize a single feed
│   ├── FeedRepository.php    All SQL: upsert/query/count, staleness, attempt state
│   ├── FeedService.php       Orchestration: refreshStale(), page(), sources()
│   └── View.php              Escaping + render() + labels/colors/time formatting
├── templates/
│   ├── layout.php            Full page shell, filters, grid, empty state, Load more
│   └── cards.php             One feed card partial (used by page + API)
├── assets/
│   ├── app.js                Load-more, relative timestamps, filter propagation
│   └── style.css             Design tokens + card/pill/header styling
├── data/                     SQLite database (gitignored, created at runtime)
└── vendor/                   Composer deps (gitignored)

Class reference

Class / file	Responsibility	Key methods
`App\Database`	Creates the SQLite file/dir, applies schema, sets WAL + busy timeout	`connect()`
`App\FeedFetcher`	Fetch one feed (with host throttle + 5xx retry), parse with laminas, extract media fields, normalize to a plain array	`fetch()`
`App\FeedRepository`	Every SQL statement; returns arrays/DTO-less rows	`syncFeeds()`, `staleFeeds()`, `upsertItems()`, `recordAttempt()`, `getItems()`, `countItems()`, `feedsStatus()`
`App\FeedService`	Business flow: refresh policy + paginated reads	`refreshStale()`, `page()`, `sources()`, `feedsStatus()`
`App\View`	Output escaping, template rendering, source labels/colors, ranges, `timeAgo()`	`e()`, `render()`, `sourceLabel()`, `sourceColor()`, `rangeLabel()`, `timeAgo()`
`src/bootstrap.php`	Builds and returns `['config','feeds','pdo','repository','fetcher','service']`	—

FeedFetcher notes: laminas does not expose the YouTube media: extension, so extractExtras() does a second DOM/XPath pass over the raw XML to pull media:thumbnail and media:description. Thumbnail resolution order is media:thumbnail → image <enclosure> → YouTube i.ytimg.com fallback. Descriptions are stripped to plain text and truncated (XSS-safe; the template escapes again on output).

Data model

items — one row per article/video/post:

column	notes
`id`	`sha256(link)`, primary key (dedup)
`source`	`youtube` / `reddit` / `rt`
`feed_label`, `feed_url`	which feed it came from
`title`, `link`, `author`
`published_at`	ISO-8601 UTC (nullable)
`description`	plain text, truncated
`thumbnail`	image URL (nullable)
`fetched_at`	ISO-8601 UTC

Indexes: idx_items_published(published_at DESC), idx_items_source(source, published_at DESC).

feeds — one row per configured feed + health state:

column	notes
`url`	primary key
`source`, `label`	from `config/feeds.php`
`last_attempt_at`, `last_success_at`	ISO-8601 UTC
`last_error`	last failure message, `NULL` on success

Configuration

config/app.php returns an array:

key	default	meaning
`db_path`	`data/feeds.sqlite`	SQLite file location
`cache_ttl`	`1800`	seconds before a healthy feed is re-fetched
`error_retry_ttl`	`120`	base retry delay for failed feeds (plus jitter)
`host_interval_ms`	`1000`	min gap between requests to the same host
`throttled_hosts`	`['www.reddit.com']`	hosts fetched sparingly
`throttled_host_max_per_refresh`	`1`	max feeds of a throttled host per refresh
`page_size`	`20`	items per page
`ranges`	`1h,2h,6h,24h,7d`	time-filter windows (key → seconds)
`request_timeout` / `connect_timeout`	`15` / `10`	Guzzle timeouts
`user_agent`	`RSSFeed/1.0 (...)`	sent to providers (matters for Reddit)

config/feeds.php returns a list of feeds:

[
    'source' => 'reddit',                    // groups the item + drives its color
    'label'  => 'r/homelab',                 // shown under the badge
    'url'    => 'https://www.reddit.com/r/homelab/.rss',
]

YouTube note: YouTube serves Atom only by channel_id, not @handle. Resolve a handle once (fetch https://www.youtube.com/@handle and read <meta itemprop="identifier"> or the canonical channel link) and store https://www.youtube.com/feeds/videos.xml?channel_id=....

Routes & query parameters

Route	Params	Returns
`index.php` (`/RSSFeed/`)	`source` (optional), `range` (optional)	full HTML page, page 1
`api/items.php`	`page` (default 1), `source`, `range`	JSON `{ html, page, page_size, count, total, has_more, range, refresh }`

source must match a source in config/feeds.php, else ignored.
range must be a key in ranges, else ignored.
Both filters are preserved across navigation and across "Load more".

Adding feeds and sources

New feed (existing source): add an entry to config/feeds.php. syncFeeds() registers it automatically on the next request; it will be fetched on the next refresh.

New source type (e.g. hackernews): add feeds with 'source' => 'hackernews', then teach the UI its identity in src/View.php:

sourceLabel('hackernews')  // tab / badge text
sourceColor('hackernews')  // hex used for the card spine, chip and active pill

The source tab and filter pill are generated automatically from sources().

Deployment notes (Apache + php-fpm on SELinux, e.g. Fedora)

Document root = project root. index.php is served for /RSSFeed/; the built-in files under api/ and assets/ are served directly.
Protect internals. .htaccess files deny HTTP access to src/, config/, vendor/, templates/, data/, and composer.*. These only affect HTTP; PHP still reads the files from disk. Directory listing is disabled.
Let the web user write the cache. data/ must be writable by the php-fpm user (e.g. apache), and on SELinux it needs the httpd_sys_rw_content_t type:
```
chmod 1777 data
chcon -R -t httpd_sys_rw_content_t data
```
Allow outbound HTTP. On SELinux, php-fpm runs in httpd_t and outbound connections are denied by default (symptom: cURL error 7 ... Could not connect to server). Enable it once:
```
sudo setsebool -P httpd_can_network_connect on
```
Without this, the app still renders the cached data but cannot refresh. As an alternative (no root), run the fetcher from a user cron/CLI process instead — CLI processes run unconfined and do have network access.

Troubleshooting

Symptom	Cause / fix
`cURL error 7 ... Could not connect`	SELinux: enable `httpd_can_network_connect` (see above).
Reddit feeds empty / `HTTP 429`	Reddit's unauthenticated limit (~1 req/window). It self-heals via `error_retry_ttl` + jitter and the per-host cap; cached items still show.
`403` on `/src/...`, `/config/...`	Expected — `.htaccess` denies those paths.
`unable to open database file`	`data/` not writable by the web user, or wrong SELinux type on `data/`.
Blank page / 500	Check the PHP error log; run `php -l` on changed files.
Stale data	Feeds refresh only when older than `cache_ttl`; delete `data/feeds.sqlite` to force a rebuild, or lower `cache_ttl`.
New source has no color	Add a case to `View::sourceLabel()` / `View::sourceColor()`.

Conventions

Strict types (declare(strict_types=1)) in all PHP files; PSR-12 style.
All SQL is parameterized; all output is escaped via App\View::e().
No comments in code unless they explain non-obvious behavior.
Tests: none yet — verify manually via the two entry points (page + API) and php -l.

16 KiB Raw Blame History