RSS-Feed/README.md

# RSS Feed Dashboard

A small PHP dashboard that aggregates RSS/Atom feeds (YouTube channels, Reddit
subreddits, RT news) into a single stream, newest first, with source and
time-range filters and AJAX pagination.

No framework, no build step. PHP + SQLite for caching, Bootstrap 5 for the UI.

---

## What it does

- Pulls **YouTube** (Atom), **Reddit** (RSS 2.0) and **RT** (RSS) feeds.
- Merges every item into one timeline ordered by `published_at` (fallback
  `fetched_at`), newest first.
- Filters by **source** (`youtube` / `reddit` / `rt`) and **time range**
  (all time, 1h, 2h, 6h, 24h, 7d).
- Lets you **manually configure the subreddit and YouTube channel
  selection** from the browser (`manage.php`, password protected): add
  channels by `@handle`, disable or remove feeds — no config editing needed.
- Paginates with a **"Load more stories"** button (AJAX, 20 per page).
- Streams the page shell (header, filters and a skeleton loader) to the browser
  **before** the feeds are fetched, so something paints within milliseconds on
  every open.
- Caches everything in **SQLite** so page loads never wait on the network; feeds
  are refreshed on demand when stale (default TTL 30 min).

---

## Tech stack

| Layer      | Choice                                                        |
|------------|---------------------------------------------------------------|
| Runtime    | PHP 8.1+ (`ext-pdo`, `ext-curl`)                              |
| Parsing    | [`laminas/laminas-feed`](https://github.com/laminas/laminas-feed) |
| HTTP       | [`guzzlehttp/guzzle`](https://github.com/guzzle/guzzle)       |
| Storage    | SQLite via PDO (WAL mode)                                     |
| Front end  | Bootstrap 5 (CDN) + vanilla JS, custom `assets/style.css`     |
| Autoload   | Composer PSR-4, `App\` → `src/`                               |

---

## Installation

### Requirements

- PHP **8.1+** with `pdo_sqlite`, `curl`, `dom`, `simplexml`, `mbstring`
- Composer (or the bundled `composer.phar`)
- A web server with PHP (Apache/nginx + php-fpm) — or PHP's built-in server for a quick look
- No database server: SQLite is embedded

Verify PHP:

```bash
php -v
php -m | grep -E 'pdo_sqlite|curl|dom|simplexml|mbstring'
```

### 1. Get the code

```bash
git clone ssh://git@192.168.0.11:222/harry/RSS-Feed.git
cd RSS-Feed
```

### 2. Install dependencies

`composer.phar` is committed with the repo, so no global Composer is required:

```bash
php composer.phar install --no-dev --optimize-autoloader
```

This populates `vendor/` (both `vendor/` and the SQLite DB are gitignored, so they
are created here). A system Composer works the same:
`composer install --no-dev --optimize-autoloader`.

If `composer.phar` is ever missing, fetch it once:

```bash
php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
php composer-setup.php --install-dir=. --filename=composer.phar
rm composer-setup.php
```

### 3. Configure

- **Feeds** — edit `config/feeds.php` (see *Configuration* for the format and the
  YouTube `@handle` → `channel_id` note), or skip this entirely and manage the
  selection from the browser on `manage.php` (see *Managing feeds*; set its
  password in `config/app.php`).
- **Runtime** — optionally tune `config/app.php` (page size, TTLs, ranges,
  timeouts, user agent).

### 4. Create the cache directory

```bash
mkdir -p data
```

The PHP/web user must be able to write `data/feeds.sqlite` (SQLite also creates
`-wal`/`-shm` files there). Make it writable by the web user:

```bash
chmod 775 data
# or, when the web user differs from the owner:
chmod 1777 data
```

Cleaner still, let the web server's user own it:

```bash
sudo chown -R www-data:www-data data   # Debian/Ubuntu
# sudo chown -R apache:apache data     # RHEL/Fedora
```

### 5. Point a web server at the project

The **project root is the web root**. Apache example:

```apache
<VirtualHost *:80>
    ServerName localhost
    DocumentRoot "/path/to/RSS-Feed"

    <Directory "/path/to/RSS-Feed">
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>
</VirtualHost>
```

Ensure `index.php` is a directory index (`DirectoryIndex index.php`) and that
`.htaccess` overrides are allowed (`AllowOverride All`) so the bundled protection
files take effect. For nginx: set `root /path/to/RSS-Feed;`, route requests to
`index.php` / `api/items.php`, and deny `src/`, `config/`, `vendor/`,
`templates/`, `data/`.

> The `.htaccess` files block HTTP access to `src/`, `config/`, `vendor/`,
> `templates/`, `data/` and `composer.*`. They do **not** affect PHP's filesystem
> reads, so the app keeps working normally.

### 6. SELinux (Fedora / RHEL only)

```bash
# Let php-fpm write the SQLite cache
chcon -R -t httpd_sys_rw_content_t data

# Let php-fpm make outbound HTTP requests (Reddit/YouTube/RT)
sudo setsebool -P httpd_can_network_connect on
```

Skipping the second command still renders cached data, but every fetch fails with
`cURL error 7 ... Could not connect to server`. See *Deployment notes*.

### 7. First run

Open the site (`http://localhost/`, or `http://localhost/RSS-Feed/` if it lives
under a parent document root). The first request fetches all feeds and builds
`data/feeds.sqlite` — this can take several seconds. Reddit may return `429` at
first; its items appear on subsequent refreshes.

### Quick start without a web server

```bash
php -S 127.0.0.1:8000 -t .
# http://127.0.0.1:8000/
```

The built-in server does not read `.htaccess`, so use it for local evaluation
only.

### Optional: scheduled refresh (no root)

If you can't change SELinux (or prefer not to fetch during web requests), run the
refresher from cron as your own user — CLI processes are unconfined and have
network access:

```bash
cd /path/to/RSS-Feed
php -r '$app = require "src/bootstrap.php"; $app["service"]->refreshStale();'
```

Add to `crontab -e` to refresh every 30 minutes:

```
*/30 * * * * cd /path/to/RSS-Feed && php -r '$app = require "src/bootstrap.php"; $app["service"]->refreshStale();' >/dev/null 2>&1
```

When the cron user differs from the web user, make sure both can write `data/`
(see step 4).

---

## Architecture

```
Browser
  │  GET /RSSFeed/ ?source=&range=                GET /RSSFeed/api/items.php ?source=&range=&page=
  ▼                                                         ▼
index.php                                            api/items.php
  └── require src/bootstrap.php  ─────────────────────────┘
          │  builds: PDO, FeedRepository, FeedFetcher, FeedService, config, feeds
          ▼
   FeedService::refreshStale()          ← runs on both entry points, respects TTL
          │
          ├─ FeedRepository::syncFeeds()      ensure every configured feed has a row
          ├─ flock data/refresh.lock          single-flight: concurrent callers skip
          ├─ FeedRepository::staleFeeds()     due feeds, minus retry backoff, jittered TTL
          ├─ host pacing                      per-refresh cap + cross-process host interval
          ├─ all stale Reddit feeds → ONE combined request (/r/a+b+c/.rss),
          │     items attributed back to each subreddit by link
          └─ for each other stale feed:
                 FeedFetcher::fetch()         Guzzle GET → laminas parse → normalize
                 FeedRepository::upsertItems() INSERT OR IGNORE (dedup by sha256(link))
                 FeedRepository::recordAttempt()  success → clears error/fail_count;
                                                  failure → fail_count++, next_retry_at

   FeedService::page(source, page, pageSize, since)   ← read path
          └─ FeedRepository::getItems()/countItems()  SQL over the SQLite cache
                 │
                 └─ View::render('cards')  server-rendered HTML
                    (api/items.php returns that HTML inside a JSON envelope)
```

**Request lifecycle (page load):** bootstrap → sync configured feeds → render
`templates/shell.php` (head, header, filters, skeleton loader) and **flush it**
→ refresh stale feeds → query one page → append the rendered
`templates/content.php` (which includes `templates/cards.php`) and drop the
loader. The shell paints while the refresh is still running; the loader never
outlives the fetch — the content chunk removes it as soon as it arrives.

**Request lifecycle (load more):** `assets/app.js` calls `api/items.php` with the
next `page` (plus current `source` and `range`), receives
`{ html, page, total, has_more, ... }`, appends `html` to `#feed`, and hides the
button when `has_more` is false.

### Caching & refresh semantics

- `items` and `feeds` live in `data/feeds.sqlite` (WAL mode, outside the web root).
- Only **enabled** feeds are refreshed and served — disabling one from the
  manage page stops both without deleting its cached items.
- A feed is **due** when its retry backoff has elapsed (`next_retry_at`) and its
  jittered TTL has passed: `now - last_attempt_at >= cache_ttl + crc32(url) % ttl_jitter`
  (jitter spreads expiry so feeds don't all refresh at once).
- **One refresh at a time.** `refreshStale()` takes a non-blocking lock on
  `data/refresh.lock`; concurrent page/API/cron requests serve the cache and
  return immediately instead of stampeding providers.
- **Rate limits are respected.** Throttled hosts (`throttled_hosts`) are limited
  to `throttled_host_max_per_refresh` fetches per refresh *and* to a per-host
  minimum interval enforced across processes. The gap between requests to the
  same host is also paced in-request (`host_interval_ms`).
- **Reddit is fetched as one request.** All due subreddits are combined into a
  single `reddit.com/r/a+b+c/.rss` call and the items are attributed back to each
  subreddit by their links. If that fails with 404/403 (a private or banned
  subreddit poisons the combined feed), the app falls back to individual fetches
  so the culprit is isolated.
- **429 handling.** A `429` carries the server's own window (`Retry-After`, or
  Reddit's `x-ratelimit-reset`); the feed is parked until then. Other failures
  back off exponentially (`error_retry_ttl` doubling per consecutive failure, up
  to `error_retry_cap`, plus jitter). Success resets the counters.
- Fetch failures are recorded (`last_error`, `fail_count`, `next_retry_at`) and
  **never delete cached items** — the dashboard keeps serving the last good data.

---

## Files

```
.
├── index.php                 Page entry point: refresh, filter, render page 1
├── manage.php                 Manual feed configuration (password protected)
├── api/items.php             JSON endpoint for "Load more" (returns rendered HTML)
├── config/
│   ├── app.php               Runtime settings (see Configuration)
│   └── feeds.php             The default feed list: [{source, label, url}, ...]
├── src/
│   ├── bootstrap.php         Wires config + classes together, returns the container
│   ├── Database.php          PDO SQLite connection + schema migration
│   ├── FeedFetcher.php       HTTP + parse + normalize a single feed
│   ├── FeedFetchException.php  Fetch failure carrying its HTTP status
│   ├── RateLimitException.php  HTTP 429, carries the server's retry window
│   ├── Backoff.php           Retry policy: server hint, else capped exponential + jitter
│   ├── FeedRepository.php    All SQL: upsert/query/count, staleness, feed management
│   ├── FeedService.php       Orchestration: refreshStale(), page(), sources()
│   ├── FeedManager.php       Manual selection: addReddit()/addYouTube()/remove()/setEnabled()
│   ├── ChannelResolver.php   YouTube @handle / channel URL → channel_id → Atom feed URL
│   ├── ManageAuth.php        manage.php gate: password, CSRF, flash messages
│   └── View.php              Escaping + render() + labels/colors/time formatting
├── templates/
│   ├── shell.php             Streamed first: head, header, filters, feed loader
│   ├── content.php           Streamed after the refresh: grid, empty state, Load more
│   ├── manage.php            Feed management page (login, add forms, feed list)
│   └── cards.php             One feed card partial (used by page + API)
├── assets/
│   ├── app.js                Load-more, relative timestamps, filter propagation
│   ├── manage.js             Confirm dialogs on the manage page
│   └── style.css             Design tokens + card/pill/header/manage styling
├── data/                     SQLite database (gitignored, created at runtime)
└── vendor/                   Composer deps (gitignored)
```

### Class reference

| Class / file | Responsibility | Key methods |
|---|---|---|
| `App\Database` | Creates the SQLite file/dir, applies schema, sets WAL + busy timeout | `connect()` |
| `App\FeedFetcher` | Fetch one feed (with host throttle + 5xx retry), parse with laminas, extract media fields, normalize to a plain array | `fetch()` |
| `App\Backoff` | Turns a failure (fail count + optional server hint) into the next retry time | `nextRetryAt()` |
| `App\FeedRepository` | Every SQL statement; returns arrays/DTO-less rows | `syncFeeds()`, `enabledFeeds()`, `staleFeeds()`, `hostLastAttempt()`, `upsertItems()`, `recordAttempt()`, `getItems()`, `countItems()`, `addFeed()`, `setFeedEnabled()`, `deleteFeed()`, `feedsStatus()` |
| `App\FeedService` | Business flow: single-flight refresh (throttled hosts, combined Reddit fetch) + paginated reads | `refreshStale()`, `page()`, `sources()`, `feedsStatus()` |
| `App\FeedManager` | Manual selection changes; validates new feeds with one immediate fetch (404 rejects the entry) | `addReddit()`, `addYouTube()`, `remove()`, `setEnabled()` |
| `App\ChannelResolver` | YouTube input (`@handle`, `UC…` ID or channel URL) → Atom feed URL; resolves handles once via the channel page (youtube.com only) | `resolve()` |
| `App\ManageAuth` | Session gate for `manage.php`: password login, CSRF tokens, flash messages | `login()`, `authed()`, `csrfToken()`, `checkCsrf()` |
| `App\View` | Output escaping, template rendering, source labels/colors, ranges, `timeAgo()` | `e()`, `render()`, `sourceLabel()`, `sourceColor()`, `rangeLabel()`, `timeAgo()` |
| `src/bootstrap.php` | Builds and returns `['config','feeds','pdo','repository','fetcher','resolver','manager','service']` | — |

`FeedFetcher` notes: laminas does not expose the YouTube `media:` extension, so
`extractExtras()` does a second DOM/XPath pass over the raw XML to pull
`media:thumbnail` and `media:description`. Thumbnail resolution order is
`media:thumbnail` → image `<enclosure>` → YouTube `i.ytimg.com` fallback.
Descriptions are stripped to plain text and truncated (XSS-safe; the template
escapes again on output).

---

## Data model

`items` — one row per article/video/post:

| column | notes |
|---|---|
| `id` | `sha256(link)`, primary key (dedup) |
| `source` | `youtube` / `reddit` / `rt` |
| `feed_label`, `feed_url` | which feed it came from |
| `title`, `link`, `author` | |
| `published_at` | ISO-8601 UTC (nullable) |
| `description` | plain text, truncated |
| `thumbnail` | image URL (nullable) |
| `fetched_at` | ISO-8601 UTC |

Indexes: `idx_items_published(published_at DESC)`,
`idx_items_source(source, published_at DESC)`.

`feeds` — one row per feed (config or manually added) + selection/health state:

| column | notes |
|---|---|
| `url` | primary key |
| `source`, `label` | from `config/feeds.php` or the manage page |
| `last_attempt_at`, `last_success_at` | ISO-8601 UTC |
| `last_error` | last failure message, `NULL` on success |
| `origin` | `config` (from `config/feeds.php`) or `manual` (added via manage.php) |
| `enabled` | `1` = selected; disabled feeds are neither refreshed nor shown |
| `fail_count` | consecutive failures; resets on success (drives exponential backoff) |
| `next_retry_at` | ISO-8601 UTC; the feed is skipped until this time (rate-limit / backoff) |

Only items whose `feed_url` belongs to an enabled feed are served — disabling or
removing a feed hides its cached stories without deleting them.

---

## Configuration

`config/app.php` returns an array:

| key | default | meaning |
|---|---|---|
| `db_path` | `data/feeds.sqlite` | SQLite file location |
| `cache_ttl` | `1800` | seconds before a healthy feed is re-fetched |
| `error_retry_ttl` | `300` | base backoff for failed feeds (doubles per consecutive failure, plus jitter) |
| `error_retry_cap` | `7200` | maximum retry backoff for a failing feed |
| `ttl_jitter` | `300` | random seconds added to `cache_ttl` so feeds don't all expire at once |
| `host_interval_ms` | `1000` | min gap between requests to the same host within one refresh |
| `throttled_hosts` | `['www.reddit.com' => 60, 'www.youtube.com' => 15]` | host => min seconds between requests (enforced across processes) |
| `throttled_host_max_per_refresh` | `1` | max feeds of a throttled host fetched per refresh (combined Reddit = 1) |
| `page_size` | `20` | items per page |
| `ranges` | `1h,2h,6h,24h,7d` | time-filter windows (key → seconds) |
| `request_timeout` / `connect_timeout` | `15` / `10` | Guzzle timeouts |
| `user_agent` | `RSSFeed/1.0 (...)` | sent to providers (matters for Reddit) |
| `manage_password` | `rssfeed-admin` | password for `manage.php`; plain or `password_hash()` value, empty = management disabled |

`config/feeds.php` returns a list of feeds:

```php
[
    'source' => 'reddit',                    // groups the item + drives its color
    'label'  => 'r/homelab',                 // shown under the badge
    'url'    => 'https://www.reddit.com/r/homelab/.rss',
]
```

**YouTube note:** YouTube serves Atom only by `channel_id`, not `@handle`.
Resolve a handle once (fetch `https://www.youtube.com/@handle` and read
`<meta itemprop="identifier">` or the canonical channel link) and store
`https://www.youtube.com/feeds/videos.xml?channel_id=...`.

---

## Managing feeds (manual configuration)

`manage.php` (linked as **Manage** in the dashboard header) configures the
subreddit/YouTube selection from the browser. It is password protected:

- Set `manage_password` in `config/app.php`. It accepts a plain password (fine
  for a LAN/dev box) or a `password_hash()` value for better hygiene:

  ```bash
  php -r 'echo password_hash("your secret", PASSWORD_DEFAULT), PHP_EOL;'
  ```

- An empty value disables the page (it shows a setup notice instead).

What you can do there:

| Action | Input | Behavior |
|---|---|---|
| Add subreddit | `homelab`, `r/homelab`, or any `reddit.com/r/…` URL | Normalized to `https://www.reddit.com/r/<name>/.rss`, labeled `r/<name>` |
| Add YouTube channel | `@handle`, `UC…` channel ID, or any channel URL | Handle resolved to `channel_id` once, stored as the Atom feed URL; the label becomes the real channel title |
| Disable / enable | toggle per feed | Disabled feeds are neither refreshed nor shown (their cached stories are kept and reappear when re-enabled) |
| Remove | manual feeds | Deleted from the selection; re-add anytime |
| Remove | config feeds | Hidden (`enabled = 0`) — `config/feeds.php` re-registers them on every sync, so they cannot be deleted from the UI |

Semantics worth knowing:

- Adding a feed **fetches it immediately** (bypassing the per-refresh Reddit
  host cap): an `HTTP 404` rejects the entry ("not found, nothing was added"),
  while a rate-limited (`429`) add is kept and fills in on a later refresh.
- Re-adding an existing feed URL **re-enables** it.
- Handle resolution only ever requests `youtube.com` pages — user input can
  never point the resolver at arbitrary URLs.
- All POST actions are CSRF-protected (session token) and use POST-then-redirect.
- `config/feeds.php` remains the default set and keeps working as before; the
  manage page layers manual feeds and on/off state on top of it (persisted in
  the `feeds` table, so they survive restarts).

---

## Routes & query parameters

| Route | Params | Returns |
|---|---|---|
| `index.php` (`/RSSFeed/`) | `source` (optional), `range` (optional) | full HTML page, page 1 |
| `api/items.php` | `page` (default 1), `source`, `range` | JSON `{ html, page, page_size, count, total, has_more, range, refresh }` |
| `manage.php` | — (POST: `login`, `logout`, `add_reddit`, `add_youtube`, `toggle`, `remove`) | management page (password gated) |

- `source` must match a `source` in `config/feeds.php`, else ignored.
- `range` must be a key in `ranges`, else ignored.
- Both filters are preserved across navigation and across "Load more".

---

## Adding feeds and sources

**New feed (existing source):** either add it on the **manage page** (no code
editing, see *Managing feeds*) or add an entry to `config/feeds.php`.
`syncFeeds()` registers config entries automatically on the next request; they
will be fetched on the next refresh.

**New source type** (e.g. `hackernews`): add feeds with `'source' => 'hackernews'`,
then teach the UI its identity in `src/View.php`:

```php
sourceLabel('hackernews')  // tab / badge text
sourceColor('hackernews')  // hex used for the card spine, chip and active pill
```

The source tab and filter pill are generated automatically from `sources()`.

### Colour system (`assets/style.css`)

The dashboard has **no house colour**. One ink ramp (`--bg` → `--ink`) carries
every neutral surface, line, and label, and the only saturated colour on screen
belongs to the sources — used strictly as a signal: filter pill, card spine,
source chip, timestamp, feed health. Two rules hold it together:

- **Bright fill, deep ink type.** Every filled state (active pill, source chip,
  primary action) is a bright fill with `--ink-deep` type — no white text on
  saturated fills, no gradient fills.
- **One hue per source, mirrored in two places.** `:root` defines the token
  (`--youtube`, `--reddit`, `--rt`) and `View::sourceColor()` returns the same
  hex for the inline `--src`/`--pill` values the templates emit. Change both.

The header rail (`.wire-rail`) is the colour legend: one segment per source, in
order.

---

## Deployment notes (Apache + php-fpm on SELinux, e.g. Fedora)

1. **Document root = project root.** `index.php` is served for `/RSSFeed/`; the
   built-in files under `api/` and `assets/` are served directly.
2. **Protect internals.** `.htaccess` files deny HTTP access to `src/`, `config/`,
   `vendor/`, `templates/`, `data/`, and `composer.*`. These only affect HTTP;
   PHP still reads the files from disk. Directory listing is disabled.
3. **Let the web user write the cache.** `data/` must be writable by the php-fpm
   user (e.g. `apache`), and on SELinux it needs the `httpd_sys_rw_content_t`
   type:

   ```bash
   chmod 1777 data
   chcon -R -t httpd_sys_rw_content_t data
   ```

4. **Allow outbound HTTP.** On SELinux, php-fpm runs in `httpd_t` and outbound
   connections are denied by default (symptom: `cURL error 7 ... Could not
   connect to server`). Enable it once:

   ```bash
   sudo setsebool -P httpd_can_network_connect on
   ```

   Without this, the app still renders the cached data but cannot refresh. As an
   alternative (no root), run the fetcher from a user cron/CLI process instead —
   CLI processes run unconfined and do have network access.

---

## Troubleshooting

| Symptom | Cause / fix |
|---|---|
| `cURL error 7 ... Could not connect` | SELinux: enable `httpd_can_network_connect` (see above). |
| Reddit feeds empty / `HTTP 429` | Reddit's unauthenticated limit is tight (~1 request/min/IP). The app now combines all subreddits into one request, honors `x-ratelimit-reset`, backs off, and serves cached items meanwhile. If it persists, set a real `user_agent` contact and/or lower the Reddit refresh rate (`throttled_hosts`). |
| `403` on `/src/...`, `/config/...` | Expected — `.htaccess` denies those paths. |
| `unable to open database file` | `data/` not writable by the web user, or wrong SELinux type on `data/`. |
| Blank page / 500 | Check the PHP error log; run `php -l` on changed files. |
| Stale data | Feeds refresh only when older than `cache_ttl`; delete `data/feeds.sqlite` to force a rebuild, or lower `cache_ttl`. |
| New source has no color | Add a case to `View::sourceLabel()` / `View::sourceColor()`. |

---

## Conventions

- Strict types (`declare(strict_types=1)`) in all PHP files; PSR-12 style.
- All SQL is parameterized; all output is escaped via `App\View::e()`.
- No comments in code unless they explain non-obvious behavior.
- Tests: none yet — verify manually via the two entry points (page + API) and
  `php -l`.