RSS-Feed/README.md

# RSS Feed Dashboard

A small PHP dashboard that aggregates RSS/Atom feeds (YouTube channels, Reddit
subreddits, RT news) into a single stream, newest first, with source and
time-range filters and AJAX pagination.

No framework, no build step. PHP + SQLite for caching, Bootstrap 5 for the UI.

---

## What it does

- Pulls **YouTube** (Atom), **Reddit** (RSS 2.0) and **RT** (RSS) feeds.
- Merges every item into one timeline ordered by `published_at` (fallback
  `fetched_at`), newest first.
- Filters by **source** (`youtube` / `reddit` / `rt`) and **time range**
  (all time, 1h, 2h, 6h, 24h, 7d).
- Paginates with a **"Load more stories"** button (AJAX, 20 per page).
- Caches everything in **SQLite** so page loads never wait on the network; feeds
  are refreshed on demand when stale (default TTL 30 min).

---

## Tech stack

| Layer      | Choice                                                        |
|------------|---------------------------------------------------------------|
| Runtime    | PHP 8.1+ (`ext-pdo`, `ext-curl`)                              |
| Parsing    | [`laminas/laminas-feed`](https://github.com/laminas/laminas-feed) |
| HTTP       | [`guzzlehttp/guzzle`](https://github.com/guzzle/guzzle)       |
| Storage    | SQLite via PDO (WAL mode)                                     |
| Front end  | Bootstrap 5 (CDN) + vanilla JS, custom `assets/style.css`     |
| Autoload   | Composer PSR-4, `App\` → `src/`                               |

---

## Installation

### Requirements

- PHP **8.1+** with `pdo_sqlite`, `curl`, `dom`, `simplexml`, `mbstring`
- Composer (or the bundled `composer.phar`)
- A web server with PHP (Apache/nginx + php-fpm) — or PHP's built-in server for a quick look
- No database server: SQLite is embedded

Verify PHP:

```bash
php -v
php -m | grep -E 'pdo_sqlite|curl|dom|simplexml|mbstring'
```

### 1. Get the code

```bash
git clone ssh://git@192.168.0.11:222/harry/RSS-Feed.git
cd RSS-Feed
```

### 2. Install dependencies

```bash
php composer.phar install
```

This populates `vendor/`. If Composer is installed globally, `composer install`
works the same. (`vendor/` and the SQLite DB are gitignored.)

### 3. Configure

- **Feeds** — edit `config/feeds.php` (see *Configuration* for the format and the
  YouTube `@handle` → `channel_id` note).
- **Runtime** — optionally tune `config/app.php` (page size, TTLs, ranges,
  timeouts, user agent).

### 4. Create the cache directory

```bash
mkdir -p data
```

The PHP/web user must be able to write `data/feeds.sqlite` (SQLite also creates
`-wal`/`-shm` files there). Make it writable by the web user:

```bash
chmod 775 data
# or, when the web user differs from the owner:
chmod 1777 data
```

Cleaner still, let the web server's user own it:

```bash
sudo chown -R www-data:www-data data   # Debian/Ubuntu
# sudo chown -R apache:apache data     # RHEL/Fedora
```

### 5. Point a web server at the project

The **project root is the web root**. Apache example:

```apache
<VirtualHost *:80>
    ServerName localhost
    DocumentRoot "/path/to/RSS-Feed"

    <Directory "/path/to/RSS-Feed">
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>
</VirtualHost>
```

Ensure `index.php` is a directory index (`DirectoryIndex index.php`) and that
`.htaccess` overrides are allowed (`AllowOverride All`) so the bundled protection
files take effect. For nginx: set `root /path/to/RSS-Feed;`, route requests to
`index.php` / `api/items.php`, and deny `src/`, `config/`, `vendor/`,
`templates/`, `data/`.

> The `.htaccess` files block HTTP access to `src/`, `config/`, `vendor/`,
> `templates/`, `data/` and `composer.*`. They do **not** affect PHP's filesystem
> reads, so the app keeps working normally.

### 6. SELinux (Fedora / RHEL only)

```bash
# Let php-fpm write the SQLite cache
chcon -R -t httpd_sys_rw_content_t data

# Let php-fpm make outbound HTTP requests (Reddit/YouTube/RT)
sudo setsebool -P httpd_can_network_connect on
```

Skipping the second command still renders cached data, but every fetch fails with
`cURL error 7 ... Could not connect to server`. See *Deployment notes*.

### 7. First run

Open the site (`http://localhost/`, or `http://localhost/RSS-Feed/` if it lives
under a parent document root). The first request fetches all feeds and builds
`data/feeds.sqlite` — this can take several seconds. Reddit may return `429` at
first; its items appear on subsequent refreshes.

### Quick start without a web server

```bash
php -S 127.0.0.1:8000 -t .
# http://127.0.0.1:8000/
```

The built-in server does not read `.htaccess`, so use it for local evaluation
only.

### Optional: scheduled refresh (no root)

If you can't change SELinux (or prefer not to fetch during web requests), run the
refresher from cron as your own user — CLI processes are unconfined and have
network access:

```bash
cd /path/to/RSS-Feed
php -r '$app = require "src/bootstrap.php"; $app["service"]->refreshStale();'
```

Add to `crontab -e` to refresh every 30 minutes:

```
*/30 * * * * cd /path/to/RSS-Feed && php -r '$app = require "src/bootstrap.php"; $app["service"]->refreshStale();' >/dev/null 2>&1
```

When the cron user differs from the web user, make sure both can write `data/`
(see step 4).

---

## Architecture

```
Browser
  │  GET /RSSFeed/ ?source=&range=                GET /RSSFeed/api/items.php ?source=&range=&page=
  ▼                                                         ▼
index.php                                            api/items.php
  └── require src/bootstrap.php  ─────────────────────────┘
          │  builds: PDO, FeedRepository, FeedFetcher, FeedService, config, feeds
          ▼
   FeedService::refreshStale()          ← runs on both entry points, respects TTL
          │
          ├─ FeedRepository::syncFeeds()      ensure every configured feed has a row
          ├─ FeedRepository::staleFeeds()     feeds older than TTL (or error cooldown)
          └─ for each stale feed:
                 FeedFetcher::fetch()         Guzzle GET → laminas parse → normalize
                 FeedRepository::upsertItems() INSERT OR IGNORE (dedup by sha256(link))
                 FeedRepository::recordAttempt()  success/error + timestamps

   FeedService::page(source, page, pageSize, since)   ← read path
          └─ FeedRepository::getItems()/countItems()  SQL over the SQLite cache
                 │
                 └─ View::render('layout' | 'cards')  server-rendered HTML
                    (api/items.php returns that HTML inside a JSON envelope)
```

**Request lifecycle (page load):** bootstrap → refresh stale feeds → validate
`source`/`range` → query one page → render `templates/layout.php` which includes
`templates/cards.php`.

**Request lifecycle (load more):** `assets/app.js` calls `api/items.php` with the
next `page` (plus current `source` and `range`), receives
`{ html, page, total, has_more, ... }`, appends `html` to `#feed`, and hides the
button when `has_more` is false.

### Caching & refresh semantics

- `items` and `feeds` live in `data/feeds.sqlite` (WAL mode, outside the web root).
- A feed is **stale** when `now - last_attempt_at >= cache_ttl`, or, if its last
  attempt errored, when `now - last_attempt_at >= error_retry_ttl + jitter`
  (jitter derived from `crc32(url)` so rate-limited feeds don't all retry at once).
- Fetch failures are recorded (`last_error`) and **never delete cached items** —
  the dashboard keeps serving the last good data.
- Reddit rate-limits hard. `throttled_hosts` caps how many feeds of a given host
  are fetched per refresh (`throttled_host_max_per_refresh`), and requests to the
  same host are spaced by `host_interval_ms`. Only 5xx responses retry in-request.

---

## Files

```
.
├── index.php                 Page entry point: refresh, filter, render page 1
├── api/items.php             JSON endpoint for "Load more" (returns rendered HTML)
├── config/
│   ├── app.php               Runtime settings (see Configuration)
│   └── feeds.php             The feed list: [{source, label, url}, ...]
├── src/
│   ├── bootstrap.php         Wires config + classes together, returns the container
│   ├── Database.php          PDO SQLite connection + schema migration
│   ├── FeedFetcher.php       HTTP + parse + normalize a single feed
│   ├── FeedRepository.php    All SQL: upsert/query/count, staleness, attempt state
│   ├── FeedService.php       Orchestration: refreshStale(), page(), sources()
│   └── View.php              Escaping + render() + labels/colors/time formatting
├── templates/
│   ├── layout.php            Full page shell, filters, grid, empty state, Load more
│   └── cards.php             One feed card partial (used by page + API)
├── assets/
│   ├── app.js                Load-more, relative timestamps, filter propagation
│   └── style.css             Design tokens + card/pill/header styling
├── data/                     SQLite database (gitignored, created at runtime)
└── vendor/                   Composer deps (gitignored)
```

### Class reference

| Class / file | Responsibility | Key methods |
|---|---|---|
| `App\Database` | Creates the SQLite file/dir, applies schema, sets WAL + busy timeout | `connect()` |
| `App\FeedFetcher` | Fetch one feed (with host throttle + 5xx retry), parse with laminas, extract media fields, normalize to a plain array | `fetch()` |
| `App\FeedRepository` | Every SQL statement; returns arrays/DTO-less rows | `syncFeeds()`, `staleFeeds()`, `upsertItems()`, `recordAttempt()`, `getItems()`, `countItems()`, `feedsStatus()` |
| `App\FeedService` | Business flow: refresh policy + paginated reads | `refreshStale()`, `page()`, `sources()`, `feedsStatus()` |
| `App\View` | Output escaping, template rendering, source labels/colors, ranges, `timeAgo()` | `e()`, `render()`, `sourceLabel()`, `sourceColor()`, `rangeLabel()`, `timeAgo()` |
| `src/bootstrap.php` | Builds and returns `['config','feeds','pdo','repository','fetcher','service']` | — |

`FeedFetcher` notes: laminas does not expose the YouTube `media:` extension, so
`extractExtras()` does a second DOM/XPath pass over the raw XML to pull
`media:thumbnail` and `media:description`. Thumbnail resolution order is
`media:thumbnail` → image `<enclosure>` → YouTube `i.ytimg.com` fallback.
Descriptions are stripped to plain text and truncated (XSS-safe; the template
escapes again on output).

---

## Data model

`items` — one row per article/video/post:

| column | notes |
|---|---|
| `id` | `sha256(link)`, primary key (dedup) |
| `source` | `youtube` / `reddit` / `rt` |
| `feed_label`, `feed_url` | which feed it came from |
| `title`, `link`, `author` | |
| `published_at` | ISO-8601 UTC (nullable) |
| `description` | plain text, truncated |
| `thumbnail` | image URL (nullable) |
| `fetched_at` | ISO-8601 UTC |

Indexes: `idx_items_published(published_at DESC)`,
`idx_items_source(source, published_at DESC)`.

`feeds` — one row per configured feed + health state:

| column | notes |
|---|---|
| `url` | primary key |
| `source`, `label` | from `config/feeds.php` |
| `last_attempt_at`, `last_success_at` | ISO-8601 UTC |
| `last_error` | last failure message, `NULL` on success |

---

## Configuration

`config/app.php` returns an array:

| key | default | meaning |
|---|---|---|
| `db_path` | `data/feeds.sqlite` | SQLite file location |
| `cache_ttl` | `1800` | seconds before a healthy feed is re-fetched |
| `error_retry_ttl` | `120` | base retry delay for failed feeds (plus jitter) |
| `host_interval_ms` | `1000` | min gap between requests to the same host |
| `throttled_hosts` | `['www.reddit.com']` | hosts fetched sparingly |
| `throttled_host_max_per_refresh` | `1` | max feeds of a throttled host per refresh |
| `page_size` | `20` | items per page |
| `ranges` | `1h,2h,6h,24h,7d` | time-filter windows (key → seconds) |
| `request_timeout` / `connect_timeout` | `15` / `10` | Guzzle timeouts |
| `user_agent` | `RSSFeed/1.0 (...)` | sent to providers (matters for Reddit) |

`config/feeds.php` returns a list of feeds:

```php
[
    'source' => 'reddit',                    // groups the item + drives its color
    'label'  => 'r/homelab',                 // shown under the badge
    'url'    => 'https://www.reddit.com/r/homelab/.rss',
]
```

**YouTube note:** YouTube serves Atom only by `channel_id`, not `@handle`.
Resolve a handle once (fetch `https://www.youtube.com/@handle` and read
`<meta itemprop="identifier">` or the canonical channel link) and store
`https://www.youtube.com/feeds/videos.xml?channel_id=...`.

---

## Routes & query parameters

| Route | Params | Returns |
|---|---|---|
| `index.php` (`/RSSFeed/`) | `source` (optional), `range` (optional) | full HTML page, page 1 |
| `api/items.php` | `page` (default 1), `source`, `range` | JSON `{ html, page, page_size, count, total, has_more, range, refresh }` |

- `source` must match a `source` in `config/feeds.php`, else ignored.
- `range` must be a key in `ranges`, else ignored.
- Both filters are preserved across navigation and across "Load more".

---

## Adding feeds and sources

**New feed (existing source):** add an entry to `config/feeds.php`. `syncFeeds()`
registers it automatically on the next request; it will be fetched on the next
refresh.

**New source type** (e.g. `hackernews`): add feeds with `'source' => 'hackernews'`,
then teach the UI its identity in `src/View.php`:

```php
sourceLabel('hackernews')  // tab / badge text
sourceColor('hackernews')  // hex used for the card spine, chip and active pill
```

The source tab and filter pill are generated automatically from `sources()`.

---

## Deployment notes (Apache + php-fpm on SELinux, e.g. Fedora)

1. **Document root = project root.** `index.php` is served for `/RSSFeed/`; the
   built-in files under `api/` and `assets/` are served directly.
2. **Protect internals.** `.htaccess` files deny HTTP access to `src/`, `config/`,
   `vendor/`, `templates/`, `data/`, and `composer.*`. These only affect HTTP;
   PHP still reads the files from disk. Directory listing is disabled.
3. **Let the web user write the cache.** `data/` must be writable by the php-fpm
   user (e.g. `apache`), and on SELinux it needs the `httpd_sys_rw_content_t`
   type:

   ```bash
   chmod 1777 data
   chcon -R -t httpd_sys_rw_content_t data
   ```

4. **Allow outbound HTTP.** On SELinux, php-fpm runs in `httpd_t` and outbound
   connections are denied by default (symptom: `cURL error 7 ... Could not
   connect to server`). Enable it once:

   ```bash
   sudo setsebool -P httpd_can_network_connect on
   ```

   Without this, the app still renders the cached data but cannot refresh. As an
   alternative (no root), run the fetcher from a user cron/CLI process instead —
   CLI processes run unconfined and do have network access.

---

## Troubleshooting

| Symptom | Cause / fix |
|---|---|
| `cURL error 7 ... Could not connect` | SELinux: enable `httpd_can_network_connect` (see above). |
| Reddit feeds empty / `HTTP 429` | Reddit's unauthenticated limit (~1 req/window). It self-heals via `error_retry_ttl` + jitter and the per-host cap; cached items still show. |
| `403` on `/src/...`, `/config/...` | Expected — `.htaccess` denies those paths. |
| `unable to open database file` | `data/` not writable by the web user, or wrong SELinux type on `data/`. |
| Blank page / 500 | Check the PHP error log; run `php -l` on changed files. |
| Stale data | Feeds refresh only when older than `cache_ttl`; delete `data/feeds.sqlite` to force a rebuild, or lower `cache_ttl`. |
| New source has no color | Add a case to `View::sourceLabel()` / `View::sourceColor()`. |

---

## Conventions

- Strict types (`declare(strict_types=1)`) in all PHP files; PSR-12 style.
- All SQL is parameterized; all output is escaped via `App\View::e()`.
- No comments in code unless they explain non-obvious behavior.
- Tests: none yet — verify manually via the two entry points (page + API) and
  `php -l`.