docs: multi-provider LLM layer plan (DeepSeek news / Anthropic panel)
Reference plan only - no code changes. Describes the BaseLLMClient refactor, AnthropicClient on the official SDK, per-consumer routing via a config.json "llm" block, tests, commit sequence, and effort estimate for the day we route the agent panel to Claude while news stays on DeepSeek. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SisdfkxnXNL9iZsXxN3T15
This commit is contained in:
Harwinder Sekhon
@@ -0,0 +1,252 @@
|
||||
================================================================================
|
||||
MULTI-PROVIDER LLM LAYER - PLAN
|
||||
(DeepSeek for news summary, Anthropic Claude for agent panel)
|
||||
Written 2026-08-08. Plan only - no code has been changed.
|
||||
================================================================================
|
||||
|
||||
WHY
|
||||
---
|
||||
Today every LLM feature (news enrichment + agent panel) goes through one
|
||||
hardwired client: llm_client.py -> DeepSeekClient, a module singleton reached
|
||||
via get_llm_client(). This plan describes how to route different features to
|
||||
different providers - e.g. keep DeepSeek for cheap high-volume news scoring,
|
||||
put the agent panel on Anthropic Claude for better reasoning.
|
||||
|
||||
The repo already solved this problem once for market data: the providers/
|
||||
package (BaseDataProvider owns shared machinery, subclasses implement only
|
||||
_fetch_raw, a registry + one config knob selects the provider). This plan
|
||||
mirrors that proven pattern at the LLM layer, scaled down to a single module.
|
||||
|
||||
BLAST RADIUS (verified by grep)
|
||||
-------------------------------
|
||||
get_llm_client() call sites:
|
||||
- news_engine.py ....... 4 sites
|
||||
- agent_panel.py ....... 3 sites (+1 timeout getattr)
|
||||
- telegram_bot.py ...... 1 site (/status health; reads private _count,
|
||||
_count_day, _daily_cap fields)
|
||||
- test_all.py .......... 41 locations stub via lc._client = _StubLLM(...)
|
||||
Out of scope: static/news_summary.php (own PHP DeepSeek call + key file,
|
||||
separate deployment).
|
||||
|
||||
DESIGN INVARIANTS (carried over unchanged)
|
||||
------------------------------------------
|
||||
1. Fail-safe contract: every call returns Optional[Dict]; any failure -> None;
|
||||
callers keep their non-LLM fallbacks; LLM output never gates signals.
|
||||
2. Single attempt, no retries (news cache cadence is the retry); per-provider
|
||||
UTC daily cap; warn-once logging; health stamps (last_ok, last_error).
|
||||
3. Static system prompts stay byte-identical constants - provider-neutral,
|
||||
and they map to prompt caching on BOTH providers (DeepSeek prefix cache;
|
||||
Anthropic cache_control ephemeral).
|
||||
4. Test compatibility: all 41 existing lc._client stub sites keep working
|
||||
with zero edits.
|
||||
|
||||
================================================================================
|
||||
1. llm_client.py REFACTOR (~195 -> ~430 lines, no new files)
|
||||
================================================================================
|
||||
|
||||
1a. BaseLLMClient - shared machinery (extracted from today's DeepSeekClient)
|
||||
- __init__(api_key, model, timeout, daily_cap, enabled)
|
||||
- public: name ("deepseek"/"anthropic"), timeout property,
|
||||
usage() -> (used_today, cap) # so telegram_bot stops reading privates
|
||||
- owns: available property, _warn_once, _under_cap() (UTC-day counter,
|
||||
lock), health stamps, and the template method:
|
||||
|
||||
def chat_json(self, system_prompt, user_payload):
|
||||
if not self.available or not self._under_cap():
|
||||
return None
|
||||
try:
|
||||
return self._request(system_prompt, user_payload) # hook
|
||||
except <provider-agnostic error mapping>:
|
||||
# stamps last_error, returns None
|
||||
|
||||
- subclasses implement ONLY _request(system_prompt, payload) -> dict|None.
|
||||
|
||||
1b. DeepSeekClient(BaseLLMClient) - behavior-identical move
|
||||
Current requests-based logic relocates into _request: same URL,
|
||||
thinking:disabled, response_format:json_object, temperature 0,
|
||||
max_tokens 2000, sort_keys=True payload, _parse_response.
|
||||
Review criterion: Module 19 tests pass UNMODIFIED (they patch
|
||||
client._session, which stays).
|
||||
|
||||
1c. AnthropicClient(BaseLLMClient) - new, official SDK
|
||||
- Official `anthropic` Python SDK (add to requirements.txt), NOT raw
|
||||
requests. LAZY IMPORT inside __init__: package missing -> warn-once +
|
||||
client unavailable, so DeepSeek-only deployments never install it.
|
||||
- Constructor: anthropic.Anthropic(api_key=..., timeout=<configured>,
|
||||
max_retries=0) # house rule: single attempt
|
||||
- _request:
|
||||
resp = self._client.messages.create(
|
||||
model=self._model, # default "claude-opus-5"
|
||||
max_tokens=4000, # covers thinking + JSON reply
|
||||
output_config={"effort": "low"}, # cheap structured takes
|
||||
system=[{"type": "text", "text": system_prompt,
|
||||
"cache_control": {"type": "ephemeral"}}],
|
||||
messages=[{"role": "user",
|
||||
"content": json.dumps(user_payload, sort_keys=True)}],
|
||||
)
|
||||
if resp.stop_reason == "refusal": # Opus 5: HTTP 200, check first
|
||||
return None
|
||||
text = first text block in resp.content
|
||||
return self._parse_json(text) # tolerant: strip ```json fences,
|
||||
# json.loads, dict check
|
||||
- DELIBERATELY ABSENT (would 400 on Claude Opus 5): temperature (removed
|
||||
on Opus 5 - do NOT copy from the DeepSeek client), budget_tokens,
|
||||
prefills, response_format.
|
||||
- Error mapping: APIConnectionError -> "network",
|
||||
APIStatusError -> "http_<code>", parse fail -> "parse"
|
||||
(same last_error kinds the /status display already understands).
|
||||
- Defaults: model claude-opus-5 (config-overridable, e.g. claude-haiku-4-5
|
||||
for ~5x cheaper), timeout 30s (Claude slower than V4-Flash), own
|
||||
daily_request_cap 500.
|
||||
|
||||
1d. Registry + per-consumer routing
|
||||
_PROVIDERS = {"deepseek": DeepSeekClient, "anthropic": AnthropicClient}
|
||||
LLM_CONSUMERS = ("news", "agent_panel")
|
||||
_clients = {} # keyed by PROVIDER name -> two consumers on the
|
||||
# same provider share one instance and one cap
|
||||
_client = None # kept: GLOBAL TEST OVERRIDE
|
||||
|
||||
def get_llm_client(consumer="news"):
|
||||
if _client is not None: # tests: lc._client = stub
|
||||
return _client # -> shadows ALL consumers
|
||||
name = get_app_config().llm_routing.get(consumer, "deepseek")
|
||||
create-once per provider, return _clients[name]
|
||||
|
||||
def reset_llm_client(): # clears _client AND _clients
|
||||
|
||||
Keeping _client as a documented test-override global is what makes the
|
||||
41 existing stub sites work untouched.
|
||||
|
||||
================================================================================
|
||||
2. CONFIG (app_config.py + config.json.example)
|
||||
================================================================================
|
||||
- New top-level "anthropic" block, mirroring "deepseek":
|
||||
{api_key, model, enabled, timeout_seconds, daily_request_cap}
|
||||
plus anthropic_api_key() resolver: ANTHROPIC_API_KEY env -> config.json -> "".
|
||||
- New top-level "llm" ROUTING block:
|
||||
{"news": "deepseek", "agent_panel": "deepseek"}
|
||||
Validation: keys within {news, agent_panel}; values within registered
|
||||
provider names. Absent block -> {} -> everything on DeepSeek (today's
|
||||
behavior; zero-migration default). AppConfig.llm_routing: Dict[str, str].
|
||||
|
||||
- The target state is then PURE CONFIG - no code changes on switch day:
|
||||
|
||||
"llm": { "news": "deepseek", "agent_panel": "anthropic" },
|
||||
"anthropic": { "api_key": "", "model": "claude-opus-5", "enabled": true,
|
||||
"timeout_seconds": 30, "daily_request_cap": 500 }
|
||||
|
||||
================================================================================
|
||||
3. CONSUMER EDITS (mechanical, ~10 lines total)
|
||||
================================================================================
|
||||
- news_engine.py: 4 call sites -> get_llm_client("news")
|
||||
- agent_panel.py: 3 call sites -> get_llm_client("agent_panel");
|
||||
getattr(client, "_timeout", ...) -> public client.timeout
|
||||
- telegram_bot.py _news_health_lines: iterate the DISTINCT providers actually
|
||||
routed (news + panel); per provider print "<name>: used/cap calls today"
|
||||
via the new usage() accessor instead of private fields. DeepSeek keeps its
|
||||
"headlines scored" extra; cap/error warnings work off base-class stamps.
|
||||
- NO changes to prompts, payload builders, caches, or the panel's own
|
||||
max_panels_per_day budget - all provider-agnostic already.
|
||||
|
||||
================================================================================
|
||||
4. TESTS (new "Module 24: LLM provider routing", ~12 tests, ~200 lines)
|
||||
================================================================================
|
||||
Reuse _StubLLM; new _FakeAnthropic (fake messages.create returning a stub
|
||||
response object) injected as client._client.
|
||||
|
||||
1. app_config: anthropic block parses/validates/defaults empty.
|
||||
2. app_config: llm routing block - valid map round-trips; unknown consumer
|
||||
key, unknown provider name, non-dict -> AppConfigError; absent -> {}.
|
||||
3. anthropic_api_key resolves env -> config.json -> empty.
|
||||
4. Base contract on BOTH subclasses (parametrized): keyless/disabled -> no
|
||||
call; daily cap blocks + UTC-roll reset; failure stamps last_error,
|
||||
success stamps last_ok.
|
||||
5. AnthropicClient parse: clean JSON -> dict; ```json-fenced -> dict;
|
||||
non-JSON -> None; stop_reason "refusal" -> None; no text block -> None.
|
||||
6. AnthropicClient request shape: NO temperature key, max_retries=0,
|
||||
cache_control present on system block (assert on captured kwargs).
|
||||
7. AnthropicClient with `anthropic` package missing (simulated) ->
|
||||
available False, chat_json None, no crash at import.
|
||||
8. Routing: no llm block -> both consumers get the SAME DeepSeek singleton
|
||||
(shared cap); routed config -> news gets DeepSeekClient while
|
||||
agent_panel gets AnthropicClient; same-provider consumers share one
|
||||
instance.
|
||||
9. Global lc._client override shadows BOTH consumers (locks the 41 legacy
|
||||
stub sites' semantics).
|
||||
10. reset_llm_client() clears both the override and the per-provider cache.
|
||||
11. End-to-end: routing agent_panel->anthropic, attach_panel reaches the
|
||||
anthropic fake (5 calls) while a news call reaches the deepseek stub.
|
||||
12. telegram _news_health_lines renders per-provider lines from usage()
|
||||
without touching private fields.
|
||||
|
||||
GATE: Modules 19 and 23 must pass UNMODIFIED - the no-regression proof for
|
||||
the refactor.
|
||||
|
||||
================================================================================
|
||||
5. DOCS
|
||||
================================================================================
|
||||
- CLAUDE.md: llm_client bullet -> provider-abstracted (BaseLLMClient;
|
||||
deepseek + anthropic registered); config.json "llm" block routes consumers;
|
||||
absent = all-DeepSeek. Add ANTHROPIC_API_KEY to env-var list. Test counts.
|
||||
- config.json.example: both new blocks.
|
||||
|
||||
================================================================================
|
||||
COMMIT SEQUENCE (each commit green)
|
||||
================================================================================
|
||||
1. llm_client: extract BaseLLMClient; DeepSeekClient unchanged behavior
|
||||
(+ base-contract tests) - pure refactor, Modules 19/23 green untouched.
|
||||
2. config: anthropic block + llm consumer-routing block (+ config tests).
|
||||
3. llm_client: AnthropicClient (official SDK, lazy import) + requirements.txt
|
||||
(+ parse/shape tests).
|
||||
4. routing: per-consumer get_llm_client + health lines - consumer call-site
|
||||
edits, telegram usage() lines, routing/e2e tests.
|
||||
5. docs: CLAUDE.md + example config.
|
||||
|
||||
================================================================================
|
||||
VERIFICATION
|
||||
================================================================================
|
||||
- python test_all.py green at every commit (~374 -> ~386).
|
||||
- Live smoke A (no behavior change): default config, main.py --panel ->
|
||||
still DeepSeek, /status unchanged.
|
||||
- Live smoke B (the goal): llm.agent_panel="anthropic" + real
|
||||
ANTHROPIC_API_KEY, run --panel -> one real Claude panel (5 requests,
|
||||
validates JSON contract + latency + refusal-free path) while a news
|
||||
refresh still hits DeepSeek; /status shows two provider lines.
|
||||
- Backtest untouched by construction (no LLM in replay path); one 30-day
|
||||
run as a formality.
|
||||
|
||||
================================================================================
|
||||
EFFORT ESTIMATE
|
||||
================================================================================
|
||||
0 new files, 8 modified, ~520 lines, est. 3-5 hours.
|
||||
|
||||
llm_client.py base class + AnthropicClient + routing +235
|
||||
test_all.py Module 24 +200
|
||||
app_config.py anthropic + llm blocks, resolver +55
|
||||
telegram_bot.py per-provider health lines +15
|
||||
news_engine.py consumer tags on call sites +4
|
||||
agent_panel.py consumer tags on call sites +4
|
||||
config.json.example blocks +14
|
||||
requirements.txt anthropic +1
|
||||
CLAUDE.md docs +10
|
||||
|
||||
================================================================================
|
||||
KEY DECISIONS EMBEDDED IN THIS PLAN
|
||||
================================================================================
|
||||
- Official Anthropic SDK, not raw requests (current API guidance for Python;
|
||||
lazy import keeps it optional for DeepSeek-only installs).
|
||||
- Default Claude model: claude-opus-5 ($5/$25 per MTok; a full 5-call panel
|
||||
~ 8k in / 500 out ~ $0.05). Config-overridable to claude-haiku-4-5.
|
||||
- Two Claude gotchas handled: temperature is REMOVED on Opus 5 (sending it
|
||||
400s - must not be copied from the DeepSeek request body), and refusals
|
||||
arrive as HTTP 200 with stop_reason "refusal" (checked before parsing,
|
||||
mapped to the normal None/fallback path).
|
||||
- Caps are per-provider, not per-consumer: two consumers on one provider
|
||||
share its cap (today's semantics); splitting across two providers doubles
|
||||
total headroom naturally.
|
||||
- The panel's own max_panels_per_day budget is untouched - it sits above
|
||||
whichever provider serves it.
|
||||
- Not in scope: static/news_summary.php (own PHP client/key), streaming,
|
||||
tool use, per-consumer model overrides (add a "model" key per consumer
|
||||
later if ever needed).
|
||||
================================================================================
|
||||
Reference in New Issue
Block a user