Skip to content

feat(llm): adopt Anthropic prompt caching + per-purpose spend tagging across CMOAgent

isidro requested to merge pmc/auto/202605201100-4400 into main

Summary

Vendor-radar adoption of two Anthropic SDK features already shipped (since Aug 2024) that CMOAgent wasn't using:

  1. Prompt caching via cache_control={"type":"ephemeral"} on the 4 static + large system prompts (digest, weekly_reset, operator-plan, experiment-proposal). After the first call within the 5-min TTL, input tokens on that block are billed at offline% of normal cost.
  2. Per-purpose spend tagging via metadata=cmo_metadata("<purpose>") on every LLM call site (previously missing on weekly_reset, learning/propose, pitch_analyzer, watcher). Lets LiteLLM spend_per_tag and PMC's /api/mc/llm-spend dashboard attribute cost per code path.

Also corrects a misleading comment in llm_client.py that claimed LiteLLM auto-injected cache_control at the proxy layer — it does not; the client must add it.

Changes

  • New helper cached_system(prompt: str) in src/agents/gaming_cmo/llm_client.py that wraps a system string in the Anthropic prompt-caching block form.
  • Caching applied in: digest.py, weekly_reset.py, agent.py (operator-plan + experiment-proposal paths).
  • Tagging added in: weekly_reset.py, agent.py (both paths), learning/propose.py, pitch_analyzer.py, watcher.py.
  • Floor bump anthropic>=0.40.0 → anthropic>=0.50.0 (lockfile already 0.95.0).
  • 7 new unit tests in tests/test_llm_client.py covering helper shape, validation, metadata invariants, and an integration assertion on digest.generate_daily_digest.

Test plan

  • uv run pytest -q → 398 passed in 6.32s (baseline 391, +7 new)
  • uv sync --extra dev resolves anthropic 0.95.0 cleanly
  • Post-merge: verify on a real digest run that response.usage.cache_creation_input_tokens / cache_read_input_tokens are non-zero (confirms LiteLLM forwards cache_control and Anthropic bills the discount).

Follow-ups (out of scope, noted in commit bodies)

  • analyst.py:331 still constructs anthropic.Anthropic inline (already tagged); migrating to get_anthropic_client() requires relaxing the module-singleton to accept per-call overrides.
  • Bigger radar items (Citations API for analyst, native web_search for watcher, memory tool for learning loop) deferred — require a product decision, not autopilot work.

Commits

  • 71f0eec2 feat(llm): adopt Anthropic prompt caching for digest + weekly_reset
  • 57c226e3 feat(llm): extend prompt caching + spend tagging to remaining call sites

Merge request reports