Nine efficiency improvements across the data pipeline:
1. NewsAPI OR batching (news_service.py + news_fetcher.py)
- Combine up to 4 bills per NewsAPI call using OR query syntax
- NEWSAPI_BATCH_SIZE=4 means ~4× effective daily quota (100→400 bill-fetches)
- fetch_news_for_bill_batch task; fetch_news_for_active_bills queues batches
2. Google News RSS cache (news_service.py)
- 2-hour Redis cache shared between news_fetcher and trend_scorer
- Eliminates duplicate RSS hits when both workers run against same bill
- clear_gnews_cache() admin helper + admin endpoint
3. pytrends keyword batching (trends_service.py + trend_scorer.py)
- Compare up to 5 bills per pytrends call instead of 1
- get_trends_scores_batch() returns scores in original order
- Reduces pytrends calls by ~5× and associated rate-limit risk
4. GovInfo ETags (govinfo_api.py + document_fetcher.py)
- If-None-Match conditional GET; DocumentUnchangedError on HTTP 304
- ETags stored in Redis (30-day TTL) keyed by MD5(url)
- document_fetcher catches DocumentUnchangedError → {"status": "unchanged"}
5. Anthropic prompt caching (llm_service.py)
- cache_control: {type: ephemeral} on system messages in AnthropicProvider
- Caches the ~700-token system prompt server-side; ~50% cost reduction on
repeated calls within the 5-minute cache window
6. Async sponsor fetch (congress_poller.py)
- New fetch_sponsor_for_bill Celery task replaces blocking get_bill_detail()
inline in poll loop
- Bills saved immediately with sponsor_id=None; sponsor linked async
- Removes 0.25s sleep per new bill from poll hot path
7. Skip doc fetch for procedural actions (congress_poller.py)
- _DOC_PRODUCING_CATEGORIES = {vote, committee_report, presidential, ...}
- fetch_bill_documents only enqueued when action is likely to produce
new GovInfo text (saves ~60–70% of unnecessary document fetch attempts)
8. Adaptive poll frequency (congress_poller.py)
- _is_congress_off_hours(): weekends + before 9AM / after 9PM EST
- Skips poll if off-hours AND last poll < 1 hour ago
- Prevents wasteful polling when Congress is not in session
9. Admin panel additions (admin.py + settings/page.tsx + api.ts)
- GET /api/admin/newsapi-quota → remaining calls today
- POST /api/admin/clear-gnews-cache → flush RSS cache
- Settings page shows NewsAPI quota remaining (amber if < 10)
- "Clear Google News Cache" button in Manual Controls
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ZIP lookup (GET /api/members/by-zip/{zip}):
- Two-step geocoding: Nominatim (ZIP → lat/lng) then Census TIGERweb
Legislative identify (lat/lng → congressional district via GEOID)
- Handles at-large states (AK, DE, MT, ND, SD, VT, WY)
- Added rep_lookup health check to admin External API Health panel
congress_api.py fixes:
- parse_member_from_api: normalize state full name → 2-letter code
(Congress.gov returns "Florida", DB expects "FL")
- parse_member_from_api: read district from top-level data field,
not current_term (district is not inside the term object)
Celery beat: schedule sync_members daily at 1 AM UTC so chamber,
district, and contact info stay current without manual triggering
Members page redesign: photo avatars, party/state/chamber chips,
phone + website links, ZIP lookup form to find your reps
Draft letter improvements: pass rep_name from ZIP lookup so letter
opens with "Dear Representative Franklin," instead of generic salutation;
add has_document filter to bills list endpoint
UX additions: HelpTip component, How It Works page, "How it works"
sidebar nav link, collections page description copy
Authored-By: Jack Levy
- Add DraftLetterPanel: collapsible UI below BriefPanel for bills with a
brief; lets users select up to 3 cited points, pick stance/tone, and
generate a plain-text letter via the configured LLM provider
- Stance pre-fills from follow mode (pocket_boost → YES, pocket_veto → NO)
and clears when the user unfollows; recipient derived from bill chamber
- Add POST /api/bills/{bill_id}/draft-letter endpoint with proper LLM
provider/model resolution from AppSetting (respects Settings page choice)
- Add generate_text() to LLMProvider ABC and all four providers
- Expose has_document on BillSchema (list endpoint) via a single batch
query; BillCard shows Brief / Pending / No text indicator per bill
Authored-By: Jack Levy
- get_llm_provider() now accepts provider + model args so DB overrides
propagate through to all provider constructors (was always reading
env vars, ignoring the admin UI selection)
- /test-llm replaced with lightweight ping (max_tokens=20) instead of
running a full bill analysis; shows model name + reply, no truncation
- /api/settings/llm-models endpoint fetches available models live from
each provider's API (OpenAI, Anthropic REST, Gemini, Ollama)
- Admin UI model picker dynamically populated from provider API;
falls back to manual text input on error; Custom model name option kept
- Default Gemini model updated: gemini-1.5-pro → gemini-2.0-flash
Co-Authored-By: Jack Levy
LLM prompts updated to output {text, citation, quote} objects for every
key_point and risk. govinfo_url stored on BillBrief (migration 0006) so
the frontend can link directly to the source document without an extra
query. AIBriefCard renders § citation chips that expand inline to show
the verbatim quote and a View source → GovInfo link. Old plain-string
briefs continue to render unchanged.
Authored-By: Jack Levy
