7e5c5b473e
Nine efficiency improvements across the data pipeline:
1. NewsAPI OR batching (news_service.py + news_fetcher.py)
- Combine up to 4 bills per NewsAPI call using OR query syntax
- NEWSAPI_BATCH_SIZE=4 means ~4× effective daily quota (100→400 bill-fetches)
- fetch_news_for_bill_batch task; fetch_news_for_active_bills queues batches
2. Google News RSS cache (news_service.py)
- 2-hour Redis cache shared between news_fetcher and trend_scorer
- Eliminates duplicate RSS hits when both workers run against same bill
- clear_gnews_cache() admin helper + admin endpoint
3. pytrends keyword batching (trends_service.py + trend_scorer.py)
- Compare up to 5 bills per pytrends call instead of 1
- get_trends_scores_batch() returns scores in original order
- Reduces pytrends calls by ~5× and associated rate-limit risk
4. GovInfo ETags (govinfo_api.py + document_fetcher.py)
- If-None-Match conditional GET; DocumentUnchangedError on HTTP 304
- ETags stored in Redis (30-day TTL) keyed by MD5(url)
- document_fetcher catches DocumentUnchangedError → {"status": "unchanged"}
5. Anthropic prompt caching (llm_service.py)
- cache_control: {type: ephemeral} on system messages in AnthropicProvider
- Caches the ~700-token system prompt server-side; ~50% cost reduction on
repeated calls within the 5-minute cache window
6. Async sponsor fetch (congress_poller.py)
- New fetch_sponsor_for_bill Celery task replaces blocking get_bill_detail()
inline in poll loop
- Bills saved immediately with sponsor_id=None; sponsor linked async
- Removes 0.25s sleep per new bill from poll hot path
7. Skip doc fetch for procedural actions (congress_poller.py)
- _DOC_PRODUCING_CATEGORIES = {vote, committee_report, presidential, ...}
- fetch_bill_documents only enqueued when action is likely to produce
new GovInfo text (saves ~60–70% of unnecessary document fetch attempts)
8. Adaptive poll frequency (congress_poller.py)
- _is_congress_off_hours(): weekends + before 9AM / after 9PM EST
- Skips poll if off-hours AND last poll < 1 hour ago
- Prevents wasteful polling when Congress is not in session
9. Admin panel additions (admin.py + settings/page.tsx + api.ts)
- GET /api/admin/newsapi-quota → remaining calls today
- POST /api/admin/clear-gnews-cache → flush RSS cache
- Settings page shows NewsAPI quota remaining (amber if < 10)
- "Clear Google News Cache" button in Manual Controls
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
64 lines
1.9 KiB
Python
64 lines
1.9 KiB
Python
from functools import lru_cache
|
|
from pydantic_settings import BaseSettings, SettingsConfigDict
|
|
|
|
|
|
class Settings(BaseSettings):
|
|
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
|
|
|
|
# URLs
|
|
LOCAL_URL: str = "http://localhost"
|
|
PUBLIC_URL: str = ""
|
|
|
|
# Auth / JWT
|
|
JWT_SECRET_KEY: str = "change-me-in-production"
|
|
JWT_EXPIRE_MINUTES: int = 60 * 24 * 7 # 7 days
|
|
|
|
# Database
|
|
DATABASE_URL: str = "postgresql+asyncpg://congress:congress@postgres:5432/pocketveto"
|
|
SYNC_DATABASE_URL: str = "postgresql://congress:congress@postgres:5432/pocketveto"
|
|
|
|
# Redis
|
|
REDIS_URL: str = "redis://redis:6379/0"
|
|
|
|
# api.data.gov (shared key for Congress.gov and GovInfo)
|
|
DATA_GOV_API_KEY: str = ""
|
|
CONGRESS_POLL_INTERVAL_MINUTES: int = 30
|
|
|
|
# LLM
|
|
LLM_PROVIDER: str = "openai" # openai | anthropic | gemini | ollama
|
|
|
|
OPENAI_API_KEY: str = ""
|
|
OPENAI_MODEL: str = "gpt-4o-mini" # gpt-4o-mini: excellent JSON quality at ~10x lower cost than gpt-4o
|
|
|
|
ANTHROPIC_API_KEY: str = ""
|
|
ANTHROPIC_MODEL: str = "claude-sonnet-4-6" # Sonnet matches Opus for structured tasks at ~5x lower cost
|
|
|
|
GEMINI_API_KEY: str = ""
|
|
GEMINI_MODEL: str = "gemini-2.0-flash"
|
|
|
|
OLLAMA_BASE_URL: str = "http://host.docker.internal:11434"
|
|
OLLAMA_MODEL: str = "llama3.1"
|
|
|
|
# Max LLM requests per minute — Celery enforces this globally across all workers.
|
|
# Safe defaults: free Gemini=15 RPM, Anthropic paid=50 RPM, OpenAI paid=500 RPM.
|
|
# Raise this in .env once you confirm your API tier.
|
|
LLM_RATE_LIMIT_RPM: int = 10
|
|
|
|
# Google Civic Information API (zip → representative lookup)
|
|
# Free key: https://console.cloud.google.com/apis/library/civicinfo.googleapis.com
|
|
CIVIC_API_KEY: str = ""
|
|
|
|
# News
|
|
NEWSAPI_KEY: str = ""
|
|
|
|
# pytrends
|
|
PYTRENDS_ENABLED: bool = True
|
|
|
|
|
|
@lru_cache
|
|
def get_settings() -> Settings:
|
|
return Settings()
|
|
|
|
|
|
settings = get_settings()
|