Same bug as members.py: health check used old CD\d+FP$ regex (no match
for CD119) and skipped GEOID. Now mirrors members.py logic: GEOID primary,
STATE+CD\d+ fallback, Congressional layer filter.
Authored by: Jack Levy
Member Effectiveness Score
- New BillCosponsor table (migration 0018) with per-bill co-sponsor
party data required for the bipartisan multiplier
- bill_category column on Bill (substantive | commemorative | administrative)
set by a cheap one-shot LLM call after each brief is generated
- effectiveness_score / percentile / tier columns on Member
- New bill_classifier.py worker with 5 tasks:
classify_bill_category — triggered from llm_processor after brief
fetch_bill_cosponsors — triggered from congress_poller on new bill
calculate_effectiveness_scores — nightly at 5 AM UTC
backfill_bill_categories / backfill_all_bill_cosponsors — one-time
- Scoring: distance-traveled pts × bipartisan (1.5×) × substance (0.1×
for commemorative) × leadership (1.2× for committee chairs)
- Percentile normalised within (seniority tier × party) buckets
- Effectiveness card on member detail page with colour-coded bar
- Admin panel: 3 new backfill/calculate controls in Maintenance section
Representation Alignment View
- New GET /api/alignment endpoint: cross-references user's stanced bill
follows (pocket_veto/pocket_boost) with followed members' vote positions
- Efficient bulk queries — no N+1 loops
- New /alignment page with ranked member list and alignment bars
- Alignment added to sidebar nav (auth-required)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Submit up to 1000 unbriefed documents to the provider Batch API in one
shot instead of individual synchronous LLM calls. Results are polled
every 30 minutes via a new Celery beat task and imported automatically.
- New worker: llm_batch_processor.py
- submit_llm_batch: guards against duplicate batches, builds JSONL
(OpenAI) or request list (Anthropic), stores state in AppSetting
- poll_llm_batch_results: checks batch status, imports completed
results with idempotency, emits notifications + triggers news fetch
- celery_app: register worker, route to llm queue, beat every 30 min
- admin API: POST /submit-llm-batch + GET /llm-batch-status endpoints
- Frontend: submitLlmBatch + getLlmBatchStatus in adminAPI; settings
page shows batch control row (openai/anthropic only) with live
progress line while batch is processing
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Nine efficiency improvements across the data pipeline:
1. NewsAPI OR batching (news_service.py + news_fetcher.py)
- Combine up to 4 bills per NewsAPI call using OR query syntax
- NEWSAPI_BATCH_SIZE=4 means ~4× effective daily quota (100→400 bill-fetches)
- fetch_news_for_bill_batch task; fetch_news_for_active_bills queues batches
2. Google News RSS cache (news_service.py)
- 2-hour Redis cache shared between news_fetcher and trend_scorer
- Eliminates duplicate RSS hits when both workers run against same bill
- clear_gnews_cache() admin helper + admin endpoint
3. pytrends keyword batching (trends_service.py + trend_scorer.py)
- Compare up to 5 bills per pytrends call instead of 1
- get_trends_scores_batch() returns scores in original order
- Reduces pytrends calls by ~5× and associated rate-limit risk
4. GovInfo ETags (govinfo_api.py + document_fetcher.py)
- If-None-Match conditional GET; DocumentUnchangedError on HTTP 304
- ETags stored in Redis (30-day TTL) keyed by MD5(url)
- document_fetcher catches DocumentUnchangedError → {"status": "unchanged"}
5. Anthropic prompt caching (llm_service.py)
- cache_control: {type: ephemeral} on system messages in AnthropicProvider
- Caches the ~700-token system prompt server-side; ~50% cost reduction on
repeated calls within the 5-minute cache window
6. Async sponsor fetch (congress_poller.py)
- New fetch_sponsor_for_bill Celery task replaces blocking get_bill_detail()
inline in poll loop
- Bills saved immediately with sponsor_id=None; sponsor linked async
- Removes 0.25s sleep per new bill from poll hot path
7. Skip doc fetch for procedural actions (congress_poller.py)
- _DOC_PRODUCING_CATEGORIES = {vote, committee_report, presidential, ...}
- fetch_bill_documents only enqueued when action is likely to produce
new GovInfo text (saves ~60–70% of unnecessary document fetch attempts)
8. Adaptive poll frequency (congress_poller.py)
- _is_congress_off_hours(): weekends + before 9AM / after 9PM EST
- Skips poll if off-hours AND last poll < 1 hour ago
- Prevents wasteful polling when Congress is not in session
9. Admin panel additions (admin.py + settings/page.tsx + api.ts)
- GET /api/admin/newsapi-quota → remaining calls today
- POST /api/admin/clear-gnews-cache → flush RSS cache
- Settings page shows NewsAPI quota remaining (amber if < 10)
- "Clear Google News Cache" button in Manual Controls
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ZIP lookup (GET /api/members/by-zip/{zip}):
- Two-step geocoding: Nominatim (ZIP → lat/lng) then Census TIGERweb
Legislative identify (lat/lng → congressional district via GEOID)
- Handles at-large states (AK, DE, MT, ND, SD, VT, WY)
- Added rep_lookup health check to admin External API Health panel
congress_api.py fixes:
- parse_member_from_api: normalize state full name → 2-letter code
(Congress.gov returns "Florida", DB expects "FL")
- parse_member_from_api: read district from top-level data field,
not current_term (district is not inside the term object)
Celery beat: schedule sync_members daily at 1 AM UTC so chamber,
district, and contact info stay current without manual triggering
Members page redesign: photo avatars, party/state/chamber chips,
phone + website links, ZIP lookup form to find your reps
Draft letter improvements: pass rep_name from ZIP lookup so letter
opens with "Dear Representative Franklin," instead of generic salutation;
add has_document filter to bills list endpoint
UX additions: HelpTip component, How It Works page, "How it works"
sidebar nav link, collections page description copy
Authored-By: Jack Levy
Weekly Digest (send_weekly_digest Celery task):
- Runs every Monday 8:30 AM UTC via beat schedule
- Queries all followed bills updated in the past 7 days per user
- Sends low-priority ntfy push (Priority: low, Tags: newspaper,calendar)
- Creates a NotificationEvent (weekly_digest type) for RSS feed visibility
- Admin can trigger immediately via POST /api/admin/trigger-weekly-digest
- Manual Controls panel now includes "Send Weekly Digest" button
Local-time quiet hours:
- Browser auto-detects IANA timezone via Intl.DateTimeFormat().resolvedOptions().timeZone
- Timezone saved to notification_prefs alongside quiet_hours_start/end on Save
- Dispatcher converts UTC → user's local time (zoneinfo stdlib) before hour comparison
- Falls back to UTC if timezone absent or unrecognised
- Quiet hours UI: 12-hour AM/PM selectors, shows detected timezone as hint
- Clearing quiet hours also clears stored timezone
Co-Authored-By: Jack Levy
Notifications:
- New /notifications page accessible to all users (ntfy + RSS config)
- ntfy now supports no-auth, Bearer token, and HTTP Basic auth (for ACL-protected self-hosted servers)
- RSS enabled/disabled independently of ntfy; token auto-generated on first GET
- Notification settings removed from admin-only Settings page; replaced with link card
- Sidebar adds Notifications nav link for all users
- notification_dispatcher.py: fan-out now marks RSS events dispatched independently
Action history:
- Migration 0012: deduplicates existing bill_actions rows and adds UNIQUE(bill_id, action_date, action_text)
- congress_poller.py: replaces existence-check inserts with ON CONFLICT DO NOTHING (race-condition safe)
- Added backfill_all_bill_actions task (no date filter) + admin endpoint POST /backfill-all-actions
Authored-By: Jack Levy
- Fetch bill actions from Congress.gov and populate the action timeline
- Add nightly batch task and beat schedule for active bill actions
- Add admin reprocess endpoint for per-bill debugging
- Add BriefPanel with "What Changed" view and version history
- Add External API Health section with per-source latency testing
- Redesign Manual Controls as health panel with status dots and descriptions
- Add Resume Analysis task for stalled LLM jobs
- Add Backfill Dates & Links task for bills with null metadata
- Fix LLM provider/model DB overrides being ignored (env vars used instead)
- Fix Gemini 404: gemini-1.5-pro deprecated → gemini-2.0-flash
- Fix Anthropic models list: use REST API directly (SDK too old for .models)
- Replace test-LLM full analysis with lightweight ping (max_tokens=20)
- Add has_document field to BillDetail; show "No bill text published" state
- Fix "Introduced: —" showing for bills with null introduced_date
- Add bills_missing_sponsor and bills_missing_metadata to admin stats
- Add GovInfo health check using /collections endpoint (fixes 500 from /packages)
Authored-By: Jack Levy
Backend:
- Add fetch_bill_actions task with pagination and idempotent upsert
- Add fetch_actions_for_active_bills nightly batch (4 AM UTC beat schedule)
- Wire fetch_bill_actions into new-bill creation and _update_bill_if_changed
- Add backfill_brief_citations task: detects pre-citation briefs by JSONB
type check, deletes them, re-queues LLM processing against stored text
(LLM calls only — zero Congress.gov or GovInfo calls)
- Add admin endpoints: POST /bills/{id}/reprocess, /backfill-citations,
/trigger-fetch-actions; add uncited_briefs count to /stats
Frontend:
- New BriefPanel component: wraps AIBriefCard, adds amber "What Changed"
badge for amendment briefs and collapsible version history with
inline brief expansion
- Swap AIBriefCard for BriefPanel on bill detail page
- Admin panel: Backfill Citations + Fetch Bill Actions buttons; amber
warning in stats when uncited briefs remain
- Add feature roadmap document with phased plan through Phase 5
Co-Authored-By: Jack Levy
- Poller now fetches bill detail on insert to get sponsor (list endpoint
has no sponsor data)
- Add backfill_sponsor_ids task + admin endpoint + UI button to fix the
1,616 existing bills with NULL sponsor_id
- Member name search now matches both "Last, First" and "First Last"
using split_part() on the stored name column; same fix applied to
global search
- Load Bill.sponsor relationship eagerly in get_member_bills to prevent
MissingGreenlet error during Pydantic serialization
- Remove .trim() on search onChange so spaces can be typed
Authored-By: Jack Levy
- GET /api/admin/stats returns total bills, docs fetched, briefs generated
(full + amendment), and remaining count
- Admin page shows stat cards + progress bar, auto-refreshes every 30s
Authored-By: Jack Levy
- User model with email/hashed_password/is_admin/notification_prefs
- JWT auth: POST /api/auth/register, /login, /me
- First registered user auto-promoted to admin
- Migration 0005: users table + user_id FK on follows (clears global follows)
- Follows, dashboard, settings, admin endpoints all require authentication
- Admin endpoints (settings writes, celery triggers) require is_admin
- Frontend: login/register pages, Zustand auth store (localStorage persist)
- AuthGuard component gates all app routes, shows app shell only when authed
- Sidebar shows user email + logout; Admin nav link visible to admins only
- Admin panel (/settings): user list with delete + promote/demote, LLM config,
data source settings, and manual celery controls
Authored-By: Jack Levy
