Nine efficiency improvements across the data pipeline:
1. NewsAPI OR batching (news_service.py + news_fetcher.py)
- Combine up to 4 bills per NewsAPI call using OR query syntax
- NEWSAPI_BATCH_SIZE=4 means ~4× effective daily quota (100→400 bill-fetches)
- fetch_news_for_bill_batch task; fetch_news_for_active_bills queues batches
2. Google News RSS cache (news_service.py)
- 2-hour Redis cache shared between news_fetcher and trend_scorer
- Eliminates duplicate RSS hits when both workers run against same bill
- clear_gnews_cache() admin helper + admin endpoint
3. pytrends keyword batching (trends_service.py + trend_scorer.py)
- Compare up to 5 bills per pytrends call instead of 1
- get_trends_scores_batch() returns scores in original order
- Reduces pytrends calls by ~5× and associated rate-limit risk
4. GovInfo ETags (govinfo_api.py + document_fetcher.py)
- If-None-Match conditional GET; DocumentUnchangedError on HTTP 304
- ETags stored in Redis (30-day TTL) keyed by MD5(url)
- document_fetcher catches DocumentUnchangedError → {"status": "unchanged"}
5. Anthropic prompt caching (llm_service.py)
- cache_control: {type: ephemeral} on system messages in AnthropicProvider
- Caches the ~700-token system prompt server-side; ~50% cost reduction on
repeated calls within the 5-minute cache window
6. Async sponsor fetch (congress_poller.py)
- New fetch_sponsor_for_bill Celery task replaces blocking get_bill_detail()
inline in poll loop
- Bills saved immediately with sponsor_id=None; sponsor linked async
- Removes 0.25s sleep per new bill from poll hot path
7. Skip doc fetch for procedural actions (congress_poller.py)
- _DOC_PRODUCING_CATEGORIES = {vote, committee_report, presidential, ...}
- fetch_bill_documents only enqueued when action is likely to produce
new GovInfo text (saves ~60–70% of unnecessary document fetch attempts)
8. Adaptive poll frequency (congress_poller.py)
- _is_congress_off_hours(): weekends + before 9AM / after 9PM EST
- Skips poll if off-hours AND last poll < 1 hour ago
- Prevents wasteful polling when Congress is not in session
9. Admin panel additions (admin.py + settings/page.tsx + api.ts)
- GET /api/admin/newsapi-quota → remaining calls today
- POST /api/admin/clear-gnews-cache → flush RSS cache
- Settings page shows NewsAPI quota remaining (amber if < 10)
- "Clear Google News Cache" button in Manual Controls
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Roll-call votes:
- Migration 0017: bill_votes + member_vote_positions tables
- Fetch vote XML directly from House Clerk / Senate LIS URLs
embedded in bill actions recordedVotes objects
- GET /api/bills/{id}/votes triggers background fetch on first view
- VotePanel on bill detail: yea/nay bar, result badge, followed
member positions with Sen./Rep. title, party badge, and state
Alert filter fix:
- _should_dispatch returns True when alert_filters is None so users
who haven't saved filters still receive all notifications
Authored-By: Jack Levy
Phase 3 completion — Personal Workflow feature set is now complete.
Collections / Watchlists:
- New tables: collections (UUID share_token, slug, public/private) and
collection_bills (unique bill-per-collection constraint)
- Full CRUD API at /api/collections with bill add/remove endpoints
- Public share endpoint /api/collections/share/{token} (no auth)
- /collections list page with inline create form and delete
- /collections/[id] detail page: inline rename, public toggle,
copy-share-link, bill search/add/remove
- CollectionPicker bookmark-icon popover on bill detail pages
- Collections nav link in sidebar (auth-required)
Shareable Brief Links:
- share_token UUID column on bill_briefs (backfilled on migration)
- Unified public share router at /api/share (brief + collection)
- /share/brief/[token] — minimal layout, full AIBriefCard, CTAs
- /share/collection/[token] — minimal layout, bill list, CTA
- Share2 button in BriefPanel header row, "Link copied!" flash
AuthGuard: /collections → AUTH_REQUIRED; /share prefix → NO_SHELL_PATHS
Authored-By: Jack Levy
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- bill_notes table (migration 0014): user_id, bill_id, content, pinned,
created_at, updated_at; unique constraint (user_id, bill_id)
- BillNote SQLAlchemy model with back-refs on User and Bill
- GET/PUT/DELETE /api/notes/{bill_id} — auth-required, one note per (user, bill)
- NotesPanel component: collapsible, auto-resize textarea, pin toggle,
save + delete; shows last-saved date and pin indicator in collapsed header
- Pinned notes render above BriefPanel; unpinned render below DraftLetterPanel
- Guests see nothing (token guard in component + query disabled)
Co-Authored-By: Jack Levy
Weekly Digest (send_weekly_digest Celery task):
- Runs every Monday 8:30 AM UTC via beat schedule
- Queries all followed bills updated in the past 7 days per user
- Sends low-priority ntfy push (Priority: low, Tags: newspaper,calendar)
- Creates a NotificationEvent (weekly_digest type) for RSS feed visibility
- Admin can trigger immediately via POST /api/admin/trigger-weekly-digest
- Manual Controls panel now includes "Send Weekly Digest" button
Local-time quiet hours:
- Browser auto-detects IANA timezone via Intl.DateTimeFormat().resolvedOptions().timeZone
- Timezone saved to notification_prefs alongside quiet_hours_start/end on Save
- Dispatcher converts UTC → user's local time (zoneinfo stdlib) before hour comparison
- Falls back to UTC if timezone absent or unrecognised
- Quiet hours UI: 12-hour AM/PM selectors, shows detected timezone as hint
- Clearing quiet hours also clears stored timezone
Co-Authored-By: Jack Levy
- Add DraftLetterPanel: collapsible UI below BriefPanel for bills with a
brief; lets users select up to 3 cited points, pick stance/tone, and
generate a plain-text letter via the configured LLM provider
- Stance pre-fills from follow mode (pocket_boost → YES, pocket_veto → NO)
and clears when the user unfollows; recipient derived from bill chamber
- Add POST /api/bills/{bill_id}/draft-letter endpoint with proper LLM
provider/model resolution from AppSetting (respects Settings page choice)
- Add generate_text() to LLMProvider ABC and all four providers
- Expose has_document on BillSchema (list endpoint) via a single batch
query; BillCard shows Brief / Pending / No text indicator per bill
Authored-By: Jack Levy
- POST /api/notifications/test/ntfy — sends a real push using current form
values (not saved settings) so auth can be verified before saving; returns
status + HTTP detail on success or error message on failure
- POST /api/notifications/test/rss — confirms the feed token exists and
returns event count; no bill FK required
- NtfyTestRequest + NotificationTestResult schemas added
- Frontend: Test button next to Save on both ntfy and RSS sections; result
shown inline as a green/red pill; uses current form state for ntfy so
the user can test before committing
All future notification types should follow the same test-before-save pattern.
Authored-By: Jack Levy
Notifications:
- New /notifications page accessible to all users (ntfy + RSS config)
- ntfy now supports no-auth, Bearer token, and HTTP Basic auth (for ACL-protected self-hosted servers)
- RSS enabled/disabled independently of ntfy; token auto-generated on first GET
- Notification settings removed from admin-only Settings page; replaced with link card
- Sidebar adds Notifications nav link for all users
- notification_dispatcher.py: fan-out now marks RSS events dispatched independently
Action history:
- Migration 0012: deduplicates existing bill_actions rows and adds UNIQUE(bill_id, action_date, action_text)
- congress_poller.py: replaces existence-check inserts with ON CONFLICT DO NOTHING (race-condition safe)
- Added backfill_all_bill_actions task (no date filter) + admin endpoint POST /backfill-all-actions
Authored-By: Jack Levy
- Fetch bill actions from Congress.gov and populate the action timeline
- Add nightly batch task and beat schedule for active bill actions
- Add admin reprocess endpoint for per-bill debugging
- Add BriefPanel with "What Changed" view and version history
- Add External API Health section with per-source latency testing
- Redesign Manual Controls as health panel with status dots and descriptions
- Add Resume Analysis task for stalled LLM jobs
- Add Backfill Dates & Links task for bills with null metadata
- Fix LLM provider/model DB overrides being ignored (env vars used instead)
- Fix Gemini 404: gemini-1.5-pro deprecated → gemini-2.0-flash
- Fix Anthropic models list: use REST API directly (SDK too old for .models)
- Replace test-LLM full analysis with lightweight ping (max_tokens=20)
- Add has_document field to BillDetail; show "No bill text published" state
- Fix "Introduced: —" showing for bills with null introduced_date
- Add bills_missing_sponsor and bills_missing_metadata to admin stats
- Add GovInfo health check using /collections endpoint (fixes 500 from /packages)
Authored-By: Jack Levy
Adds Google Trends, NewsAPI, and Google News RSS scoring for members,
mirroring the existing bill interest pipeline. Member profiles now show
a Public Interest chart (with signal breakdown) and a Related News panel.
Key changes:
- New member_trend_scores + member_news_articles tables (migration 0008)
- fetch_gnews_articles() added to news_service for unlimited RSS article storage
- Bill news fetcher now combines NewsAPI + Google News RSS (more coverage)
- New member_interest Celery worker with scheduled news + trend tasks
- GET /members/{id}/trend and /news API endpoints
- TrendChart redesigned with signal breakdown badges and bar+line combo chart
- NewsPanel accepts generic article shape (bills and members)
Co-Authored-By: Jack Levy
Lazy-enriches member profiles on first view via Congress.gov detail API.
Adds office address, phone, official website, congress.gov link, birth
year, terms history, leadership roles, and sponsored/cosponsored counts.
Includes DB migration 0007 for new member columns.
Co-Authored-By: Jack Levy
LLM prompts updated to output {text, citation, quote} objects for every
key_point and risk. govinfo_url stored on BillBrief (migration 0006) so
the frontend can link directly to the source document without an extra
query. AIBriefCard renders § citation chips that expand inline to show
the verbatim quote and a View source → GovInfo link. Old plain-string
briefs continue to render unchanged.
Authored-By: Jack Levy
- User model with email/hashed_password/is_admin/notification_prefs
- JWT auth: POST /api/auth/register, /login, /me
- First registered user auto-promoted to admin
- Migration 0005: users table + user_id FK on follows (clears global follows)
- Follows, dashboard, settings, admin endpoints all require authentication
- Admin endpoints (settings writes, celery triggers) require is_admin
- Frontend: login/register pages, Zustand auth store (localStorage persist)
- AuthGuard component gates all app routes, shows app shell only when authed
- Sidebar shows user email + logout; Admin nav link visible to admins only
- Admin panel (/settings): user list with delete + promote/demote, LLM config,
data source settings, and manual celery controls
Authored-By: Jack Levy
