From a96bd024e9d43455e38817abd9fdff0214f726a3 Mon Sep 17 00:00:00 2001 From: Jack Levy Date: Sun, 15 Mar 2026 01:10:52 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20v1.0.0=20=E2=80=94=20full=20documentati?= =?UTF-8?q?on=20update?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - ROADMAP.md: mark all v0.9.8–v0.9.10 items shipped; Phase 4 accountability features complete; v1.0 criteria all met; update to reflect current state as of v0.9.10 - DEPLOYING.md: add SMTP/email section, ENCRYPTION_SECRET_KEY entry, fix OPENAI_MODEL default (gpt-4o → gpt-4o-mini), add pocketveto.org reference - UPDATING.md: replace personal git remote with YOUR_GIT_REMOTE placeholder for public deployability - ARCHITECTURE.md: add member_scores table, alignment API, LLM Batch API, email unsubscribe, bill tab UI, topic tags constant, Fernet encryption pattern, feature history through v0.9.10 Authored by: Jack Levy --- ARCHITECTURE.md | 92 ++++++++++++++++++++++++++++++++++++++++++++++++- DEPLOYING.md | 15 +++++++- ROADMAP.md | 18 +++++++--- UPDATING.md | 6 ++-- 4 files changed, 121 insertions(+), 10 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 41668f8..69b94a0 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -397,6 +397,25 @@ composite = newsapi_pts + gnews_pts + gtrends_pts # range 0–100 --- +### `member_scores` +Nightly-computed effectiveness scores for members of Congress. + +| Column | Type | Notes | +|---|---|---| +| id | int (PK) | | +| member_id | varchar (FK → members, CASCADE) | bioguide_id | +| score_date | date | | +| sponsored_count | int | Bills sponsored in current congress | +| advanced_count | int | Sponsored bills that advanced at least one stage | +| cosponsored_count | int | Bills cosponsored | +| enacted_count | int | Sponsored bills enacted into law | +| composite_score | float | Weighted effectiveness score 0–100 | +| calculated_at | timestamptz | | + +Unique constraint: `(member_id, score_date)`. Nightly Celery task `calculate_member_effectiveness_scores` populates this table. + +--- + ### `app_settings` Key-value store for runtime-configurable settings. @@ -406,6 +425,7 @@ Key-value store for runtime-configurable settings. | `llm_provider` | Overrides `LLM_PROVIDER` env var | | `llm_model` | Overrides provider default model | | `congress_poll_interval_minutes` | Overrides env var | +| `llm_active_batch` | JSON blob tracking in-flight LLM batch jobs (OpenAI / Anthropic Batch API) | --- @@ -562,6 +582,7 @@ Auth header: `Authorization: Bearer ` | POST | `/test/rss` | Required | Generate a test RSS entry and return event count. | | POST | `/test/follow-mode` | Required | Simulate a follow-mode notification to preview delivery behavior. | | GET | `/history` | Required | Recent notification events (dispatched + pending). | +| GET | `/unsubscribe/{token}` | — | One-click email unsubscribe; invalidates token and disables email for the user. | ### `/api/collections` @@ -606,6 +627,12 @@ Auth header: `Authorization: Bearer ` | POST | `/bills/{bill_id}/reprocess` | Admin | Queue document + action fetches for a specific bill (debugging). | | GET | `/task-status/{task_id}` | Admin | Celery task status and result. | +### `/api/alignment` + +| Method | Path | Auth | Description | +|---|---|---|---| +| GET | `/` | Required | Representation Alignment View — for each followed member, compares their roll-call vote positions against the user's followed topics. Returns alignment breakdown per member. Neutral presentation — no scorecard framing. | + ### `/api/health` | Method | Path | Description | @@ -644,7 +671,9 @@ Auth header: `Authorization: Bearer ` | Every 12 hours (at :30) | `fetch_news_for_active_members` | Ongoing | | Daily 3 AM UTC | `calculate_all_member_trend_scores` | Nightly | | Daily 4 AM UTC | `fetch_actions_for_active_bills` | Nightly | +| Daily 5 AM UTC | `calculate_member_effectiveness_scores` | Nightly | | Every 5 minutes | `dispatch_notifications` | Continuous | +| Every 10 minutes | `poll_llm_batch_results` | Continuous (when a batch is active) | | Mondays 8:30 AM UTC | `send_weekly_digest` | Weekly | --- @@ -788,6 +817,15 @@ class ReverseBrief: **Amendment brief prompt** focuses on what changed between document versions. +### LLM Batch API + +OpenAI and Anthropic support async batch endpoints that process requests at ~50% of the standard per-token cost, with results delivered within 24 hours. + +- `submit_llm_batch` Celery task — collects pending documents, submits a batch request to the provider's batch endpoint, and stores the batch ID + document mapping in `AppSetting("llm_active_batch")`. +- `poll_llm_batch_results` Celery task — runs every 10 minutes; checks batch status; on completion, writes `BillBrief` records from the returned results and clears the active batch state. +- Batch mode is opt-in per provider. When active, `process_document_with_llm` routes jobs to the batch queue instead of calling the real-time API directly. +- State key `llm_active_batch` in `app_settings` is a JSON blob: `{provider, batch_id, document_ids: [...]}`. + **Smart truncation:** Bills exceeding the token budget are trimmed — 75% of budget from the start (preamble/purpose), 25% from the end (enforcement/effective dates), with an omission notice in the middle. **Token budgets:** @@ -808,7 +846,7 @@ class ReverseBrief: |---|---| | `/` | Dashboard — personalized feed + trending bills | | `/bills` | Browse all bills with search, chamber/topic filters, pagination | -| `/bills/[id]` | Bill detail — brief with § citations, action timeline, news, trend chart, draft letter, notes | +| `/bills/[id]` | Bill detail — tabbed UI (Analysis, Timeline, Votes, Notes); AI brief with § citations, action timeline, roll-call votes, draft letter, personal notes | | `/members` | Browse members of Congress, filter by chamber/party/state | | `/members/[id]` | Member profile — bio, contact info, leadership roles, service history, sponsored bills, public interest trend chart, recent news | | `/following` | User's followed bills, members, and topics with accordion sections and topic filters | @@ -863,6 +901,19 @@ Line chart of `composite_score` over time with tooltip breakdown of each data so **`WelcomeBanner.tsx`** Dismissible onboarding card rendered at the top of the dashboard. Shown only to guests (no JWT token). On dismiss — via the × button, the "Dismiss" link, or the "Browse Bills" CTA — sets `localStorage["pv_seen_welcome"] = "1"` and hides permanently. Reads localStorage after mount to avoid hydration mismatch; renders nothing until client-side state is resolved. +### Topic Tags (`lib/topics.ts`) + +Shared constant `KNOWN_TOPICS: string[]` — the canonical list of 20 policy topics used across the app. Topic tag pills on the bill detail page and bill listing pages are filtered to this list, preventing spurious LLM-generated tags from appearing as clickable filters. + +```typescript +export const KNOWN_TOPICS = [ + "agriculture", "budget", "defense", "education", "energy", + "environment", "finance", "foreign-policy", "healthcare", "housing", + "immigration", "infrastructure", "judiciary", "labor", "science", + "social-security", "taxation", "technology", "trade", "veterans" +] +``` + ### Utility Functions (`lib/utils.ts`) ```typescript @@ -940,6 +991,9 @@ All LLM providers implement the same interface. Switching providers is a single ### Redis-backed Beat Schedule (RedBeat) The Celery Beat schedule is stored in Redis rather than in memory. This means the beat scheduler can restart without losing schedule state or double-firing tasks. +### Encrypted Sensitive Preferences +ntfy passwords (basic auth credentials stored in `notification_prefs` JSONB) are encrypted at rest using Fernet symmetric encryption (`app/core/crypto.py`). The encryption key is read from `ENCRYPTION_SECRET_KEY` env var. Fernet uses AES-128-CBC with HMAC-SHA256 authentication. The key must be generated once (`Fernet.generate_key()`) and must not change after data is written — re-encryption is required if the key is rotated. All other notification prefs (ntfy topic URL, token auth, quiet hours, etc.) are stored in plaintext. + ### Docker DNS Re-resolution Nginx uses `resolver 127.0.0.11 valid=10s` (Docker's internal DNS) so upstream container IPs are refreshed every 10 seconds. Without this, nginx caches the IP at startup and returns 502 errors after any container is recreated. @@ -1192,6 +1246,42 @@ Nginx uses `resolver 127.0.0.11 valid=10s` (Docker's internal DNS) so upstream c - How It Works page updated with accurate per-mode default alert sets and filter customization guidance - No DB migration required — `alert_filters` stored in existing JSONB column +### v0.9.5–v0.9.7 — Roll-Call Votes, Discovery Alerts & API Optimizations + +- `bill_votes` and `member_vote_positions` tables; on-demand fetch from Congress.gov `/votes` endpoint +- `VotePanel` on bill detail shows yea/nay bar + followed member positions +- Discovery alert filters: member follow and topic follow events carry `source=member_follow|topic_follow`; filtered via `alert_filters.{source}.enabled` and per-category booleans +- Event payloads carry `follow_mode`, `matched_member_name`, `matched_member_id`, `matched_topic` for "why" reason lines in ntfy + notification history UI +- API optimizations: quota batching, ETag caching, async sponsor resolution + +### v0.9.8 — LLM Batch API + +- OpenAI and Anthropic async batch endpoints integrated; ~50% cost reduction for brief generation +- `submit_llm_batch` and `poll_llm_batch_results` Celery tasks +- Batch state stored in `AppSetting("llm_active_batch")` as a JSON blob +- Admin panel shows active batch status and allows manual batch submission/poll trigger + +### v0.9.9–v0.9.10 — Accountability & Email Notifications + +**Accountability (Phase 4):** +- Vote History surfaced in the bill action timeline; roll-call vote events tagged in the timeline view +- Member Effectiveness Score: nightly `calculate_member_effectiveness_scores` Celery task; transparent formula (sponsored, advanced, cosponsored, enacted); `member_scores` table; displayed on member profile with formula explanation +- Representation Alignment View: `GET /api/alignment` compares stanced follows vs member roll-call vote positions; neutral, informational presentation + +**Email Notifications (v0.9.10):** +- SMTP integration using `smtplib`; port 465 = `SMTP_SSL`; port 587 = `STARTTLS` +- Resend-compatible (and generic SMTP relay) +- HTML email templates with bill summary, action details, and one-click unsubscribe token +- Per-user opt-in via notification settings UI +- Unsubscribe tokens are single-use, stored in the DB, and invalidated on use + +**UX & Polish (v0.9.x):** +- Bill detail page refactored to a four-tab layout: Analysis, Timeline, Votes, Notes +- Topic tag pills on bill detail and listing pages — filtered to `KNOWN_TOPICS` from `frontend/lib/topics.ts` +- Collapsible sidebar: icon-only mode, collapse state persisted to `localStorage` +- Favicon: Landmark icon +- ntfy passwords encrypted at rest with Fernet (`app/core/crypto.py`); key from `ENCRYPTION_SECRET_KEY` + --- ## Deployment diff --git a/DEPLOYING.md b/DEPLOYING.md index 92be939..22fea33 100644 --- a/DEPLOYING.md +++ b/DEPLOYING.md @@ -28,6 +28,7 @@ Google Trends (`pytrends`) needs no key. ```bash git clone https://git.jackhlevy.com/jack/civicstack.git +# (Replace with your own fork URL or download a release from pocketveto.org) cd civicstack ``` @@ -50,6 +51,9 @@ PUBLIC_URL= # leave blank unless you have a public do # Auth — generate with: python -c "import secrets; print(secrets.token_hex(32))" JWT_SECRET_KEY=your-generated-secret +# Encryption key for sensitive prefs (generate once, never change after data is written) +ENCRYPTION_SECRET_KEY= # generate: python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())" + # PostgreSQL POSTGRES_USER=congress POSTGRES_PASSWORD=your-strong-password @@ -64,7 +68,7 @@ DATA_GOV_API_KEY=your-api-key # LLM — pick one LLM_PROVIDER=openai OPENAI_API_KEY=sk-... -OPENAI_MODEL=gpt-4o +OPENAI_MODEL=gpt-4o-mini ``` Other providers (swap in place of the OpenAI block): @@ -92,6 +96,15 @@ PYTRENDS_ENABLED=true # Google Trends; disable if hitting rate CONGRESS_POLL_INTERVAL_MINUTES=30 # how often to check Congress.gov ``` +```env +# Email notifications (optional — requires SMTP relay, e.g. Resend) +SMTP_HOST=smtp.resend.com +SMTP_PORT=465 +SMTP_USER=resend +SMTP_PASSWORD=re_your-api-key +SMTP_FROM=alerts@yourdomain.com +``` + --- ## 3. Build and start diff --git a/ROADMAP.md b/ROADMAP.md index e9233e4..b9ebb31 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,6 +1,6 @@ # PocketVeto — Roadmap -> Consolidated feature roadmap and release history. Shipped items reflect what is live as of v0.9.3. +> Consolidated feature roadmap and release history. Shipped items reflect what is live as of v0.9.10. --- @@ -57,6 +57,7 @@ - [x] Change-driven alerts — `categorize_action()` maps action text to 6 named categories; all three follow types (bill, sponsor, topic) covered - [x] Granular per-mode alert filters — 8 independently toggleable alert types per follow mode (Follow / Pocket Veto / Pocket Boost); preset defaults; tabbed UI with per-tab save - [x] Roll-call votes — `bill_votes` + `member_vote_positions` tables; on-demand fetch from Congress.gov `/votes` endpoint; VotePanel on bill detail shows yea/nay bar + followed member positions +- [x] Email notifications — SMTP / Resend integration; HTML templates; one-click unsubscribe tokens; per-user opt-in (v0.9.10) ### UX & Polish - [x] Party badges — solid red/blue/slate, readable in light and dark mode @@ -65,6 +66,11 @@ - [x] Following page — accordion sections, search, topic filters - [x] Member search — "First Last" and "Last, First" both match via PostgreSQL `split_part()` - [x] How It Works page — feature guide covering follow modes, alert filter customization, collections, notifications, AI briefs +- [x] Bill detail page tab UI — four tabs: Analysis, Timeline, Votes, Notes +- [x] Topic tag pills on bill detail and listing pages — clickable, filtered to 20 known topics +- [x] Collapsible sidebar — icon-only mode, `localStorage` persistence +- [x] Favicon — Landmark icon +- [x] LLM Batch API — OpenAI + Anthropic async batch endpoints; 50% cost reduction; state tracked in `AppSetting("llm_active_batch")` --- @@ -72,9 +78,9 @@ ### Phase 4 — Accountability -- [ ] **Vote History & Timeline** — surface roll-call votes in the action timeline; add a "Votes" filter to the bills list so users can find bills that have had floor votes. -- [ ] **Member Effectiveness Score** — nightly Celery task; transparent formula: sponsored bills, bills advanced through stages, co-sponsorships, committee participation, bills enacted. Stored in `member_scores`. Displayed on member profile with formula explanation. -- [ ] **Representation Alignment View** — for each followed member, show how their votes and actions align with the user's followed topics. Neutral presentation — no scorecard framing. +- [x] **Vote History & Timeline** — surface roll-call votes in the action timeline; add a "Votes" filter to the bills list so users can find bills that have had floor votes. +- [x] **Member Effectiveness Score** — nightly Celery task; transparent formula: sponsored bills, bills advanced through stages, co-sponsorships, committee participation, bills enacted. Stored in `member_scores`. Displayed on member profile with formula explanation. +- [x] **Representation Alignment View** — for each followed member, show how their votes and actions align with the user's followed topics. Neutral presentation — no scorecard framing. ### Phase 5 — Search & Polish @@ -107,7 +113,9 @@ v1.0 ships when the following are all live: - [x] Draft letter generator - [x] Public browsing (no account required to read) - [x] Multi-user auth with admin panel -- [ ] Member effectiveness score +- [x] Member effectiveness score - [x] Roll-call vote data All other items above are post-v1.0. + +All v1.0 criteria are met. v1.0 pending final code review and documentation pass. diff --git a/UPDATING.md b/UPDATING.md index e4af0c6..390c4e7 100644 --- a/UPDATING.md +++ b/UPDATING.md @@ -9,7 +9,7 @@ How to push new code from your development machine and pull it on the production The workflow is: ``` -Local machine → git push → git.jackhlevy.com → (pull on server) → docker compose up --build -d +Local machine → git push → YOUR_GIT_REMOTE → (pull on server) → docker compose up --build -d ``` You develop locally, push to the Gitea remote, then update the production server — either manually over SSH or via an automated webhook. @@ -42,7 +42,7 @@ On the server, clone from your Gitea instance: ```bash ssh user@YOUR_SERVER_IP cd /opt # or wherever you want to host it -git clone https://git.jackhlevy.com/jack/civicstack.git +git clone https://YOUR_GIT_REMOTE.git cd civicstack ``` @@ -54,7 +54,7 @@ If your Gitea repo is private, create a **deploy token** in Gitea: Store credentials so `git pull` doesn't prompt: ```bash # Using a personal access token stored in the URL -git remote set-url origin https://YOUR_TOKEN@git.jackhlevy.com/jack/civicstack.git +git remote set-url origin https://YOUR_TOKEN@YOUR_GIT_REMOTE.git ``` Verify: