diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 3f0aef6..41668f8 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -312,7 +312,7 @@ News articles correlated to a specific member of Congress. | email | varchar (unique) | | | hashed_password | varchar | bcrypt | | is_admin | bool | First registered user = true | -| notification_prefs | jsonb | ntfy topic URL, ntfy auth method/token/credentials, ntfy enabled, RSS token, quiet_hours_start/end (0–23 local), timezone (IANA name, e.g. `America/New_York`) | +| notification_prefs | jsonb | ntfy topic URL, ntfy auth method/token/credentials, ntfy enabled, RSS token, quiet_hours_start/end (0–23 local), timezone (IANA name, e.g. `America/New_York`), alert_filters (nested dict: `{neutral: {...}, pocket_veto: {...}, pocket_boost: {...}}` — 8 boolean keys per mode) | | rss_token | varchar (nullable) | Unique token for personal RSS feed URL | | created_at | timestamptz | | @@ -418,11 +418,39 @@ Stores notification events for dispatching to user channels (ntfy, RSS). | user_id | int (FK → users, CASCADE) | | | bill_id | varchar (FK → bills, SET NULL) | nullable | | event_type | varchar | `new_document`, `new_amendment`, `bill_updated`, `weekly_digest` | -| payload | jsonb | `{bill_title, bill_label, brief_summary, bill_url, milestone_tier}` | +| payload | jsonb | `{bill_title, bill_label, brief_summary, bill_url, action_category, milestone_tier}` | | dispatched_at | timestamptz (nullable) | NULL = pending dispatch | | created_at | timestamptz | | -`milestone_tier` in payload: `"progress"` (passed, signed, markup, conference, etc.) or `"referral"` (committee referral). Neutral follows silently skip referral-tier events; pocket_veto and pocket_boost receive them as early warnings. +`action_category` in payload (new events): one of `vote`, `presidential`, `committee_report`, `calendar`, `procedural`, `referral`. `milestone_tier` is retained for backwards compatibility (`"referral"` or `"progress"`). The dispatcher checks `notification_prefs.alert_filters[follow_mode][action_category]` to decide whether to send. `new_document` and `new_amendment` events are filtered by event type directly (not action_category). + +--- + +### `collections` +Named, curated groups of bills. Shareable via UUID token. + +| Column | Type | Notes | +|---|---|---| +| id | int (PK) | | +| user_id | int (FK → users, CASCADE) | | +| name | varchar | 1–100 characters | +| slug | varchar | URL-safe version of name | +| is_public | bool | Signals inclusion in future public directory | +| share_token | uuid | Unique share URL token — read-only for non-owners | +| created_at | timestamptz | | + +--- + +### `collection_bills` +Join table linking bills to collections. + +| Column | Type | Notes | +|---|---|---| +| collection_id | int (FK → collections, CASCADE) | | +| bill_id | varchar (FK → bills, CASCADE) | | +| added_at | timestamptz | | + +Unique constraint: `(collection_id, bill_id)`. --- @@ -444,6 +472,8 @@ Stores notification events for dispatching to user channels (ntfy, RSS). | `0012_dedupe_bill_actions_unique.py` | Unique constraint on `(bill_id, action_date, action_text)` for idempotent action ingestion | | `0013_add_follow_mode.py` | `follow_mode` column on `follows` (`neutral` / `pocket_veto` / `pocket_boost`) | | `0014_add_bill_notes.py` | `bill_notes` table with unique constraint `(user_id, bill_id)` | +| `0015_add_collections.py` | `collections` table (`id`, `user_id`, `name`, `slug`, `is_public`, `share_token` UUID, `created_at`) | +| `0016_add_collection_bills.py` | `collection_bills` join table (`collection_id` FK, `bill_id` FK, `added_at`); `share_token` UUID column on `bill_briefs` | Migrations run automatically on API startup: `alembic upgrade head`. @@ -533,6 +563,26 @@ Auth header: `Authorization: Bearer ` | POST | `/test/follow-mode` | Required | Simulate a follow-mode notification to preview delivery behavior. | | GET | `/history` | Required | Recent notification events (dispatched + pending). | +### `/api/collections` + +| Method | Path | Auth | Description | +|---|---|---|---| +| GET | `/` | Required | Current user's collections with bill count. | +| POST | `/` | Required | Create collection `{name, is_public}`. Generates slug + share_token. | +| GET | `/{id}` | Required | Collection detail with bills. | +| PUT | `/{id}` | Required | Update `{name, is_public}`. | +| DELETE | `/{id}` | Required | Delete collection (owner only). | +| POST | `/{id}/bills/{bill_id}` | Required | Add bill to collection. | +| DELETE | `/{id}/bills/{bill_id}` | Required | Remove bill from collection. | +| GET | `/share/{token}` | — | Public read-only view of a collection by share token. | + +### `/api/share` + +| Method | Path | Auth | Description | +|---|---|---|---| +| GET | `/brief/{token}` | — | Public brief + bill data by share token (from `bill_briefs.share_token`). | +| GET | `/collection/{token}` | — | Public collection + bills by share token. | + ### `/api/admin` | Method | Path | Auth | Description | @@ -610,12 +660,15 @@ Auth header: `Authorization: Bearer ` has no sponsor data), upserts Member, sets bill.sponsor_id ↳ New bills → fetch_bill_documents.delay(bill_id) ↳ Updated bills → fetch_bill_documents.delay(bill_id) if changed - ↳ Updated bills → emit bill_updated notification if action is a milestone: - - "progress" tier: passed/failed, signed/vetoed, enacted, markup, conference, - reported from committee, placed on calendar, cloture, roll call - → all follow types (bill, sponsor, topic) receive notification - - "referral" tier: referred to committee - → pocket_veto and pocket_boost only; neutral follows silently skip + ↳ Updated bills → emit bill_updated notification if categorize_action() returns a category: + - vote: passed, failed, agreed to, roll call + - presidential: signed, vetoed, enacted, presented to the president + - committee_report: markup, ordered to be reported, ordered reported, reported by, discharged + - calendar: placed on + - procedural: cloture, conference + - referral: referred to + → All three follow types (bill, sponsor, topic) receive notification. + Whether it is dispatched depends on the user's per-mode alert_filters in notification_prefs. 2. document_fetcher.fetch_bill_documents(bill_id) ↳ Gets text versions from Congress.gov (XML preferred, falls back to HTML/PDF) @@ -755,14 +808,20 @@ class ReverseBrief: |---|---| | `/` | Dashboard — personalized feed + trending bills | | `/bills` | Browse all bills with search, chamber/topic filters, pagination | -| `/bills/[id]` | Bill detail — brief with § citations, action timeline, news, trend chart | +| `/bills/[id]` | Bill detail — brief with § citations, action timeline, news, trend chart, draft letter, notes | | `/members` | Browse members of Congress, filter by chamber/party/state | | `/members/[id]` | Member profile — bio, contact info, leadership roles, service history, sponsored bills, public interest trend chart, recent news | -| `/following` | User's followed bills, members, and topics | +| `/following` | User's followed bills, members, and topics with accordion sections and topic filters | | `/topics` | Browse and follow policy topics | +| `/collections` | User's collections (watchlists) — create, manage, share | +| `/collections/[id]` | Collection detail — bills in the collection, share link | +| `/notifications` | Notification settings — ntfy config, alert filters (per follow mode), quiet hours, digest mode, RSS | +| `/how-it-works` | Feature guide covering follow modes, collections, notifications, AI briefs, bill browsing | | `/settings` | Admin panel (admin only) | | `/login` | Email + password sign-in | | `/register` | Account creation | +| `/share/brief/[token]` | Public shareable brief view — no sidebar, no auth required | +| `/share/collection/[token]` | Public shareable collection view — no sidebar, no auth required | ### Key Components @@ -1100,6 +1159,39 @@ Nginx uses `resolver 127.0.0.11 valid=10s` (Docker's internal DNS) so upstream c - Tailwind content scan extended to include `lib/` directory - Nginx DNS resolver fix: prevents stale-IP 502s after container restarts +### v0.9.0 — Collections, Shareable Links & Notification Improvements + +**Collections / Watchlists:** +- `collections` and `collection_bills` tables (migrations 0015/0016) +- Named, curated bill groups with public/private flag and UUID share token +- `CollectionPicker` popover on bill detail page — create or add to existing collections from the bookmark icon +- `/collections` page — list, create, manage collections +- `/collections/[id]` — collection detail with share link +- `GET /api/collections/share/{token}` — public read-only collection view + +**Shareable Brief Links:** +- `share_token` UUID column on `bill_briefs` (migration 0016) +- Share button (Share2 icon) in `BriefPanel` copies a public link +- `/api/share/brief/{token}` and `/api/share/collection/{token}` — public router, no auth +- `/share/brief/[token]` and `/share/collection/[token]` — public frontend pages, no sidebar +- `AuthGuard` updated: `/collections` → AUTH_REQUIRED; `/share/` prefix → NO_SHELL (uses `.startsWith()`) + +**Notification Improvements:** +- Quiet hours dispatch fix: events held during quiet window fire correctly on next run +- Digest mode: bundled ntfy summary on daily or weekly schedule instead of per-event pushes +- `send_notification_digest` Celery task with `digest_frequency: "daily" | "weekly"` setting +- Notification history panel on settings page split into direct follows vs topic follows + +### v0.9.3 — Granular Per-Mode Alert Filters + +- `categorize_action()` replaces `is_milestone_action()` / `is_referral_action()` — maps action text to one of 6 named categories: `vote`, `presidential`, `committee_report`, `calendar`, `procedural`, `referral` +- `action_category` stored in every `NotificationEvent.payload` going forward; `milestone_tier` retained for RSS/history backward compatibility +- `alert_filters` added to `notification_prefs` JSONB — nested dict: `{neutral: {...}, pocket_veto: {...}, pocket_boost: {...}}`, each with 8 boolean keys covering all event types (`new_document`, `new_amendment`, and the 6 action categories) +- Dispatcher `_should_dispatch()` checks `prefs["alert_filters"][follow_mode][key]` — replaces two hardcoded per-mode suppression blocks +- Notifications settings page: tabbed **Alert Filters** section (Follow / Pocket Veto / Pocket Boost tabs), each with 8 independent toggles, Milestones parent checkbox (indeterminate-aware), **Load defaults** revert button, and per-tab **Save** button +- How It Works page updated with accurate per-mode default alert sets and filter customization guidance +- No DB migration required — `alert_filters` stored in existing JSONB column + --- ## Deployment diff --git a/DEPLOYING.md b/DEPLOYING.md new file mode 100644 index 0000000..92be939 --- /dev/null +++ b/DEPLOYING.md @@ -0,0 +1,227 @@ +# Deploying PocketVeto + +Step-by-step guide for standing up the full stack on a fresh server. + +--- + +## Prerequisites + +**Server:** +- Linux (Ubuntu 22.04+ or Debian 12 recommended) +- Docker Engine 24+ and Docker Compose v2 (`docker compose` — note: no hyphen) +- At least 2 GB RAM (4 GB recommended if running an Ollama LLM locally) +- Port 80 open to the internet (and 443 if you add SSL) + +**API keys you will need:** + +| Key | Where to get it | Required? | +|---|---|---| +| `DATA_GOV_API_KEY` | [api.data.gov/signup](https://api.data.gov/signup/) — free, instant | **Yes** | +| One LLM key (OpenAI / Anthropic / Gemini) | Provider dashboard | **Yes** (or use Ollama) | +| `NEWSAPI_KEY` | [newsapi.org](https://newsapi.org) — free tier (100 req/day) | Optional | + +Google Trends (`pytrends`) needs no key. + +--- + +## 1. Get the code + +```bash +git clone https://git.jackhlevy.com/jack/civicstack.git +cd civicstack +``` + +--- + +## 2. Configure environment + +```bash +cp .env.example .env +nano .env # or your preferred editor +``` + +**Minimum required values:** + +```env +# Network +LOCAL_URL=http://YOUR_SERVER_IP # or https://yourdomain.com if behind SSL +PUBLIC_URL= # leave blank unless you have a public domain + +# Auth — generate with: python -c "import secrets; print(secrets.token_hex(32))" +JWT_SECRET_KEY=your-generated-secret + +# PostgreSQL +POSTGRES_USER=congress +POSTGRES_PASSWORD=your-strong-password +POSTGRES_DB=pocketveto + +# Redis +REDIS_URL=redis://redis:6379/0 + +# Congress.gov + GovInfo (shared api.data.gov key) +DATA_GOV_API_KEY=your-api-key + +# LLM — pick one +LLM_PROVIDER=openai +OPENAI_API_KEY=sk-... +OPENAI_MODEL=gpt-4o +``` + +Other providers (swap in place of the OpenAI block): +```env +# Anthropic +LLM_PROVIDER=anthropic +ANTHROPIC_API_KEY=sk-ant-... +ANTHROPIC_MODEL=claude-sonnet-4-6 + +# Gemini +LLM_PROVIDER=gemini +GEMINI_API_KEY=AIza... +GEMINI_MODEL=gemini-2.0-flash + +# Ollama (local model — server must be running on the host) +LLM_PROVIDER=ollama +OLLAMA_BASE_URL=http://host.docker.internal:11434 +OLLAMA_MODEL=llama3.1 +``` + +Optional extras: +```env +NEWSAPI_KEY=your-newsapi-key # enables richer news correlation +PYTRENDS_ENABLED=true # Google Trends; disable if hitting rate limits +CONGRESS_POLL_INTERVAL_MINUTES=30 # how often to check Congress.gov +``` + +--- + +## 3. Build and start + +```bash +docker compose up --build -d +``` + +This will: +1. Pull base images (postgres, redis, nginx, node) +2. Build the API, worker, beat, and frontend images +3. Start all 7 containers +4. Run `alembic upgrade head` automatically inside the API container on startup +5. Seed the Celery Beat schedule in Redis + +**First build takes 3–8 minutes** depending on your server. Subsequent builds are faster (Docker layer cache). + +--- + +## 4. Verify it's running + +```bash +docker compose ps +``` + +All services should show `Up`: +``` +civicstack-api-1 Up +civicstack-beat-1 Up +civicstack-frontend-1 Up +civicstack-nginx-1 Up 0.0.0.0:80->80/tcp +civicstack-postgres-1 Up (healthy) +civicstack-redis-1 Up (healthy) +civicstack-worker-1 Up +``` + +Check the API health endpoint: +```bash +curl http://localhost/api/health +# → {"status":"ok","timestamp":"..."} +``` + +Open `http://YOUR_SERVER_IP` in a browser. + +--- + +## 5. Create the admin account + +Navigate to `http://YOUR_SERVER_IP/register` and create the first account. + +**The first registered account automatically becomes admin.** All subsequent accounts are regular users. The admin account gets access to the Settings page with the pipeline controls, LLM switching, and user management. + +--- + +## 6. Trigger initial data load + +Log in as admin and go to **Settings**: + +1. **Trigger Poll** — fetches bills updated in the last 60 days from Congress.gov (~5–10 minutes to complete) +2. **Sync Members** — syncs current Congress members (~2 minutes) + +The Celery workers then automatically: +- Fetch bill text from GovInfo +- Generate AI briefs (rate-limited at 10/minute) +- Fetch news articles and calculate trend scores + +You can watch progress in **Settings → Pipeline Status**. + +--- + +## 7. Optional: Domain + SSL + +If you have a domain name pointing to the server, add an SSL terminator in front of nginx. The simplest approach is Caddy as a reverse proxy: + +```bash +# Install Caddy +apt install -y debian-keyring debian-archive-keyring apt-transport-https +curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg +curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | tee /etc/apt/sources.list.d/caddy-stable.list +apt update && apt install caddy +``` + +`/etc/caddy/Caddyfile`: +``` +yourdomain.com { + reverse_proxy localhost:80 +} +``` + +```bash +systemctl reload caddy +``` + +Caddy handles HTTPS certificates automatically via Let's Encrypt. + +After adding SSL, update `.env`: +```env +PUBLIC_URL=https://yourdomain.com +``` + +Then rebuild the API so the new URL is used in notification payloads: +```bash +docker compose up --build -d api +``` + +--- + +## Useful commands + +```bash +# View logs for a service +docker compose logs --tail=50 api +docker compose logs --tail=50 worker +docker compose logs -f worker # follow in real time + +# Restart a service +docker compose restart worker + +# Run a database query +docker compose exec postgres psql -U congress pocketveto + +# Apply any pending migrations manually +docker compose exec api alembic upgrade head + +# Open a Python shell inside the API container +docker compose exec api python +``` + +--- + +## Troubleshooting + +See `TROUBLESHOOTING.md` for common issues (502 errors after rebuild, wrong postgres user, frontend changes not showing, etc.). diff --git a/MVP threshold this make v1 complete.md b/MVP threshold this make v1 complete.md deleted file mode 100644 index 934bc4b..0000000 --- a/MVP threshold this make v1 complete.md +++ /dev/null @@ -1,93 +0,0 @@ - -### Phase 3 — Personal Workflow & Context - -_Focus: Allowing users to organize data and adding the "Cross-Bill" intelligence layer._ - -- **Collections / Watchlists:** `collections` table for public/private bill groupings shareable via UUID tokens. - -- **Personal Notes:** Private `bill_notes` table to store user stances, tags, and pinned insights on bill detail pages. - - -- **Cross-Bill Referencing (New):** - - - **Extraction:** Update LLM prompts to identify and extract related bill identifiers (e.g., "H.R. 1234") from text. - - - **Referential Table:** Implement `bill_references` to map source-to-target bill relationships with context quotes. - - - **Ghost Hydration:** Create the `/api/admin/bills/hydrate` endpoint to fetch and analyze referenced bills not currently in the DB. - - - **UI Hover Previews:** Use `Radix UI` popovers in the `BriefPanel` to show snapshots of referenced legislation without leaving the page. - -- **Weekly Digest:** Celery beat task to dispatch 7-day summaries of followed bills via ntfy and RSS. - - ---- - -### Phase 4 — Accountability & Member Scoring - -_Focus: Deep-diving into legislative performance and voting records._ - -- **Votes & Committees:** Fetch roll-call votes and committee actions from Congress.gov into a new `bill_votes` table. - -- **Member Effectiveness Score:** Nightly Celery task calculating scores based on sponsored bills, advancement milestones, and "bills enacted" metrics. - -- **Bipartisan Progress Metric (Proposed):** Enhance effectiveness scores by weighting co-sponsorship across party lines and committee movement in opposing chambers. - -- **Representation Alignment:** Neutral view showing how member votes align with a user’s followed topics. - - ---- - -### Phase 5 — Systemic Intelligence - -_Focus: Enhancing search and providing broader legislative context._ - -- **Search Improvements:** Add filters for bill type, status, chamber, and date; implement "Search within member bills". - -- **Topic-Scoped Search:** Allow users to search legislation exclusively within followed topics like "Healthcare" or "Energy". - -- **Member Effectiveness Transparency:** Display the "Member Effectiveness" formula on profile pages to maintain non-partisan trust. - - ---- - -### Phase 6 — Desktop & Polish - -_Focus: Optimization for professional-grade power users._ - -- **Desktop View:** Multi-column layout with sticky sidebars and expanded grids optimized for large screens. - -- **Notification Channels v2:** Expand beyond ntfy/RSS to include Discord webhooks, Telegram, and SMTP email. - -- **Backfill Tasks:** Automate `first_name` and `last_name` population for all member records. - - ---- - -## 🛠️ Updated Technical Implementation Details - -### Database Schema Updates - -|**Table**|**New Columns / Purpose**| -|---|---| -|**`bill_references`**|`source_id`, `target_id`, `context_quote` to link related bills.| -|**`bill_notes`**|`user_id`, `bill_id`, `content`, `stance`, `tags`, `pinned`.| -|**`collections`**|`user_id`, `name`, `slug`, `is_public` for shared watchlists.| -|**`bill_votes`**|Stores roll-call data and results for accountability tracking.| - -### Enhanced LLM Pipeline - -The `llm_processor.py` task will now perform a dual-pass: - -1. **Analysis:** Generate the standard brief with citations and fact/inference labels. - -2. **Mapping:** Extract bill IDs found in the text and insert them into the `bill_references` table. If a target ID is missing from the `bills` table, it triggers a low-priority `hydrate_bill` task. - - -### Performance Guardrails - -- **Selective Hydration:** The poller will only "Ghost Hydrate" referenced bills if they are from the current or previous Congress to prevent API exhaustion. - -- **Cached Previews:** Referenced bill snapshots for hover tooltips will be cached in Redis to minimize database hits during dashboard scrolling. - - diff --git a/PocketVeto — Feature Roadmap.md b/PocketVeto — Feature Roadmap.md deleted file mode 100644 index a1ea744..0000000 --- a/PocketVeto — Feature Roadmap.md +++ /dev/null @@ -1,215 +0,0 @@ -## Roadmap - -- [x] Docker Stack — PostgreSQL, Redis, FastAPI, Celery, Next.js, Nginx fully containerized -- [x] Bill Polling — Congress.gov incremental sync every 30 min, filtered to legislation that can become law -- [x] Document Fetching — GovInfo bill text retrieval with smart truncation for token budgets -- [x] LLM Analysis — Multi-provider AI briefs (OpenAI, Anthropic, Gemini, Ollama) with amendment diffing -- [x] News Correlation — NewsAPI + Google News RSS articles linked to bills via topic tags -- [x] Trend Scoring — Composite zeitgeist score (0–100) from NewsAPI + Google News + Google Trends, nightly -- [x] Full-text Search — PostgreSQL tsvector search across bills and members -- [x] Follows — Per-user follows for bills, members, and topics -- [x] Dashboard — Personalized feed + trending bills -- [x] Multi-user Auth — JWT email/password auth, admin role, user management panel -- [x] Admin Panel — LLM provider switching, pipeline stats, manual task triggers -- [x] Citations — Every AI brief key point and risk cites the section + verbatim quote from bill text -- [x] Citation UI — § chips expand inline to show quote + GovInfo source link -- [x] Party Badges — Solid red/blue/slate badges readable in light and dark mode -- [x] Nginx DNS Fix — Resolver directive prevents stale-IP 502s after container restarts -- [x] Sponsor Linking — Poller fetches bill detail for sponsor; backfill task fixes existing bills -- [x] Member Search — "First Last" and "Last, First" both match via PostgreSQL split_part() -- [x] Search Spaces — Removed .trim() on keystroke that ate spaces in search inputs -- [x] Mobile UI — Responsive layout: sidebar collapses, cards stack, touch-friendly controls -- [x] Member BIO & Photo — Display member headshots (photo_url already stored, not yet shown in UI) -- [x] Bill Action Fetching — BillAction table populated via Congress.gov actions endpoint; nightly batch + event-driven on bill change -- [x] What Changed (Amendment Briefs) — BriefPanel surfaces amendment briefs with "What Changed" badge and collapsible version history -- [x] Source Viewer — "View source" link in § citation popover opens GovInfo document in new tab (Option A; Option B = in-app highlighted viewer deferred pending UX review) -- [x] Admin Reprocess — POST /api/admin/bills/{bill_id}/reprocess queues document + action fetches for a specific bill -- [x] LLM Model Picker — live model list fetched from each provider's API; custom model name fallback -- [x] Admin Health Panel — bill pipeline breakdown table, external API health tests with latency, manual controls with status dots + task polling -- [x] Chamber Badges — amber/gold for Senate, slate/silver for House; applied on bill cards and detail pages -- [x] Action History Fallback — shows latest_action_date/text while full history loads; full timeline once fetched -- [x] Backfill All Actions — admin task to fetch action history for all pre-existing bills -- [x] Notifications (Phase 1) — ntfy dispatch, RSS feed, per-user settings UI, 5-min dispatcher beat task -- [x] Brief Regeneration UI — admin button to delete existing briefs for a bill and re-queue LLM processing. Useful for improving citation/diff logic without a full re-poll. (Backend reprocess endpoint already exists.) -- [x] Follow Modes — `neutral | pocket_veto | pocket_boost` on the `follows` table; FollowButton mode selector with descriptions and tooltips -- [x] Public Browsing — unauthenticated guests browse bills, members, topics, and trending dashboard; AuthModal gates follow/interactive actions; sidebar and nav adapt to guest state -- [x] Draft Letter Generator — collapsible panel on bill detail pages; select up to 3 cited brief points, stance auto-fills from follow mode, recipient derived from chamber, ZIP optional and never stored; calls configured LLM provider -- [x] Bill Text Status Indicators — BillCard shows Brief / Pending / No text badge; backed by a single batch query on the list endpoint - ---- - -## To Do - ---- - -### Phase 2 — High Impact *(can run in parallel after Phase 1)* - -- [x] **Change-driven Alerts** — milestone keywords expanded (markup, conference, referral tier); topic followers now receive bill_updated milestone events; committee referral events delivered to pocket_veto/boost but suppressed for neutral; all three follow types covered for both tiers. -- [x] **Fact vs Inference Labeling** — `label: "cited_fact" | "inference"` on every cited key_point and risk; prompt engineering updated for all providers; "Inferred" badge in citation UI; backfill task for existing briefs. - ---- - -### Phase 3 — Personal Workflow - -- [ ] **Collections / Watchlists** — `collections` (id, user_id, name, slug, is_public) + `collection_bills` join table. UI to create/manage collections and filter dashboard by collection. Shareable via public slug URL (read-only for non-owners). -- [x] **Personal Notes** — `bill_notes` table (user_id, bill_id, content, pinned). Collapsible panel on bill detail page. Pinned notes float above the brief. Private — auth-gated, never shown to guests. -- [ ] **Shareable Links** — UUID token on briefs and collections → public read-only view, no login required. Same token system for both. No expiry by default. UUID (not sequential) to prevent enumeration. -- [x] **Weekly Digest** — Celery beat task (Monday 8:30 AM UTC), queries followed bills for changes in the past 7 days, formats a low-noise summary, dispatches via ntfy + RSS. Admin can trigger immediately from Manual Controls. - ---- - -### Phase 4 — Accountability - -- [ ] **Votes & Committees** — fetch roll-call votes and committee referrals/actions from Congress.gov. New `bill_votes` table. UI: timeline entries for committee actions (already partially populated from bill actions) + vote results filterable by followed members and topics. -- [ ] **Member Effectiveness Score** — nightly Celery task; transparent formula: sponsored bills count, bills advanced through stages, co-sponsored, committee participation, "bills enacted" metric. Stored in `member_scores` table. Displayed on member profile with formula explanation. -- [ ] **Representation Alignment View** — for each followed member, show how their votes and actions align with the user's followed topics. Based purely on followed members (no ZIP/district storage). Neutral presentation — no scorecard dunking. - ---- - -### Phase 5 — Polish *(slot in anytime, independent)* - -- [ ] **Search Improvements** — filters on global search (bill type, status, chamber, date range); search within a member's sponsored bills; topic-scoped search. -- [ ] **Desktop View** — wider multi-column layout optimized for large screens (sticky sidebar, expanded grid, richer bill detail layout). -- [ ] **first_name / last_name Backfill** — Celery task to populate empty first/last from stored "Last, First" `name` field via split. - ---- - -### Later / Backlog - -- [ ] **Notification Channels v2** — email (SMTP), Discord webhook, Telegram bot (after ntfy + RSS v1 ships) -- [ ] **Source Viewer Option B** — in-app bill text viewer with cited passage highlighted and scroll-to-anchor. Deferred pending UX review of Option A (GovInfo link). -- [ ] **Raw Diff Panel** — Python `difflib` diff between stored document versions, shown as collapsible "Raw Changes" below amendment brief. Zero API calls. Deferred — AI amendment brief is the primary "what changed" story. -- [ ] **Shareable Collection Subscriptions** — "Follow this collection" mechanic so other users can subscribe to a public collection and get its bills added to their feed. -- [x] Pocket Veto mode (follow stance) — toggle on a bill to treat it as “I don’t want this to pass”; adds to watchlist and triggers milestone alerts -- [x] Follow modes — Neutral + Pocket Veto + Pocket Boost; FollowButton is a mode selector with explanation tooltips -- [ ] Pocket Veto notification rules — alert only on advancement milestones + failure outcomes (failed committee / failed floor / stalled) — notification dispatcher needs to filter by follow_mode - - -### PocketVeto function - -#### How it should work (so it’s useful and not cringey) - -Instead of “follow/unfollow,” each bill gets a **Follow Mode**: - -- **Follow** (neutral): “Keep me posted on meaningful changes.” - -- **Pocket Veto** (oppose): “Alert me if this bill is advancing toward passage.” - -- (Optional later) **Pocket Boost** (support): “Alert me when action is needed / when it’s in trouble.” also suggest an action the user can take to let their representatives know that you support this bill. - - -For Pocket Veto specifically, the key is **threshold alerts**, not spam: - -- **Committee referral** - -- **Committee hearing scheduled** - -- **Markup scheduled** - -- **Reported out of committee** - -- **Placed on calendar** - -- **Floor vote scheduled** - -- **Passed chamber** - -- **Conference / reconciliation activity** - -- **Sent to President** - -- **Signed / Vetoed** - - -And the “failed” side: - -- **Failed in committee** - -- **Failed floor vote** - -- **Stalled** (no action for X days while similar bills move) - - -#### Why it’s valuable for “normal people” - -Most people don’t want to follow politics continuously. They want: - -- “Tell me if the bad thing is about to happen.” - That’s exactly what Pocket Veto mode does. - - -#### Guardrail to keep it non-partisan / non-toxic - -Make it explicit in UI copy: - -- It’s a **personal alert preference**, not a moral label. - -- It doesn’t publish your stance unless you share it. - - -#### Data model addition (simple) - -Add fields to `follows` (or a new table): - -- `follow_mode`: `neutral | pocket_veto | pocket_boost` - -- `alert_sensitivity`: `low | medium | high` (optional) - - -Then alert rules can be: - -- neutral: material changes - -- pocket_veto: only “advancing toward passage” milestones - -- pocket_boost: “action points” + milestones - - -Yes — that’s a solid idea **if it’s done as a “welcome + how it works” nudge**, not an annoying pop-up that blocks the UI. - -A toast can work, but for a first-time user you’ll usually get better results with a **dismissible banner** or a **one-time “welcome” card** on the dashboard, because: - -* toasts disappear (people miss them) -* first-run onboarding usually needs at least one click (“Got it” / “Start here”) - -### Best pattern (low effort, high impact) - -**First visit → show a dismissible Welcome card/banner** (top of dashboard) with: - -* 1 sentence purpose -* 3 bullets of key features -* 2 buttons: **“Add my first follow”** and **“See a demo bill”** -* “Don’t show again” checkbox (or implicit on dismiss) - -You can still use a toast, but make it: - -* sticky until dismissed -* or paired with a banner/card - -### What it should say (copy you can paste) - -**Title:** Welcome to PocketVeto -**Body (tight):** - -* Follow bills, members, or topics (low-noise) -* See *what changed* in plain English -* Verify every claim with **Back to Source** citations - **Buttons:** “Add a follow” | “Load demo” - -### Implementation detail (no creepy tracking) - -Store a simple flag: - -* `localStorage.setItem("pv_seen_welcome", "1")` - -Don’t store it server-side unless you already have user accounts and it’s part of preferences. - -### Backlog item (checkboxes) - -* [x] First-visit welcome UI (banner/card — guests only, shown on dashboard) -* [x] Dismiss + “don’t show again” (localStorage `pv_seen_welcome`) -* [x] CTA: Browse Bills -* [ ] CTA: Load demo data (optional) -* [ ] Link: “How it works” page/modal (optional) - -If you tell me your UI stack (sounds like Next.js + shadcn/ui), I can give you a drop-in component for a clean welcome card + toast behavior. diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..102a257 --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,112 @@ +# PocketVeto — Roadmap + +> Consolidated feature roadmap and release history. Shipped items reflect what is live as of v0.9.3. + +--- + +## Shipped + +### Foundation & Core Pipeline +- [x] Docker Compose stack — PostgreSQL, Redis, FastAPI, Celery, Next.js, Nginx fully containerized +- [x] Bill polling — Congress.gov incremental sync, filtered to legislation that can become law (`hr`, `s`, `hjres`, `sjres`) +- [x] Document fetching — GovInfo bill text retrieval with smart truncation for LLM token budgets +- [x] LLM analysis — multi-provider AI briefs (OpenAI, Anthropic, Gemini, Ollama) with amendment diffing +- [x] Citations — every key point and risk cites the bill section + verbatim quote; `cited_fact` / `inference` label +- [x] Citation UI — § chips expand inline with quote + GovInfo source link; "Inferred" badge on analytical items +- [x] News correlation — NewsAPI + Google News RSS articles linked to bills via topic tags +- [x] Trend scoring — composite zeitgeist score (0–100) from NewsAPI + Google News + Google Trends, nightly +- [x] Full-text search — PostgreSQL tsvector across bills and members +- [x] Multi-user auth — JWT email/password, admin role, user management panel +- [x] Admin panel — LLM provider/model switching, pipeline stats, external API health, manual task triggers +- [x] Bill type filtering — only legislation that can become law; 60-day seed window on fresh install + +### Member Profiles & Mobile +- [x] Member profiles — bio, contact info, leadership roles, service history, sponsored/cosponsored counts +- [x] Member interest scoring — composite trend score (NewsAPI + GNews + pytrends), nightly; lazy-loaded on first profile view +- [x] Member news — articles correlated to members via name search +- [x] Mobile UI — responsive layout, slide-in drawer, hamburger header, touch-friendly controls + +### Follows & Personalization +- [x] Follows — per-user bill, member, and topic follows +- [x] Follow modes — `neutral | pocket_veto | pocket_boost` with mode-selector UI and descriptions +- [x] Dashboard — personalized feed from followed bills/members/topics + trending +- [x] Public browsing — unauthenticated guests browse bills, members, topics, and trending; `AuthModal` gates interactive actions +- [x] Welcome banner — dismissible onboarding card for guests; `localStorage` dismiss state + +### Bill Detail Features +- [x] Bill action pipeline — full legislative action history from Congress.gov; nightly batch + event-driven on change +- [x] Action timeline — full history with latest-action fallback while history loads +- [x] Amendment briefs — "What Changed" badge; collapsible version history panel +- [x] Bill text status indicators — Brief / Pending / No text badge on BillCard (single batch query) +- [x] Draft letter generator — select up to 3 cited brief points, generate constituent letter via configured LLM; ZIP not stored +- [x] Personal notes — private per-bill note with pin, auto-resize textarea; one note per user per bill +- [x] Sponsor linking — poller fetches bill detail for sponsor; backfill task for existing bills +- [x] "No bill text" state — shown on detail page when GovInfo has nothing published + +### Collections & Sharing +- [x] Collections / Watchlists — named groups of bills, public/private, UUID share token; `CollectionPicker` popover on bill detail +- [x] Shareable brief links — `share_token` on `bill_briefs`; public `/share/brief/[token]` page, no login required +- [x] Shareable collection links — public `/share/collection/[token]` page, no login required + +### Notifications +- [x] ntfy push — Celery dispatcher POSTs to user's ntfy topic; optional token/basic auth +- [x] RSS feed — private tokenized XML feed per user; always real-time +- [x] Quiet hours — local-time window (IANA timezone auto-detected by browser); events queued and sent as batch after window +- [x] Digest mode — bundled ntfy summary on daily or weekly schedule instead of per-event pushes +- [x] Weekly digest — Monday 8:30 AM UTC; 7-day summary of followed bill activity via ntfy + RSS +- [x] Change-driven alerts — `categorize_action()` maps action text to 6 named categories; all three follow types (bill, sponsor, topic) covered +- [x] Granular per-mode alert filters — 8 independently toggleable alert types per follow mode (Follow / Pocket Veto / Pocket Boost); preset defaults; tabbed UI with per-tab save + +### UX & Polish +- [x] Party badges — solid red/blue/slate, readable in light and dark mode +- [x] Chamber badges — amber/gold for Senate, slate/silver for House +- [x] Fact vs inference labeling — `cited_fact` / `inference` on every cited point; backfill task for existing briefs +- [x] Following page — accordion sections, search, topic filters +- [x] Member search — "First Last" and "Last, First" both match via PostgreSQL `split_part()` +- [x] How It Works page — feature guide covering follow modes, alert filter customization, collections, notifications, AI briefs + +--- + +## Up Next + +### Phase 4 — Accountability + +- [ ] **Votes & Committees** — fetch roll-call votes from Congress.gov into a `bill_votes` table. UI: vote results filterable by followed members and topics; timeline entries for committee actions. +- [ ] **Member Effectiveness Score** — nightly Celery task; transparent formula: sponsored bills, bills advanced through stages, co-sponsorships, committee participation, bills enacted. Stored in `member_scores`. Displayed on member profile with formula explanation. +- [ ] **Representation Alignment View** — for each followed member, show how their votes and actions align with the user's followed topics. Neutral presentation — no scorecard framing. + +### Phase 5 — Search & Polish + +- [ ] **Search Improvements** — filters on global search (bill type, status, chamber, date range); search within a member's sponsored bills; topic-scoped search. +- [ ] **Desktop View** — wider multi-column layout for large screens: sticky sidebar, expanded grid, richer bill detail. +- [ ] **first_name / last_name Backfill** — Celery task to populate empty first/last from stored "Last, First" `name` field via split (affects ~10% of member records). + +### Backlog + +- [ ] **Notification Channels v2** — email (SMTP), Discord webhook, Telegram bot. +- [ ] **Cross-Bill Referencing** — extract referenced bill IDs from LLM briefs; `bill_references` table; hover preview popover in BriefPanel for referenced legislation. +- [ ] **Source Viewer Option B** — in-app bill text viewer with cited passage highlighted and scroll-to-anchor. Deferred pending UX review of the current GovInfo-link approach. +- [ ] **Raw Diff Panel** — `difflib` diff between stored document versions, shown as collapsible "Raw Changes" below the amendment brief. Zero API calls. +- [ ] **Shareable Collection Subscriptions** — "Follow this collection" so other users can subscribe to a public collection and get its bills added to their feed. + +--- + +## v1.0 Definition + +v1.0 ships when the following are all live: + +- [x] Full bill pipeline (poll → fetch text → LLM brief → news → trend score) +- [x] Cited AI briefs with fact/inference labels +- [x] Follow modes (neutral / pocket_veto / pocket_boost) +- [x] Granular per-mode notification filters +- [x] Push (ntfy) + RSS + digest + quiet hours +- [x] Collections with share links +- [x] Shareable brief links +- [x] Personal notes +- [x] Draft letter generator +- [x] Public browsing (no account required to read) +- [x] Multi-user auth with admin panel +- [ ] Member effectiveness score +- [ ] Roll-call vote data + +All other items above are post-v1.0. diff --git a/UPDATING.md b/UPDATING.md new file mode 100644 index 0000000..e4af0c6 --- /dev/null +++ b/UPDATING.md @@ -0,0 +1,253 @@ +# Updating PocketVeto — Remote Server Setup & Deployment Workflow + +How to push new code from your development machine and pull it on the production server. + +--- + +## Overview + +The workflow is: + +``` +Local machine → git push → git.jackhlevy.com → (pull on server) → docker compose up --build -d +``` + +You develop locally, push to the Gitea remote, then update the production server — either manually over SSH or via an automated webhook. + +--- + +## 1. SSH access to the production server + +Make sure you can SSH into the server without a password: + +```bash +# On your local machine — generate a key if you don't have one +ssh-keygen -t ed25519 -C "pocketveto-deploy" + +# Copy the public key to the server +ssh-copy-id user@YOUR_SERVER_IP +``` + +Test: +```bash +ssh user@YOUR_SERVER_IP "echo ok" +``` + +--- + +## 2. Server: clone the repo and authenticate + +On the server, clone from your Gitea instance: + +```bash +ssh user@YOUR_SERVER_IP +cd /opt # or wherever you want to host it +git clone https://git.jackhlevy.com/jack/civicstack.git +cd civicstack +``` + +If your Gitea repo is private, create a **deploy token** in Gitea: + +- Gitea → Repository → Settings → Deploy Keys → Add Deploy Key (read-only is fine) +- Or: Gitea → User Settings → Applications → Generate Token + +Store credentials so `git pull` doesn't prompt: +```bash +# Using a personal access token stored in the URL +git remote set-url origin https://YOUR_TOKEN@git.jackhlevy.com/jack/civicstack.git +``` + +Verify: +```bash +git pull # should succeed with no password prompt +``` + +--- + +## 3. Option A — Manual update (simplest) + +SSH in and run: + +```bash +ssh user@YOUR_SERVER_IP +cd /opt/civicstack + +git pull origin main +docker compose up --build -d +``` + +That's it. Docker rebuilds only the images that changed (layer cache means unchanged services rebuild in seconds). Migrations run automatically when the API container restarts. + +**One-liner from your local machine:** +```bash +ssh user@YOUR_SERVER_IP "cd /opt/civicstack && git pull origin main && docker compose up --build -d" +``` + +--- + +## 4. Option B — Deploy script + +Create `/opt/civicstack/deploy.sh` on the server: + +```bash +#!/bin/bash +set -e + +cd /opt/civicstack + +echo "==> Pulling latest code" +git pull origin main + +echo "==> Building and restarting containers" +docker compose up --build -d + +echo "==> Done. Current status:" +docker compose ps +``` + +```bash +chmod +x /opt/civicstack/deploy.sh +``` + +Now from your local machine: +```bash +ssh user@YOUR_SERVER_IP /opt/civicstack/deploy.sh +``` + +--- + +## 5. Option C — Automated webhook (Gitea → server) + +This triggers a deploy automatically every time you push to `main`. + +### 5a. Create a webhook listener on the server + +Install a simple webhook runner. The easiest is [`webhook`](https://github.com/adnanh/webhook): + +```bash +apt install webhook +``` + +Create `/etc/webhook/hooks.json`: +```json +[ + { + "id": "civicstack-deploy", + "execute-command": "/opt/civicstack/deploy.sh", + "command-working-directory": "/opt/civicstack", + "response-message": "Deploying...", + "trigger-rule": { + "match": { + "type": "payload-hmac-sha256", + "secret": "your-webhook-secret", + "parameter": { "source": "header", "name": "X-Gitea-Signature-256" } + } + } + } +] +``` + +Start the webhook service: +```bash +# Test it first +webhook -hooks /etc/webhook/hooks.json -port 9000 -verbose + +# Or create a systemd service (recommended) +``` + +`/etc/systemd/system/webhook.service`: +```ini +[Unit] +Description=Webhook listener for civicstack deploys +After=network.target + +[Service] +ExecStart=/usr/bin/webhook -hooks /etc/webhook/hooks.json -port 9000 +Restart=on-failure +User=root + +[Install] +WantedBy=multi-user.target +``` + +```bash +systemctl enable --now webhook +``` + +Expose port 9000 (or proxy it through nginx/Caddy at a path like `/hooks/`). + +### 5b. Add the webhook in Gitea + +- Gitea → Repository → Settings → Webhooks → Add Webhook → Gitea +- **Target URL:** `http://YOUR_SERVER_IP:9000/hooks/civicstack-deploy` +- **Secret:** same value as `your-webhook-secret` above +- **Trigger:** Push events → branch `main` + +Now every `git push origin main` automatically triggers a deploy. + +--- + +## 6. Checking the deployed version + +After any update you can confirm what's running: + +```bash +# Check the git commit on the server +ssh user@YOUR_SERVER_IP "cd /opt/civicstack && git log --oneline -3" + +# Check container status +ssh user@YOUR_SERVER_IP "cd /opt/civicstack && docker compose ps" + +# Hit the health endpoint +curl http://YOUR_SERVER_IP/api/health +``` + +--- + +## 7. Rolling back + +If a bad deploy goes out: + +```bash +ssh user@YOUR_SERVER_IP +cd /opt/civicstack + +# Roll back to the previous commit +git revert HEAD --no-edit # preferred — creates a revert commit, keeps history clean + +# Or hard reset if you're sure (discards the bad commit locally — use with care) +# git reset --hard HEAD~1 + +git push origin main # if using Option C, this triggers a new deploy automatically +docker compose up --build -d # if manual +``` + +--- + +## 8. Environment and secrets + +`.env` is **not** tracked in git. If you need to update a secret or add a new API key on the server: + +```bash +ssh user@YOUR_SERVER_IP +nano /opt/civicstack/.env + +# Then restart only the affected services (usually api + worker) +cd /opt/civicstack +docker compose up -d --no-build api worker beat +``` + +`--no-build` skips the rebuild step — only a config reload is needed for env var changes. + +--- + +## Summary + +| Goal | Command | +|---|---| +| Manual deploy | `ssh server "cd /opt/civicstack && git pull && docker compose up --build -d"` | +| One-step deploy script | `ssh server /opt/civicstack/deploy.sh` | +| Automated on push | Gitea webhook → webhook listener → `deploy.sh` | +| Rollback | `git revert HEAD` + redeploy | +| Update env only | Edit `.env` on server + `docker compose up -d --no-build api worker beat` | +| Check what's running | `ssh server "cd /opt/civicstack && git log --oneline -1 && docker compose ps"` | diff --git a/frontend/app/how-it-works/page.tsx b/frontend/app/how-it-works/page.tsx index 63eca02..485e5d3 100644 --- a/frontend/app/how-it-works/page.tsx +++ b/frontend/app/how-it-works/page.tsx @@ -5,6 +5,7 @@ import { Calendar, Clock, FileText, + Filter, Heart, HelpCircle, Rss, @@ -82,29 +83,46 @@ export default function HowItWorksPage() { {/* Following */}

- Follow any bill to track it. PocketVeto checks for changes — new text, amendments, status - updates — and notifies you through your configured channels. Three modes let you tune the - signal to your interest level. + Follow any bill to track it. PocketVeto checks for changes and notifies you through your + configured channels. Three modes let you tune the signal to your interest level — each + with its own independent set of alert filters.

- - Alerts on all material changes: new text published, amendments filed, and status updates - like floor votes or committee referrals. + + The standard mode. Default alerts: new bill text, amendments filed, chamber votes, + presidential action, and committee reports. - Alerts only when the bill advances toward becoming law — new official text, floor - scheduling, or committee passage. Useful when you oppose a bill and only need to act - if it gains traction. + For bills you oppose and only want to hear about if they gain real traction. Default + alerts: chamber votes and presidential action only — no noise from early committee or + document activity. - All the alerts of neutral mode, plus periodic reminders to contact your representative. - Use this when you actively support a bill and want to stay engaged. + For bills you actively support. Default alerts: everything — new text, amendments, + votes, presidential action, committee reports, calendar placement, procedural moves, + and committee referrals. Also adds “Find Your Rep” action buttons to push + notifications.
+ +
+

+ Adjusting alert filters +

+

+ The defaults above are starting points. In{" "} + Notifications → Alert Filters, + each mode has its own tab with eight independently toggleable alert types. For example, + a Follow bill where you don't care about committee reports — uncheck it and only + that mode is affected. Hit Load defaults on any tab to revert to the + preset above. +

+
+

- You can also follow members (get alerts when they sponsor new bills) and{" "} - topics (get alerts when new bills matching that topic are briefed). - Member and topic follows use the neutral mode only. + You can also follow members (alerts when they sponsor new bills) and{" "} + topics (alerts when new bills matching that topic are briefed). + Member and topic follows use the Follow mode filters.

diff --git a/frontend/app/notifications/page.tsx b/frontend/app/notifications/page.tsx index 229a913..0ef4774 100644 --- a/frontend/app/notifications/page.tsx +++ b/frontend/app/notifications/page.tsx @@ -49,7 +49,7 @@ const MODES = [ key: "neutral", label: "Follow", description: "Bills you follow in standard mode", - preset: { new_document: true, new_amendment: false, vote: true, presidential: true, committee_report: true, calendar: false, procedural: false, referral: false }, + preset: { new_document: true, new_amendment: true, vote: true, presidential: true, committee_report: true, calendar: false, procedural: false, referral: false }, }, { key: "pocket_veto",