Add bill action pipeline, admin health panel, and LLM provider fixes

- Fetch bill actions from Congress.gov and populate the action timeline
- Add nightly batch task and beat schedule for active bill actions
- Add admin reprocess endpoint for per-bill debugging
- Add BriefPanel with "What Changed" view and version history
- Add External API Health section with per-source latency testing
- Redesign Manual Controls as health panel with status dots and descriptions
- Add Resume Analysis task for stalled LLM jobs
- Add Backfill Dates & Links task for bills with null metadata
- Fix LLM provider/model DB overrides being ignored (env vars used instead)
- Fix Gemini 404: gemini-1.5-pro deprecated → gemini-2.0-flash
- Fix Anthropic models list: use REST API directly (SDK too old for .models)
- Replace test-LLM full analysis with lightweight ping (max_tokens=20)
- Add has_document field to BillDetail; show "No bill text published" state
- Fix "Introduced: —" showing for bills with null introduced_date
- Add bills_missing_sponsor and bills_missing_metadata to admin stats
- Add GovInfo health check using /collections endpoint (fixes 500 from /packages)

Authored-By: Jack Levy
This commit is contained in:
Jack Levy
2026-03-01 11:06:14 -05:00
parent defc2c116d
commit 5eebc2f196
10 changed files with 586 additions and 74 deletions

View File

@@ -199,3 +199,55 @@ def backfill_brief_citations(self):
return {"total": total, "queued": queued, "skipped": skipped}
finally:
db.close()
@celery_app.task(bind=True, name="app.workers.llm_processor.resume_pending_analysis")
def resume_pending_analysis(self):
"""
Two-pass backfill for bills missing analysis:
Pass 1 — Documents with no brief (LLM tasks failed/timed out):
Find BillDocuments that have raw_text but no BillBrief, re-queue LLM.
Pass 2 — Bills with no document at all:
Find Bills with no BillDocument, re-queue document fetch (which will
then chain into LLM if text is available on GovInfo).
"""
db = get_sync_db()
try:
# Pass 1: docs with raw_text but no brief
docs_no_brief = db.execute(text("""
SELECT bd.id
FROM bill_documents bd
LEFT JOIN bill_briefs bb ON bb.document_id = bd.id
WHERE bb.id IS NULL AND bd.raw_text IS NOT NULL
""")).fetchall()
queued_llm = 0
for row in docs_no_brief:
process_document_with_llm.delay(row.id)
queued_llm += 1
time.sleep(0.1)
# Pass 2: bills with no document at all
bills_no_doc = db.execute(text("""
SELECT b.bill_id
FROM bills b
LEFT JOIN bill_documents bd ON bd.bill_id = b.bill_id
WHERE bd.id IS NULL
""")).fetchall()
queued_fetch = 0
from app.workers.document_fetcher import fetch_bill_documents
for row in bills_no_doc:
fetch_bill_documents.delay(row.bill_id)
queued_fetch += 1
time.sleep(0.1)
logger.info(
f"resume_pending_analysis: {queued_llm} LLM tasks queued, "
f"{queued_fetch} document fetch tasks queued"
)
return {"queued_llm": queued_llm, "queued_fetch": queued_fetch}
finally:
db.close()