{"id":31271763,"url":"https://github.com/evalops/congress-bill-search","last_synced_at":"2025-10-07T10:00:08.944Z","repository":{"id":316291527,"uuid":"1054609965","full_name":"evalops/congress-bill-search","owner":"evalops","description":"High-quality congressional bill search with hybrid BM25+vector similarity using DuckDB, TEI embeddings, and GovInfo API. Local deployment with Docker.","archived":false,"fork":false,"pushed_at":"2025-09-11T05:01:25.000Z","size":16,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-23T20:12:51.776Z","etag":null,"topics":["bm25","congressional-bills","docker","duckdb","embeddings","govinfo-api","hybrid-search","reranking","search-engine","tei","text-search","vector-search"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/evalops.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-11T05:01:08.000Z","updated_at":"2025-09-11T09:23:14.000Z","dependencies_parsed_at":"2025-09-23T20:23:00.085Z","dependency_job_id":null,"html_url":"https://github.com/evalops/congress-bill-search","commit_stats":null,"previous_names":["evalops/congress-bill-search"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/evalops/congress-bill-search","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evalops%2Fcongress-bill-search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evalops%2Fcongress-bill-search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evalops%2Fcongress-bill-search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evalops%2Fcongress-bill-search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/evalops","download_url":"https://codeload.github.com/evalops/congress-bill-search/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evalops%2Fcongress-bill-search/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278755165,"owners_count":26040034,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-07T02:00:06.786Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bm25","congressional-bills","docker","duckdb","embeddings","govinfo-api","hybrid-search","reranking","search-engine","tei","text-search","vector-search"],"created_at":"2025-09-23T20:02:09.998Z","updated_at":"2025-10-07T10:00:08.924Z","avatar_url":"https://github.com/evalops.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Congressional Bill Search System\n\nA high-quality bill text and metadata search system using:\n- **GovInfo API** for authoritative bill text (BILLS) and metadata (BILLSTATUS)\n- **DuckDB** for storage with BM25 FTS and HNSW vector search  \n- **Local embeddings** via TEI + nomic-embed-text-v1.5 (256d Matryoshka)\n- **Hybrid search** (BM25 prefilter → vector rerank → optional cross-rerank)\n\n## Quick Start\n\n1. **Get GovInfo API Key**\n   ```bash\n   # Register at https://api.govinfo.gov/docs/\n   export GOVINFO_API_KEY=your_key_here\n   ```\n\n2. **Install Dependencies**\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n3. **Start TEI Services**\n   ```bash\n   docker compose up -d\n   \n   # Wait for models to download and services to start\n   # Check with: curl http://localhost:8080/health\n   ```\n\n4. **Initialize Database**\n   ```bash\n   duckdb congress_119.duckdb -c \".read bootstrap_duckdb.sql\"\n   ```\n\n5. **Ingest Bills (119th Congress sample)**\n   ```bash\n   export CONGRESS=119 LIMIT=100\n   python ingest_govinfo.py\n   ```\n\n6. **Generate Embeddings**\n   ```bash\n   python embed_and_index.py\n   ```\n\n7. **Search!**\n   ```bash\n   python search.py \"definition of covered AI system\"\n   python search.py \"climate change mitigation\" --rerank --limit 5\n   ```\n\n## System Architecture\n\n```\nGovInfo API (BILLS + BILLSTATUS)\n       ↓\n   DuckDB Storage\n   ├─ BM25 FTS (fragments.text)\n   └─ HNSW VSS (fragments.embedding)\n       ↓\n   Hybrid Search Pipeline:\n   1. BM25 prefilter (500 candidates)\n   2. Vector similarity (cosine distance)  \n   3. Optional cross-encoder rerank (Top-25)\n```\n\n## Configuration\n\n### Environment Variables\n\n- `GOVINFO_API_KEY` - Required GovInfo API key\n- `CONGRESS` - Congress number to ingest (default: 119)\n- `CONGRESS_DB` - Database path (default: congress_119.duckdb)  \n- `LIMIT` - Limit ingestion count (default: 0 = no limit)\n- `TEI_URL` / `TEI_EMBED_URL` - TEI embeddings endpoint (default: http://localhost:8080)\n- `TEI_RERANK_URL` - TEI reranking endpoint (default: http://localhost:8081)\n- `EMBED_DIM` - Embedding dimensions (default: 256, supports 64-768)\n\n### TEI Services\n\nThe `docker-compose.yml` runs two TEI instances:\n\n- **Port 8080**: `nomic-ai/nomic-embed-text-v1.5` for embeddings\n- **Port 8081**: `BAAI/bge-reranker-v2-m3` for reranking\n\nFor GPU support, uncomment the `deploy` sections in docker-compose.yml.\n\n## Usage Examples\n\n### Basic Search\n```bash\npython search.py \"healthcare reform\"\n```\n\n### Filtered Search  \n```bash\npython search.py \"artificial intelligence\" --congress 119 --bill-type hr --limit 5\n```\n\n### With Reranking\n```bash\npython search.py \"climate change adaptation\" --rerank\n```\n\n### Programmatic Usage\n```python\nfrom search import BillSearchEngine\n\nwith BillSearchEngine(\"congress_119.duckdb\", \n                     \"http://localhost:8080\", \n                     \"http://localhost:8081\") as engine:\n    results = engine.hybrid_search(\n        \"definition of covered AI system\",\n        limit=10,\n        use_rerank=True\n    )\n    for result in results:\n        print(f\"{result['bill_id']}: {result['title']}\")\n```\n\n## Data Coverage\n\n- **Bills**: XML text from 113th Congress → current (2013+)\n- **Metadata**: BILLSTATUS with actions, subjects, policy areas\n- **Updates**: Current congress updated every ~4 hours via GovInfo\n\n## Performance \u0026 Sizing\n\n- **256-dim embeddings**: ~1KB per fragment  \n- **3M fragments**: ~3GB for vectors + HNSW overhead\n- **Search latency**: BM25 + vector ~100-500ms, +rerank ~1-2s\n- **Memory**: HNSW index must fit in RAM when loaded\n\n## Sharding Strategy\n\nUse one database per Congress for manageable sizing:\n- `congress_113.duckdb`, `congress_114.duckdb`, etc.  \n- Query multiple databases for cross-congress search\n- HNSW index rebuilds are fast per-congress\n\n## Extensions \u0026 Improvements\n\n1. **Subject search**: Index `subjects` array as FTS side table\n2. **Deduplication**: Content hash across versions (IH→RH→ENR)  \n3. **USLM integration**: Add enrolled bills/laws corpus\n4. **Caching**: Redis for frequent queries\n5. **API**: FastAPI wrapper for REST endpoints\n\n## Troubleshooting\n\n### TEI Services Not Starting\n```bash\n# Check logs\ndocker compose logs tei-embed\ndocker compose logs tei-rerank\n\n# Verify health  \ncurl http://localhost:8080/health\ncurl http://localhost:8081/health\n```\n\n### HNSW Index Issues\n```bash\n# Experimental persistence flag required\n# If index corrupts, rebuild:\nduckdb congress_119.duckdb -c \"DROP INDEX IF EXISTS frags_hnsw_cos\"\npython embed_and_index.py  # Rebuilds index\n```\n\n### Memory Issues\n```bash\n# Reduce embedding dimensions\nexport EMBED_DIM=128  # vs default 256\n\n# Or reduce prefilter\npython search.py \"query\" --prefilter-limit 200  # vs default 500\n```\n\n## Development\n\n### Run Tests\n```bash\npytest tests/\n```\n\n### Format Code  \n```bash\nblack *.py\n```\n\n### Add New Congress\n```bash\nexport CONGRESS=120 CONGRESS_DB=congress_120.duckdb\nduckdb congress_120.duckdb -c \".read bootstrap_duckdb.sql\"\npython ingest_govinfo.py\npython embed_and_index.py\n```\n\n## License\n\nMIT License - see LICENSE file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevalops%2Fcongress-bill-search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fevalops%2Fcongress-bill-search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevalops%2Fcongress-bill-search/lists"}