{"id":25925244,"url":"https://github.com/ellisvalentiner/legislation-embeddings","last_synced_at":"2025-08-12T01:14:50.985Z","repository":{"id":277992720,"uuid":"928058098","full_name":"ellisvalentiner/legislation-embeddings","owner":"ellisvalentiner","description":"Embeddings for U.S. Congress legislation","archived":false,"fork":false,"pushed_at":"2025-06-25T17:36:04.000Z","size":1124,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-25T18:39:02.601Z","etag":null,"topics":["data","embeddings","machine-learning","nlp","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ellisvalentiner.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-06T01:42:10.000Z","updated_at":"2025-06-25T17:36:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"13d3b5af-a1cb-4114-8420-c9cf5a34d7c7","html_url":"https://github.com/ellisvalentiner/legislation-embeddings","commit_stats":null,"previous_names":["ellisvalentiner/legislation-embeddings"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ellisvalentiner/legislation-embeddings","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ellisvalentiner%2Flegislation-embeddings","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ellisvalentiner%2Flegislation-embeddings/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ellisvalentiner%2Flegislation-embeddings/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ellisvalentiner%2Flegislation-embeddings/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ellisvalentiner","download_url":"https://codeload.github.com/ellisvalentiner/legislation-embeddings/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ellisvalentiner%2Flegislation-embeddings/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269982632,"owners_count":24507431,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-11T02:00:10.019Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","embeddings","machine-learning","nlp","python"],"created_at":"2025-03-03T18:40:28.170Z","updated_at":"2025-08-12T01:14:50.978Z","avatar_url":"https://github.com/ellisvalentiner.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# U.S. Congress Legislation Embeddings\n\nThis repository contains code for generating embeddings for legislation from the U.S. Congress.\n\n***\n\nThis is an example project to demonstrate how to generate embeddings for documents, such as U.S. Congress legislation.\nThe document embeddings can be used for a variety of purposes, such as identifying similar legislation.\n\n### Legislation Data\n\nCongress has an official [bulk data API](https://www.govinfo.gov/bulkdata/) for accessing legislation data.\nThe data is organized by each Congress, their sessions, and includes bills, simple resolutions, joint resolutions, and concurrent resolutions.\n\n| Legislation Type             | Short Code |\n|:-----------------------------|:-----------|\n| House Concurrent Resolution  | `hconres`  |\n| House Joint Resolution       | `hjres`    |\n| House Bill                   | `hr`       |\n| House Resolution             | `hres`     |\n| Senate Bill                  | `s`        |\n| Senate Concurrent Resolution | `sconres`  |\n| Senate Joint Resolution      | `sjres`    |\n| Senate Rule                  | `sres`     |\n\nEach file is in XML format and consists of the legislation text and metadata.\n\n## Usage\n\n### Requirements\n\nThis project uses [`uv`](https://github.com/astral-sh/uv) to manage project dependencies.\n\n### Setup\n\n```bash\n# Clone the repo\ngit clone https://github.com/ellisvalentiner/legislation-embeddings.git\n# Change into the repo directory\ncd legislation-embeddings\n# Install dependencies\nuv sync\n```\n\n### Crawling\n\nThis project uses `scrapy` to crawl the bulk data API and extract the data into a format that can be used for training embeddings.\nThe crawler is configured to start at the beginning of each Congress, crawl through the nested directory structure, and extract all legislation.\nThe legislation data are XML files containing the legislation text and metadata.\n\nYou can run the crawler with:\n\n```bash\nuv run scrapy crawl legislation\n```\n\nThis will walk the hierarchy of the Congress legislation API and download all legislation data into a directory (`/data`).\nIt also creates a `download_state.json` file that tracks the state of the crawler doesn't download legislation that has already been downloaded.\nRe-running the crawler will skip legislation that has already been downloaded, although it will still crawl through the hierarchy.\n\n### Embedding\n\nThe legislation data is then processed to generate embeddings using the [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) model.\nThis is a relatively small model, so it's possible to run this code locally on a laptop.\nThe model is a sentence transformer model that can embed the entire text of a legislation document to a 384-dimensional vector.\n\nThe embeddings are stored in a [Chroma](https://www.trychroma.com/) vector database (in the `/embeddings` directory).\n\nYou can generate and store embeddings for legislation using the following code:\n\n```bash\nuv run process\n```\n\nBy default, only 10,000 random legislation files are processed.\nYou can change the number of files processed by setting the `limit` parameter in the `config.py` file or by passing the `--limit` argument to the script.\n\n```bash\n# Reduce the number of files processed to 100\nuv run process --limit 100\n```\n\n:notebook: Note any of the `Config` parameters can be changed by their command line arguments.\n\n### Search\n\nOnce the data is processed and embeddings are stored in the vector database, the embeddings can be searched.\n\nFor example, to search the vector database for legislation related to the Judiciary, you can run the following code:\n\n```bash\nuv run search --query \"Judiciary\"\n# 115 S296 IS: Federal Courts of Appeals Modernization Act\n# 118 S5229 IS: Judicial Modernization and Transparency Act\n# 117 HR 6577 IH: Real Courts, Rule of Law Act of 2022\n# 118 S833 IS: Sunshine in the Courtroom Act of 2023\n# 115 HRES 105 IH: Expressing the Sense of the House of Representatives that an Independent Judiciary is Fundamental to American Democracy.\n```\n\nNote the results include legislation without \"Judiciary\" in the title but likely related to the Judiciary based on the context.\n\n### Semi-supervised Learning\n\nThe embeddings can be used for semi-supervised learning by labeling the embeddings with topic tags.\n\nBy querying the legislation embeddings, you can retrieve a list of the most similar legislation to a given topic.\nThe matching legislation can be used as examples for training a supervised model, such as a classifier, to classify new legislation files.\n\n\nFor example, to label the embeddings with topic tags, you can run the following code:\n\n```bash\nuv run label\n```\n\nThis will label 10 legislation files per topic with tags and save the labeled embeddings to a CSV file (`labeled_embeddings.csv`).\nBy default, the topics are: agriculture, economy, education, energy, environment, health, housing, immigration, infrastructure, national security, social security, transportation, veterans.\n\nYou can generate model provided labels for your own topics by providing a comma-separated list to the `--topics` flag:\n\n```bash\nuv run label --topics=speech,gun,car\n```\n\n:notebook: Note it is possible for a topic to not have any matching legislation.\n\nIn practice, legislation likely should be labeled with multiple topics, and the labels may not be mutually exclusive.\nFor example, a legislation may be labeled with both \"economy\" and \"health\" topics.\nHowever, for the purpose of this project, we will only use one topic per legislation.\n\n### Dimensionality Reduction and Visualization\n\nVisualizing the legislation embeddings can be useful for understanding the data.\nSince the embeddings are 384-dimensional vectors, we need to reduce them to a lower dimensional space for visualization.\n\nThe embeddings can be reduced to two dimensions for visualization using [Uniform Manifold Approximation and Projection (UMAP)](https://umap-learn.readthedocs.io/en/latest/).\n\nTo reduce the embeddings to 2D space, run the following command:\n\n```bash\nuv run reduce\n```\n\nThis will generate a CSV file (`reduced_embeddings.csv`) with the embeddings reduced to a 2D space.\n\nThe embeddings can be visualized using Plotly.\n\n```bash\nuv run visualize\n```\n\nThis will generate a PNG file (`fig-topic.png`) with each point colored by the topic label.\n\n![](out/fig.png)\n\nSince we only labeled a sample of legislation with topics, most points are colored gray.\nThe legislation are organized into a discernible structure, rather than being randomly scattered.\nThis suggests that the embeddings are capturing context or properties of the legislation.\nThere are several outlier points and clusters that are far from the rest of the data.\n\nThe labeled legislation are colored by topic.\nWe can see, for most topics, legislation with the same label tend to be close to each other in the 2D space.\nHowever, some the legislation for some topics are more spread out.\nThis may be due to the fact that some topics are governed by a wider range of legislation or that some topic keywords may have multiple meanings.\nFor example, the topic \"Environment\" may refer to environmental issues, the environmental impact of legislation, or simply the \"political environment\" whereas \"Agriculture\" has a more singular meaning.\n\n### Outliers\n\nWe can look for outliers – legislation that is not closely located to other legislation.\nThese are legislation that are not near other legislation in the lower dimensional space.\n\nThe outliers can be found using the following code:\n\n```bash\nuv run outlier\n# 117 S217 IS: Patient Matching Improvement Act of 2021\n# 115 HR 2872 IH: Promoting Hydropower Development at Existing Nonpowered Dams Act\n# 116 HR 6020 EH: To require an evaluation by the Government Accountability Office of the social, economic, and historic contributions that Minor League Baseball has made to American life and culture.\n# 114 HR 1033 IH: John P. Parker House Study Act\n# 115 HR 220 ENR: To authorize the expansion of an existing hydroelectric project, and for other purposes.\n```\n\n## Wrap-up\n\nThis project demonstrates how to generate embeddings for legislation using the U.S. Congress API.\nThe embeddings can be used for a variety of purposes, such as identifying similar legislation.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fellisvalentiner%2Flegislation-embeddings","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fellisvalentiner%2Flegislation-embeddings","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fellisvalentiner%2Flegislation-embeddings/lists"}