{"id":13625898,"url":"https://github.com/fedspendingtransparency/usaspending-api","last_synced_at":"2026-01-17T18:36:04.714Z","repository":{"id":37493303,"uuid":"65394827","full_name":"fedspendingtransparency/usaspending-api","owner":"fedspendingtransparency","description":"Server application to serve U.S. federal spending data via a RESTful API","archived":false,"fork":false,"pushed_at":"2026-01-14T19:29:28.000Z","size":102114,"stargazers_count":389,"open_issues_count":76,"forks_count":149,"subscribers_count":49,"default_branch":"master","last_synced_at":"2026-01-14T21:38:50.844Z","etag":null,"topics":["api","api-blueprint","black","database","database-setup","django","django-rest-framework","docker","dredd","elasticsearch","etl","federal-spending-data","local-database","markdown","postgres-database","postgresql","pytest","python3","restful-api"],"latest_commit_sha":null,"homepage":"https://www.usaspending.gov","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fedspendingtransparency.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2016-08-10T15:39:45.000Z","updated_at":"2026-01-13T07:53:47.000Z","dependencies_parsed_at":"2023-10-04T20:45:30.627Z","dependency_job_id":"9fd9315b-bcd2-4348-8634-1fecdbf89899","html_url":"https://github.com/fedspendingtransparency/usaspending-api","commit_stats":null,"previous_names":[],"tags_count":93,"template":false,"template_full_name":null,"purl":"pkg:github/fedspendingtransparency/usaspending-api","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fedspendingtransparency%2Fusaspending-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fedspendingtransparency%2Fusaspending-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fedspendingtransparency%2Fusaspending-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fedspendingtransparency%2Fusaspending-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fedspendingtransparency","download_url":"https://codeload.github.com/fedspendingtransparency/usaspending-api/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fedspendingtransparency%2Fusaspending-api/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28516135,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T18:28:00.501Z","status":"ssl_error","status_checked_at":"2026-01-17T18:28:00.150Z","response_time":85,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","api-blueprint","black","database","database-setup","django","django-rest-framework","docker","dredd","elasticsearch","etl","federal-spending-data","local-database","markdown","postgres-database","postgresql","pytest","python3","restful-api"],"created_at":"2024-08-01T21:02:05.027Z","updated_at":"2026-01-17T18:36:04.701Z","avatar_url":"https://github.com/fedspendingtransparency.png","language":"Python","readme":"# \u003cp align=\"center\"\u003e\u003cimg src=\"https://www.usaspending.gov/img/logo@2x.png\" alt=\"USAspending API\"\u003e\u003c/p\u003e\n\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/python/black) [![Pull Request Checks](https://github.com/fedspendingtransparency/usaspending-api/actions/workflows/pull-request-checks.yaml/badge.svg?branch=staging)](https://github.com/fedspendingtransparency/usaspending-api/actions/workflows/pull-request-checks.yaml) [![Test Coverage](https://codeclimate.com/github/fedspendingtransparency/usaspending-api/badges/coverage.svg)](https://codeclimate.com/github/fedspendingtransparency/usaspending-api/coverage) [![Code Climate](https://codeclimate.com/github/fedspendingtransparency/usaspending-api/badges/gpa.svg)](https://codeclimate.com/github/fedspendingtransparency/usaspending-api)\n\n_This API is utilized by USAspending.gov to obtain all federal spending data which is open source and provided to the public as part of the DATA Act._\n\n![USAspending Landing Page](readme.jpg?raw=true \"Readme\")\n\n## Creating a Development Environment\n\nEnsure the following dependencies are installed and working prior to continuing:\n\n### Requirements\n- [`docker`](https://docs.docker.com/install/) which will handle the other application dependencies.\n- [`docker compose`](https://docs.docker.com/compose/)\n- `bash` or another Unix Shell equivalent\n    - Bash is available on Windows as [Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/install-win10)\n- [`git`](https://git-scm.com/downloads)\n- [`make`](https://www.gnu.org/software/make/) for running build/test/run targets in the `Makefile`. (Run `$ make` for a list of targets.)\n\n_**If not using Docker, you'll need to install app components on your machine:**_\n\u003e _Using Docker is recommended since it provides a clean environment. Setting up your own local environment requires some technical abilities and experience with modern software tools._\n\n- Command line package manager\n    - Windows' WSL bash uses `apt`\n    - MacOS users can use [`Homebrew`](https://brew.sh/)\n    - Linux users already know their package manager (`yum`, `apt`, `pacman`, etc.)\n- [`PostgreSQL`](https://www.postgresql.org/download/) version 13.x (with a dedicated `data_store_api` database)\n- [`Elasticsearch`](https://www.elastic.co/downloads/elasticsearch) version 7.1\n- `Python` version 3.10 environment\n  - Highly recommended to use a virtual environment. There are various tools and associated instructions depending on preferences\n  - See [Required Python Libraries](#required-python-libraries) for an example using `uv`\n- [`uv`](https://github.com/astral-sh/uv) python package/project manager\n\n### Cloning the Repository\nNow, navigate to the base file directory where you will store the USAspending repositories\n\n```shell\nmkdir -p usaspending \u0026\u0026 cd usaspending\ngit clone https://github.com/fedspendingtransparency/usaspending-api.git\ncd usaspending-api\n```\n### Environment Variables\n\nChoose an option between `.env` and `.envrc` that best fits your preferred workflow. Pay close attention to the values in these environment variables as usage of `localhost` vs a container's name differ between local setups.\n\n#### Create Your `.env` File (recommended)\nCopy the template `.env` file with local runtime environment variables defined. Change as needed for your environment. _This file is git-ignored and will not be committed by git if changed._\n\n```shell\ncp .env.template .env\n```\n\nA `.env` file is a common way to manage environment variables in a declarative file. Certain tools, like `docker compose`, will read and honor these variables.\n\n#### Create Your `.envrc` File\n_[direnv](https://direnv.net/) is a shell extension that automatically runs shell commands in a `.envrc` file (commonly env var `export` commands) when entering or exiting a folder with that file_\n\nCreate a `.envrc` file in the repo root, which will be ignored by git. Change credentials and ports as-needed for your local dev environment.\n\n```shell\nexport DATABASE_URL=postgres://usaspending:usaspender@localhost:5432/data_store_api\nexport ES_HOSTNAME=http://localhost:9200\nexport BROKER_DB=postgres://admin:root@localhost:5435/data_broker\n```\n\nIf `direnv` does not pick this up after saving the file, type\n\n```shell\ndirenv allow\n```\n_Alternatively, you could skip using `direnv` and just export these variables in your shell environment._\n\n**Just make sure your env vars declared in the shell and in `.env` match for a consistent experience inside and outside of Docker**\n\n### Build `usaspending-backend` Docker Image\n_This image is used as the basis for running application components and running containerized setup services._\n\n```shell\ndocker compose --profile usaspending build\n```\n\n_:bangbang: Re-run this command if any python package dependencies change (in `pyproject.toml`/`uv.lock`), since they are baked into the docker image at build-time._\n\n### Database Setup\nA postgres database is required to run the app. You can run it in a `postgres` docker container (preferred), or run a PostgreSQL server on your local machine. In either case, it will be empty until data is loaded.\n\n- :warning: If running your own PostgreSQL server be sure to:\n    1. Have a DB named `data_store_api`\n    2. A superuser role (user), e.g. `ALTER ROLE \u003c\u003crole/user you created\u003e\u003e WITH SUPERUSER;`\n    3. Cross-check your `.env` or `.envrc` files if used to be sure it references your DBs user, password, host, and port where needed\n\n##### Start the Postgres DB Container\n_If not using your own local install..._\n\n```shell\ndocker compose --profile usaspending up -d usaspending-db\n```\n... will create and run a Postgres database.\n\nUse the following commands to create necessary users and set the `usaspending` user's search_path\n\n```shell\ndocker exec -it usaspending-db sh -c \" \\\n    psql \\\n        -h localhost \\\n        -p 5432 \\\n        -U usaspending \\\n        -d data_store_api \\\n        -c 'CREATE USER etl_user;' \\\n        -c 'CREATE USER readonly;' \\\n        -c 'ALTER USER usaspending SET search_path TO public,raw,int,temp,rpt;' \\\n\"\n```\n\n##### Bring DB Schema Up-to-Date\n\n- To run [Django migrations](https://docs.djangoproject.com/en/2.2/topics/migrations/).\n    ```shell\n    docker compose run --rm usaspending-manage python3 -u manage.py migrate\n    ```\n- To provision the materialized views which are required by certain API endpoints.\n    ```shell\n    docker compose run --rm usaspending-manage python3 -u manage.py matview_runner --dependencies\n    ```\n\n##### Seeding and Loading Database Data\n_To just get essential reference data, you can run:_\n\n-  To load essential reference data (agencies, program activity codes, CFDA program data, country codes, and others).\n    ```shell\n    docker compose run --rm usaspending-manage python3 -u manage.py load_reference_data\n    ```\n\n_Alternatively, to download a fully populuated production snapshot of the database (full or a subset) and restore it into PostgreSQL, use the `pg_restore` tool as described here: [USAspending Database Download](https://onevoicecrm.my.site.com/usaspending/s/database-download)_\n\n- Recreate matviews with the command documented in the previous section if this is done\n\n_**Executing individual data-loaders** to load in data is also possible, but requires more familiarity with those ad-hoc scripts and commands, and also requires an external data source (Data Broker DB, or external file, etc.) from which to load the data._\n\n- For details on loading reference data, Data Accountability Broker Submissions, and current USAspending data into the API, see [loading_data.md](loading_data.md).\n- For details on how our data loaders modify incoming data, see [data_reformatting.md](data_reformatting.md).\n\n### Elasticsearch Setup\nSome API endpoints reach into Elasticsearch for data.\n\n```shell\ndocker compose --profile usaspending up -d usaspending-es\n```\n... will create and start a single-node Elasticsearch cluster as a docker container with data persisted to a docker volume.\n\n- The cluster should be reachable via at http://localhost:9200 (\"You Know, for Search\").\n\n- Optionally, to see log output, use `docker compose logs usaspending-es` (these logs are stored by docker even if you don't use this).\n\nWhile not required, it is highly recommended to also create the Kibana docker container for querying the Elasticsearch cluster.\n\n```shell\ndocker compose --profile usaspending up usaspending-kibana-es\n```\n\n#### Generate Elasticsearch Indexes\nThe following will generate the indexes:\n\n```shell\nCURR_DATE=$(date '+%Y-%m-%d-%H-%M-%S')\ndocker compose run --rm usaspending-manage python3 -u manage.py elasticsearch_indexer --create-new-index --index-name \"$CURR_DATE-transactions\" --load-type transaction\ndocker compose run --rm usaspending-manage python3 -u manage.py elasticsearch_indexer --create-new-index --index-name \"$CURR_DATE-awards\" --load-type award\ndocker compose run --rm usaspending-manage python3 -u manage.py elasticsearch_indexer --create-new-index --index-name \"$CURR_DATE-recipients\" --load-type recipient\ndocker compose run --rm usaspending-manage python3 -u manage.py elasticsearch_indexer --create-new-index --index-name \"$CURR_DATE-subaward\" --load-type subaward\n```\n\n### Observability/Tracing Setup\nThe project uses grafana, tempo, and opentelemetry for observability.  This enables developers to trace functionality across multiple services resulting in better visibility of issues.\nThere are two options for using the observability framework in the local development setup.  You can log traces to the console by setting the `TOGGLE_OTEL_CONSOLE_LOGGING` environmental variable to `True` for the `usaspending-api` container.\nAlternatively you can run the `otel-collector`, `tempo`, and `grafana` containers to more fully replicate the production observability framework.  These containers are part of the `tracing` profile and can be brought up with:\n```shell\ndocker compose --profile tracing up -d\n```\nWhen using these containers the `TOGGLE_OTEL_CONSOLE_LOGGING` must be set to `False` and `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` must be set to `http://otel-collector:4318/v1/traces`\n\n## Running the API\n\nRun the following to bring up the Django app for the RESTful API:\n\n```shell\ndocker compose --profile usaspending up usaspending-api\n```\n\nYou can update environment variables in `settings.py` (buckets, elasticsearch, local paths) and they will be mounted and used when you run this.\n\nThe application will now be available at `http://localhost:8000`.\n\n_**NOTE**: if the code was run outside of Docker then compiled Python files will potentially trip up the docker environment. A useful command to run for clearing out the files on your host is:_\n\n```shell\nfind . | grep -E \"(__pycache__|\\.pyc|\\.pyo$)\" | xargs rm -rf\n```\n\n#### Using the API\n\nIn your local development environment, available API endpoints may be found at `http://localhost:8000/docs/endpoints`.\n\nDeployed production API endpoints and docs are found by following links here: `https://api.usaspending.gov`.\n\n## Running Tests\n\n### Test Setup\n\n1. Build the base `usaspending-backend` Docker image (the test container is based on this Docker image). In the parent **usaspending-api** directory run:\n\n    ```shell\n    docker build -t usaspending-backend .\n    ```\n\n2. Start the Spark containers for the Spark related tests\n    ```shell\n    docker compose --profile spark up -d\n    ```\n\n3. To run all USAspending tests in the docker services run\n    ```shell\n    docker compose run --rm -e BROKER_DB='' usaspending-test\n    ```\n\n_**NOTE**: If an env var named `BROKER_DB` is set, Broker Integration tests will attempt to be run as well. If doing so, Broker dependencies must be met (see below) or ALL tests will fail hard. Running the above command with `-e BROKER_DB=''` is a precaution to keep them excluded, unless you really want them (see below if so)._\n\nTo run tests locally and not in the docker services, you need:\n\n1. **Postgres**: A running PostgreSQL database server _(See [Database Setup above](#database-setup))_\n1. **Elasticsearch**: A running Elasticsearch cluster _(See [Elasticsearch Setup above](#elasticsearch-setup))_\n1. **Environment Variables**: Tell python where to connect to the various data stores _(See [Environmnet Variables](#environment-variables))_\n1. **Install python version with uv**: python is installed with uv  _(See below)_\n1. **Sync uv project**: uv project is synced with lockfile  _(See below)_\n\n_**NOTE**: Running test locally might require you to run `export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES`. As discussed [here](https://github.com/rails/rails/issues/38560), there is an issue than can cause some of the Spark tests to fail without this environment variable set._\n\n#### Install python version with uv\nInstall the currently supported python version with uv\n\n```shell\nuv python install 3.10.12\n```\n\n#### Sync uv project with lockfile\nSetup uv project and check version info:\n\n```shell\nmake local-dev-setup\n```\n\n\nOnce these are satisfied, run:\n\n```shell\nmake tests\n```\n\nor, alternatively to skip using `make`\n\n```shell\nuv run pytest\n```\n\n### Including Broker Integration Tests\nSome automated integration tests run against a [Broker](https://github.com/fedspendingtransparency/data-act-broker-backend) database. If certain dependencies to run such integration tests are not satisfied, those tests will bail out and be marked as _Skipped_.\n(You can see messages about those skipped tests by adding the `-rs` flag to pytest, like: `pytest -rs`)\n\nTo satisfy these dependencies and include execution of these tests, do the following:\n\n1. Ensure the `Broker` source code is checked out alongside this repo at `../data-act-broker-backend`\n1. Ensure you have [`Docker`](https://docs.docker.com/install/) installed and running on your machine\n1. Ensure you have built the `Broker` backend Docker image by running:\n\n    ```shell\n    docker build -t dataact-broker-backend ../data-act-broker-backend\n    ```\n1. Ensure you have the `BROKER_DB` environment variable set, and it points to what will be a live PostgreSQL server (no database required) at the time tests are run.\n    1. _WARNING: If this is set at all, then ALL above dependencies must be met or ALL tests will fail (Django will try this connection on ALL tests' run)_\n    1. This DB could be one you always have running in a local Postgres instance, or one you spin up in a Docker container just before tests are run\n1. If invoking `pytest` within a docker container (e.g. using the `usaspending-test` container), you _must_ mount the host's docker socket. This is declared already in the `docker-compose.yml` file services, but would be done manually with: `-v /var/run/docker.sock:/var/run/docker.sock`\n\n_**NOTE**: Broker source code should be re-fetched and image rebuilt to ensure latest integration is tested_\n\nRe-running the test suite using `pytest -rs` with these dependencies satisfied should yield no more skips of the broker integration tests.\n\n**Example Test Invocations of _Just a Few_ Broker Integration Tests:** (_i.e. using `-k`_)\n\n_From within a container_\n\n_**NOTE**: `BROKER_DB` is set in the `docker-compose.yml` file (and could pick up `.env` values, if set)_\n\n```shell\ndocker compose run --rm usaspending-test pytest --capture=no --verbose --tb=auto --no-cov --log-cli-level=INFO -k test_broker_integration\n```\n\n_From Developer Desktop_\n\n_**NOTE**: `BROKER_DB` is set in the `.envrc` file and available in the shell_\n```shell\npytest --capture=no --verbose --tb=auto --no-cov --log-cli-level=INFO -k test_broker_integration\n```\n\n## Contributing\n\nTo submit fixes or enhancements, or to suggest changes, see [CONTRIBUTING.md](CONTRIBUTING.md).\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffedspendingtransparency%2Fusaspending-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffedspendingtransparency%2Fusaspending-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffedspendingtransparency%2Fusaspending-api/lists"}