{"id":35017299,"url":"https://github.com/navapbc/grants-ingest","last_synced_at":"2026-05-19T13:11:48.345Z","repository":{"id":281704967,"uuid":"934910809","full_name":"navapbc/grants-ingest","owner":"navapbc","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-14T06:44:59.000Z","size":2889,"stargazers_count":1,"open_issues_count":15,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-05-14T08:35:12.840Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/navapbc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-02-18T15:49:38.000Z","updated_at":"2026-05-14T06:45:11.000Z","dependencies_parsed_at":"2025-04-22T08:22:13.434Z","dependency_job_id":"b7b6462b-2c6b-431c-ac02-f147e8d38130","html_url":"https://github.com/navapbc/grants-ingest","commit_stats":null,"previous_names":["navapbc/grants-ingest"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/navapbc/grants-ingest","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/navapbc%2Fgrants-ingest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/navapbc%2Fgrants-ingest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/navapbc%2Fgrants-ingest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/navapbc%2Fgrants-ingest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/navapbc","download_url":"https://codeload.github.com/navapbc/grants-ingest/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/navapbc%2Fgrants-ingest/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33217969,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-19T07:54:09.561Z","status":"ssl_error","status_checked_at":"2026-05-19T07:54:08.508Z","response_time":58,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-27T05:26:24.859Z","updated_at":"2026-05-19T13:11:48.340Z","avatar_url":"https://github.com/navapbc.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Grants Ingest Service\n\nIngests and indexes data related to grants.\n\n\n## What's this?\n\nThis repository contains both IaC (terraform) and runtime code for the Grants Ingest pipeline\nservice. The purpose of this service is to collect information about grant opportunities from\nthird party sources, such as [Grants.gov](https://grants.gov), and organize that data into\nper-grant data records that can be consumed whenever updates to the underlying data occur.\n\n\n## Architecture\n\nThis service consists of an event-driven pipeline that uses AWS Lambda for running various\ncompute tasks. Additionally, Amazon S3 is used for archival storage of raw source data, while\nDynamoDB provides an active index of processed grant opportunity records. At the end of the ingestion\npipeline, newly-created/-updated grant opportunity records are sent to an event bus that delivers\nthe updates to subscribers.\n\n\u003cdetails\u003e\n  \u003csummary\u003eHigh-level architecture\u003c/summary\u003e\n  \u003cimg src=\"docs/high-level-architecture.png\"/\u003e\n\u003c/details\u003e\n\u003cdetails\u003e\n  \u003csummary\u003eComponent-level architecture\u003c/summary\u003e\n  \u003cimg src=\"docs/component-level-architecture.png\"/\u003e\n\u003c/details\u003e\n\n\n## Code Organization\n\nCode for this service can generally be considered under two categories:\n\n\n### IaC\n\nInfrastructure-as-code (IaC) used to provision the target environment, which is written\nwith Terraform, and normally run during deployment. The main Terraform project is located in\nthis directory; infrastructure specific to a particular step in the pipeline is organized into\nTerraform modules located within the `modules` subdirectory.\n\n\n### Runtime\n\nRuntime code that executes (e.g. by AWS Lambda) within the target environment in response to\nsome triggering event. Runtime code is written using Go (currently targeting version `1.23.x`),\nwhich is organized in the repository root directory according to the following conventions:\n\n- `cmd/`: This directory contains one subdirectory per Lambda function, and should provide a single\n`main` package that can be compiled into a per-function binary. Each subdirectory of `cmd`\nshould have a name that obviously aligns with the particular Lambda function for which it is\nwritten (which generally should correspond to the Terraform module directory used to provision\nthe Lambda function and its dependencies). For example:\n  - Lambda handler code: `cmd/DownloadGrantsGovDB`\n  - Terraform module: `modules/DownloadGrantsGovDB`\n- `pkg/`: This directory contains \"library code\" used by one or more Lambda functions in the project,\norganized into per-package subdirectories according to Go convention.\n- `internal/`: This directory is similar to `pkg/` but contains packages that are only intended\nto ever be used internally by this project and make no guarantees about third-party compatibility.\nGenerally, reusable libraries providing common functionality used by multiple Lambda functions\nshould reside here.\n\n\n## Development\n\nDuring development, infrastructure and runtime code can be tested by configuring Terraform\nto target an actual (\"sandbox\", dedicated for development) AWS environment or a mock AWS\nenvironment simulated with [LocalStack](https://localstack.cloud/) if an AWS sandbox environment\nis unavailable.\n\n\n### Configuring Terraform and LocalStack\n\n\n#### Prerequisites\n\nTo begin, make sure the following tools are available in your development workspace:\n- [Go](https://go.dev/)\n  - Ensure you're using the version found in [go.mod](https://github.com/usdigitalresponse/grants-ingest/blob/main/go.mod)\n  - If you need to manage multiple Go versions on your machine, you can follow [official guidance](https://go.dev/doc/manage-install)\n    or use a tool like [asdf](https://github.com/asdf-vm/asdf), [gvm](https://github.com/moovweb/gvm), or [goenv](https://github.com/go-nv/goenv)\n- [AWS CLI v2](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)\n- [Terraform](https://developer.hashicorp.com/terraform/downloads)\n  - Download the version found in [.terraform-version](https://github.com/usdigitalresponse/grants-ingest/blob/main/terraform/.terraform-version)\n- [LocalStack](https://docs.localstack.cloud/getting-started/installation)\n- [tflocal](https://github.com/localstack/terraform-local)\n- [awslocal](https://github.com/localstack/awscli-local)\n- [tfenv](https://github.com/tfutils/tfenv) *Optional, but recommended.*\n\n**Note:** This document assumes usage of [Docker-Compose method](https://docs.localstack.cloud/getting-started/installation/#docker-compose).\nHowever, you can choose whatever LocalStack installation option best suits your environment.\n\n\n\u003e [!TIP]\n\u003e Check that you have the necessary dependencies for LocalStack development by running `task local:check-dependencies`\n\n\n#### Environment Variables\n\nWe recommend setting the following environment variables in your development workspace when working\nwith LocalStack:\n\n- `AWS_SDK_LOAD_CONFIG=true`\n- `AWS_REGION=us-west-2`\n- `AWS_DEFAULT_REGION=us-west-2`\n- `AWS_ACCESS_KEY_ID=testing`\n- `AWS_SECRET_ACCESS_KEY=testing`\n- `LOCALSTACK_VOLUME_DIR`\n    - Should be set to whatever path you want to persist LocalStack data.\n    If not set, the default behavior is to target a `./volume` subdirectory\n    from where `docker compose` is run.\n\n\n#### Provisioning Infrastructure\n\n#### Quickstart\n\nCertain steps described in the \"Manual Provisioning\" section below may be achieved using the following\nshortcut commands provided by this repository's Taskfile:\n- **Prerequisite:** Ensure LocalStack is started by running `docker compose up -d`\n- **Deploy to LocalStack for the first time:** `task local:from-scratch`\n- **Subsequent deployments to LocalStack:** `task local:deploy`\n- **Invoke the `DownloadGrantsGovDB` Lambda function:** `task local:invoke-DownloadGrantsGovDB`\n\nSee `.taskfile/local.yml` for more information and other shortcuts.\n\n\n#### Manual Provisioning\n\nAfter starting LocalStack, create a terraform state deployment bucket in the localstack environment.  This guide, as well as the provided Terraform backend file for local development, assumes that\nthis bucket is named `local-terraform` and has a region of `us-west-2`.\n\nTo create this bucket, navigate to the terraform folder, then run the following command:\n```bash\nawslocal s3 mb s3://local-terraform\n```\nIf using the provided [docker-compose](https://github.com/usdigitalresponse/grants-ingest/blob/cf8a6220ed4e7332978520ec1d63bb344b738972/docker-compose.yml#L21)\nthen this runs when the localstack container is started.\n\nNext, initialize the Terraform project using the `local.s3.tfbackend` Terraform state backend\nconfiguration file provided by this repository. Note that you may need to modify the `endpoint`\nsetting in this file if your LocalStack environment uses a different host/port other than the\ndefault of `localhost:4566`.\n\n```bash\ntflocal init -backend-config=\"local.s3.tfbackend\" -reconfigure\n```\n\nOnce this command completes successfully, use `tflocal` to \"provision\" mock infrastructure in your LocalStack environment.  This will create the Lambda functions, so it requires the functions to be\nbuilt first.  Use `task build` to build the functions.  Then use the `local.tfvars` file provided by this\nrepository to provision the mock infrastructure:\n```bash\ntflocal apply -var-file=local.tfvars\n```\n\n**Hint:** It's normal for `task build` and/or `tflocal apply` to run for up to a few minutes.\nIf you notice that these commands are taking an excessively long time or seem to hang, try\ncancelling the operation, then run `task prebuild-lambda` and try running your command again.\n\n\n#### Running Lambda Functions\n\nOnce your Lambda function has been deployed to your running LocalStack environment, you can\ninvoke it in order to test its execution and observe log output. Since each Lambda function\nbehaves differently and expects different inputs, the exact nature of the debugging cycle\nwill vary between functions. However, the following example demonstrating invocation of the\nfunction defined in `module.DownloadGrantsGovDB` may be used as a starting point.\n\nIn general, Lambda functions are invoked with a JSON payload that represents the `event`\ninput expected by the Lambda handler. Again, the structure of this payload varies depending\non the invocation source(s) configured for each Lambda function. In the case of the Lambda\nfunction defined in `module.DownloadGrantsGovDB`, the expected payload is a JSON object\ncontaining a `\"timestamp\"` key whose value is an ISO-8601 timestamp string, from which\nthe date of a desired Grants.gov database export is derived. For example:\n\n```json\n{\"timestamp\": \"2023-03-20T05:00:00-04:00\"}\n```\n\nWhen using `awslocal` (or the official AWS CLI, for that matter), also note that the payload\nmust be provided as a Base64-encoded encoded representation of the payload. The following example\ndemonstrates an invocation of a Lambda function named `grants-ingest-DownloadGrantsGovDB`\nthat provides the above example JSON in Base64 encoding and prints the Lambda output to stdout:\n\n```bash\nawslocal lambda invoke \\\n    --function-name grants-ingest-DownloadGrantsGovDB \\\n    --payload $(printf '{\"timestamp\": \"2023-03-20T05:00:00-04:00\"}' | base64) \\\n    /dev/stdout\nnull\n{\n    \"StatusCode\": 200,\n    \"LogResult\": \"\",\n    \"ExecutedVersion\": \"$LATEST\"\n}\n```\n\nThe above output displays the Lambda function invocation result following its successful execution;\nthe output you see will vary depending on the input, the function version that was invoked,\nwhether an unhandled error cause execution to fail, etc.\n\nIn order to view execution logs outputted during Lambda invocation, use CloudWatch Logs (just as\nyou would when conducting tests against a genuine AWS environment). The following command can\nbe used to observe log output in real-time:\n\n```bash\nawslocal logs tail /aws/lambda/grants-ingest-DownloadGrantsGovDB --follow\n```\n\nWhen actively debugging, it is useful to run a command similar to the above in a separate terminal\nprior to invoking the Lambda function under test, as it only displays logs emitted after the\n`awslocal logs tail` started. You can use the `--since` option (with or without `--follow`)\nin order to display historical logs from previous invocations. For example, the following command\nshows logs emitted in the past 1 hour, and will continue to display new logs as they are emitted:\n\n```bash\nawslocal logs tail /aws/lambda/grants-ingest-DownloadGrantsGovDB --since 1h --follow\n```\n\n\n### Running Common Tasks\n\nThis repository provides a `Taskfile.yml` file for defining and running common tasks related\nto development. You can [install the `task` utility](https://taskfile.dev/installation/)\nand then run `task` in your command-line environment to see a list of the available helpers.\n\n\n### Go compiler architecture\n\nThis project is configured to target ARM64 CPU architecture when building Go binaries,\nsince that is what we run on staging and production.\nSome contributors who develop on 64-bit Intel machines may have issues running Lambdas on LocalStack with this configuration.\nCurrently, the suggested workaround is to make the following modifications to target 64-bit Intel when building locally and deploying to LocalStack:\n\n- In `Taskfile.yml`, modify the `go-build-lambda` task's command to set `GOARCH=amd64` (replaces `GOARCH=arm64`).\n- In `terraform/local.tfvars`, add the following Terraform input variable: `lambda_arch = \"x86_64\"`.\n\n\n#### Quick Reference\n\nThe following items can be referred to as a quick \"cheat-sheet\" for development:\n- Install Go dependencies: `go mod download` or `go get ./...`\n- Run Go unit tests: `task test`\n- Ensure all Go and Terraform code is formatted properly: `task fmt`\n- Warm up your Go cache: `task prebuild-lambda`\n  - We recommend running this command periodically, especially before compiling Go binaries,\n  running `tflocal plan`, and/or `tflocal apply`.\n- Compile binary Lambda function handlers: `task build`\n- Compile the CLI tool: `task build-cli`\n- Run all QA checks normally executed during CI: `task check`\n- Initialize and deploy a test environment after starting LocalStack: `task local:from-scratch`\n\n\n## Contributing\n\nThis project wouldn’t exist without the hard work of\n[many people](https://github.com/usdigitalresponse/grants-ingest/graphs/contributors)! Please\nsee [CONTRIBUTING.md](https://github.com/usdigitalresponse/grants-ingest/blob/main/CONTRIBUTING.md)\nto find out how you can help.\n\n\n## Releasing to Production\n\n**Note:** Releases are versioned using a `YYYY.inc` scheme that represents the year of the\nrelease, and the incremental release number for that year. You can view a list of all historical\nreleases on the [Releases page](https://github.com/usdigitalresponse/grants-ingest/releases).\n\nMaintainers with the requisite access may release to Production by performing the following\nsteps:\n1. Navigate to the [list of Releases](https://github.com/usdigitalresponse/grants-ingest/releases)\nfor this repository.\n2. Locate the draft for the next release, and click the pencil icon to edit.\n3. Provide a high-level summary of the release in the **Summary** section of the release notes.\n4. Optionally, make any necessary edits to the other sections of the prepared release notes.\n5. Ensure \"Set as a pre-release\" is checked at the bottom of the edit page.\n6. Click the \"Publish Release\" button.\n\nAt this point, the release will be published (as a pre-release) and the\n[deployment pipeline](https://github.com/usdigitalresponse/grants-ingest/actions/workflows/deploy-production.yml)\nwill automatically begin preparing the changes that will be rolled out to Production.\n\nOnce a Terraform plan has been created for the release, repository administrators will be notified\nfor review and final approval. After the plan has been approved and applied to Production,\nthe release will be automatically updated to remove the pre-release state, and a timestamp\nfor the deployment will be appended to the release notes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnavapbc%2Fgrants-ingest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnavapbc%2Fgrants-ingest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnavapbc%2Fgrants-ingest/lists"}