{"id":22538086,"url":"https://github.com/cfpb/hmda-platform","last_synced_at":"2025-05-16T06:02:25.043Z","repository":{"id":38859458,"uuid":"50141662","full_name":"cfpb/hmda-platform","owner":"cfpb","description":"The HMDA Submission backend applications.","archived":false,"fork":false,"pushed_at":"2025-04-30T19:37:49.000Z","size":117308,"stargazers_count":106,"open_issues_count":62,"forks_count":98,"subscribers_count":32,"default_branch":"master","last_synced_at":"2025-04-30T20:42:49.542Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cfpb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-01-21T22:39:17.000Z","updated_at":"2025-04-25T14:30:32.000Z","dependencies_parsed_at":"2023-11-16T05:28:56.460Z","dependency_job_id":"8522393e-228a-4148-9f73-6ad9621985dc","html_url":"https://github.com/cfpb/hmda-platform","commit_stats":{"total_commits":7624,"total_committers":56,"mean_commits":"136.14285714285714","dds":0.748688352570829,"last_synced_commit":"9a9fb92fcd17e99708d59b17586d5cea10f9c938"},"previous_names":[],"tags_count":161,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cfpb%2Fhmda-platform","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cfpb%2Fhmda-platform/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cfpb%2Fhmda-platform/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cfpb%2Fhmda-platform/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cfpb","download_url":"https://codeload.github.com/cfpb/hmda-platform/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254478160,"owners_count":22077675,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-07T11:10:07.518Z","updated_at":"2025-05-16T06:02:20.034Z","avatar_url":"https://github.com/cfpb.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HMDA Platform\n![Static Badge](https://img.shields.io/badge/coverage-91%25-green)\n\n## Introduction\n\nThe [Home Mortgage Disclosure Act (HMDA) Platform](http://ffiec.cfpb.gov/) is a [Regulatory technology](https://en.wikipedia.org/wiki/Regulatory_technology) application for financial institutions to submit mortgage information as described in the [Filing Instruction Guide (FIG)](https://s3.amazonaws.com/cfpb-hmda-public/prod/help/2020-hmda-fig.pdf). The HMDA-Platform parses data as submitted by mortgage leading institutions and validates the information for edits (Syntactical, Validity, Quality, and Macro as-per the instructions in the FIG) before submitting the data. The HMDA-Platform supports [quarterly](https://ffiec.cfpb.gov/documentation/2020/quarterly-filing-dates/) and [yearly](https://ffiec.cfpb.gov/documentation/2019/annual-filing-dates/) filing periods. For detailed information on Home Mortgage Disclosure Act (HMDA), checkout the [About HMDA page](https://www.consumerfinance.gov/policy-compliance/rulemaking/final-rules/regulation-c-home-mortgage-disclosure-act/) on the CFPB website.\n\n[Please watch this short video](https://youtu.be/C_73Swgyc4g) to view how HMDA Platform transforms the data upload, validation, and submission process.\n\n## Linked Projects\n\n|  Project | Repo Link  | Description  | \n|---|---|---|\n| Frontend  |  https://github.com/cfpb/hmda-frontend | ReactJS Front-end repository powering the [HMDA Platform](http://ffiec.cfpb.gov/)  \n| HMDA-Help  | https://github.com/cfpb/hmda-help  | ReactJS Front-end repository powering HMDA Help - used to resolve and troubleshoot issues in filing  |   \n| LARFT | https://github.com/cfpb/hmda-platform-larft  |  Repo for the [Public Facing LAR formatting tool](https://ffiec.cfpb.gov/tools/lar-formatting) |   \n| HMDA Test Files  | https://github.com/cfpb/hmda-test-files  | Repo for automatically generating various different test files for HMDA Data  |   \n| HMDA Census |  https://github.com/cfpb/hmda-census | ETL for geographic and Census data used by the HMDA Platform\n| HMDA Data Science | https://github.com/cfpb/HMDA_Data_Science_Kit | Repo for HMDA Data science work as well as Spark codebase for [Public Facing A\u0026D Reports](https://ffiec.cfpb.gov/data-publication/disclosure-reports/2018)\n\n## Contents\n\n- [TS and LAR File Specs](#ts-and-lar-file-specs)\n- [End-to-End filing GIF](#end-to-end-filing-gif)\n- [Technical Overview](#technical-overview)\n  * [Microservices](#microservices)\n- [HMDA Platform Technical Architecture](#hmda-platform-technical-architecture)\n- [HMDA Data Browser Technical Architecture](#hmda-data-browser-technical-architecture)\n- [Running with sbt](#running-with-sbt)\n- [One-line Cloud Deployment to Dev/Prod](#one-line-cloud-deployment-to-devprod)\n- [Docker Hub](#docker-hub)\n- [One-line Local Development Environment (No Auth)](#one-line-local-development-environment-no-auth)\n- [Automated Testing](#automated-testing)\n- [Postman Collection](#postman-collection)\n- [API Documentation](#api-documentation)\n- [Sprint Cadence](#sprint-cadence)\n- [Code Formatting](#code-formatting)\n- [Development Process](#development-process)\n- [Contributing](#contributing)\n- [Issues](#issues)\n- [Open source licensing info](#open-source-licensing-info)\n- [Credits and references](#credits-and-references)\n\n## TS and LAR File Specs\n\nThe data is submitted in a flat pipe (`|`) delimited TXT file. The text file is split into two parts: Transmission (TS) File -- first line in the file and Loan Application Register (LAR) -- all remaining lines of the file. Below are the links to the file specifications for data collected in years 2018 - current.\n- [Transmission  File Spec](https://github.com/cfpb/hmda-platform/blob/master/docs/spec/Public_File_TS_Spec.csv)\n- [Loan Application Register File Spec](https://github.com/cfpb/hmda-platform/blob/master/docs/spec/Public_File_LAR_Spec.csv)\n\n## End-to-End filing GIF\n\nThe [hmda-frontend](https://github.com/cfpb/hmda-frontend) uses Cypress to test the end-to-end filing process from the end user perspective. The GIF below shows the automated filing process via Cypree - no human intervention.\n\n![Cypress automated filing test](https://github.com/cfpb/hmda-frontend/raw/master/readme-files/filing-2020-q1-cypress.gif)\n\n## Technical Overview\n\nThis repository contains the code for the entirety of the public facing [HMDA Platform](http://ffiec.cfpb.gov/) backend. This platform has been designed to accommodate the needs of the HMDA filing process by financial institutions, as well as the data management, publication, aggregation, reporting, analyzing, visualizing, and downloading the HMDA data set.\n\nThe HMDA Platform follows a loosely coupled [event driven](https://en.wikipedia.org/wiki/Event-driven_architecture) [micro-services architecture](https://en.wikipedia.org/wiki/Microservices) with API-first [(API Documentation)](https://cfpb.github.io/hmda-platform/#hmda-api-documentation) design principles. The entire platform is built on open source frameworks and remains cloud vendor agnostic.\n\n### Microservices\n\nThe code base contained in this repository includes the following microservices that work together in support of the HMDA Platform.\n\n- [HMDA Platform](https://github.com/cfpb/hmda-platform/tree/master/hmda): The entire backend API for [public facing filing platform](https://ffiec.cfpb.gov/filing/2019/). Used for processing the uploaded TXT files and validating them in a non-blocking, I/O streaming way. The APIs are built to be able to process various file sizes, from small (few lines) to large (1.5M+ lines), text files simultaneously without impeding the scalability or availability of the platform. The platform contains code for customizable data edits, a [Domain Specific Language (DSL)](https://en.wikipedia.org/wiki/Domain-specific_language) for coding the data edits, and submitting events to Kafka topics.    \n\n- [Check Digit](https://github.com/cfpb/hmda-platform/tree/master/check-digit): The entire backend API for [public facing check digit tool](https://ffiec.cfpb.gov/tools/check-digit). The Check Digit tool is used to (1) Generate a two character check-digit based on an Legal Entity Identifier (LEI) and (2) Validate that a check-digit is calculated correctly for any complete Universal Loan Identifier (ULI). This APIs are built to process multiple row CSV files as well as one time processing.\n\n- [Institutions API](https://github.com/cfpb/hmda-platform/tree/master/institutions-api): Read only API for fetching details about an LEI. This microservice also listens to events put on the `institutions-api` Kafka topic for Creating, updating, and deleting institution data from PostgreSQL.\n\n- [Data Publisher](https://github.com/cfpb/hmda-platform/tree/master/hmda-data-publisher): This microservice runs on a scheduled basis to make internal / external data available for research purposes via object stores such as S3. The schedule for the job is configurable via [K8s config map](https://github.com/cfpb/hmda-platform/blob/master/kubernetes/config-maps/schedule-configmap.yaml)\n\n- [Ratespread](https://github.com/cfpb/hmda-platform/tree/master/ratespread-calculator): Public facing API for the [ratespread calculator](https://ffiec.cfpb.gov/tools/rate-spread). This calculator provides rate spreads for HMDA reportable loans with a final action date on or after January 1st, 2018. This API supports streaming CSV uploads as well as one-time calculations.\n\n- [Modified LAR](https://github.com/cfpb/hmda-platform/tree/master/modified-lar): Event driven service of [modified-lar reports](https://ffiec.cfpb.gov/data-publication/modified-lar/2019). Each time a filer successfully submits the data, the modified-lar micro-service generates a modified-lar report and puts it in the public object store (e.g. S3). Any re-submissions automatically re-generate new modified-lar reports.\n\n- [IRS Publisher](https://github.com/cfpb/hmda-platform/tree/master/irs-publisher): Event driven service of [irs-disclosure-reports](https://ffiec.cfpb.gov/data-publication/disclosure-reports/). Each time a filer successfully submits the data, the irs-publisher microservice generates the IRS report.\n\n- [HMDA Reporting](https://github.com/cfpb/hmda-platform/tree/master/hmda-reporting): Real-time, public facing API for getting information (LEI number, institution name, and year) on LEIs who have successfully submitted their data.\n\n- [HMDA Analytics](https://github.com/cfpb/hmda-platform/tree/master/hmda-analytics): Event driven service to insert, delete, update information in PostgreSQL each time there is a successful submission. The data inserted maps with the Census data to provide information for MSAMds. It also adds race, sex, and ethnicity categorization to the data.\n\n- [HMDA Dashboard](https://github.com/cfpb/hmda-platform/tree/master/hmda-dashboard): Authenticated APIs to view realtime analytics for the filings happening on the platform. The dashboard includes summarized statistics, data trends, and supports data visualizations via frontend.\n\n- [Rate imit](https://github.com/cfpb/hmda-platform/tree/master/rate-limit): Rate limiter service working in-sync with [ambassador](https://www.getambassador.io/docs/latest/topics/running/services/rate-limit-service/) to limit the number of times in a given time period that the API can be called. If the rate limit is reached, a 503 error code is sent.\n\n- [HMDA Data Browser](https://github.com/cfpb/hmda-platform/tree/master/data-browser): Public facing API for [HMDA Data Browser](https://ffiec.cfpb.gov/data-browser/). This API makes the entire dataset available for summarized statistics, deep analysis, as well as geographic map layout.\n\n- [Email Service](https://github.com/cfpb/hmda-platform/tree/master/email-service): Event driven service to send an automated email to the filer on each successful submission.\n\n\n## HMDA Platform Technical Architecture\n\nThe image below shows the cloud vendor agnostic technical architecture for the HMDA Platform.\n\n\u003ca href=\"diagrams/hmda_platform_diagram.png\"\u003e\u003cimg src=\"diagrams/hmda_platform_diagram.png\" style=\"border: 2px solid #000;\" /\u003e\u003c/a\u003e\n\n## HMDA Data Browser Technical Architecture\n\n\u003ca href=\"data-browser/README.md\"\u003ePlease view the README for HMDA Data Browser\u003c/a\u003e\n\n## Installations\nBefore running the HMDA Platform, make sure to have the following installed:\n\n### MacOS\n1. Install [Homebrew](https://brew.sh/):\n    ```bash\n    /bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\"\n    ```\n2. Install [Docker](https://www.docker.com/):\n    ```bash\n    brew install docker\n    ```\n3. Install one of following docker engine implementations:\n    * [Docker Desktop](https://docs.docker.com/desktop/install/mac-install/)\n    * [Podman](https://podman.io/)\n    * [Colima](https://github.com/abiosoft/colima/)\n    * [Rancher Desktop](https://rancherdesktop.io/)\n4. Go to the following link for instructions to install OpenJDK:\n    * [Java (version 13.0.2) for MacOS](/docs/JavaInstall.md)\n5. Install Scala (version 2.12 for compatibility issues):\n    ```bash \n    brew install scala@2.12\n    ```\n6. Install [sdkman](https://sdkman.io/install):\n    ```bash\n    curl -s \"https://get.sdkman.io\" | bash\n    ```\n7. Install [sbt](https://www.scala-sbt.org/) using `sdkman` (not `brew`):\n    ```bash\n    sdk install sbt \u003cVERSION\u003e\n    ```\n    * Replace `\u003cVERSION\u003e` with the version used in `project/build.properties` or higher\n\n8. Clone the repo and go into the repo directory:\n    ```bash\n    git clone https://github.com/cfpb/hmda-platform.git\n    cd hmda-platform\n    ```\n\n### Apple Silicon\n\nThe current platform and specifically Cassandra have problems running on \"Apple silicon\" architecture. If your laptop **About This Mac** information shows an Apple M1 or later chip, this applies to you. This will cause test suites to abort.\n\nThe current solution is to install, build and run with an amd64-compatible JDK.\n\n```bash\nbrew install asdf\narch -x86_64 asdf plugin-add java https://github.com/halcyon/asdf-java.git\narch -x86_64 asdf install java openjdk-13.0.2\nexport JAVA_HOME=$HOME/.asdf/installs/java/openjdk-13.0.2\n```\n\n## Running Locally\n\n### Running with sbt\n\nThe HMDA Platform can run locally using [`sbt`](https://www.scala-sbt.org/) with an [embedded Cassandra](https://doc.akka.io/docs/alpakka-kafka/current/) and [embedded Kafka](https://doc.akka.io/docs/alpakka-kafka/current/). To get started:\n\n1. Export the following environment variables:\n    ```bash\n    export CASSANDRA_CLUSTER_HOSTS=localhost\n    export APP_PORT=2551\n    ```\n2. Open terminal with `hmda-platform` root as the  working directory\n3. Start sbt and run the platform with the following commands:\n    ```bash\n    sbt\n    [...]\n    sbt:hmda-root\u003e project hmda-platform\n    sbt:hmda-platform\u003e reStart\n    ```\n### Building and running the .jar       \nTo build JVM artifacts, from the sbt prompt first choose the project you want to build and use the assembly command:   \n   ```bash\n    sbt\n    sbt:root\u003e project check-digit\n    sbt:check-digit\u003eassembly\n   ```\nThis task will create a fat jar, which can be executed on any JDK9 compliant JVM:   \n`java -jar target/scala-2.12/check-digit.jar`\n### Running with docker compose\n\nThe platform and it's dependency services, Kafka, Cassandra and PostgreSQL, can run locally using [Docker Compose](https://docs.docker.com/compose/).\nThe entire filing plaform can be spun up using a one line command. Using this locally running instance of Platform One, no authentication is needed. \n\n```bash\n# Bring up every service (e.g., \"hmda-platform\", \"hmda-analytics\", \"institutions-api\")\ndocker compose up\n\n# Bring up a single service (e.g., \"hmda-platform\")\ndocker compose up hmda-platform\n```\n\nAdditionally, there are several environment varialbes that can be configured/changed. The platform uses sensible defaults for each one. However, if required they can be overridden:\n\n```bash\nCASSANDRA_CLUSTER_HOSTS\nCASSANDRA_CLUSTER_DC\nCASSANDRA_CLUSTER_USERNAME\nCASSANDRA_CLUSTER_PASSWORD\nCASSANDRA_JOURNAL_KEYSPACE\nCASSANDRA_SNAPSHOT_KEYSPACE\nKAFKA_CLUSTER_HOSTS\nAPP_PORT\nHMDA_HTTP_PORT\nHMDA_HTTP_ADMIN_PORT\nHMDA_HTTP_PUBLIC_PORT\nMANAGEMENT_PORT\nHMDA_CASSANDRA_LOCAL_PORT\nHMDA_LOCAL_KAFKA_PORT\nHMDA_LOCAL_ZK_PORT\nWS_PORT\n```\n\n### Access locally built platform APIs\nThe following API endpoints are accessible when running the platform locally using either [Running with sbt](#running-with-sbt) or [Running with docker compose](#running-with-docker-compose).\n* [hmda-admin-api](http://localhost:8081)   \n* [hmda-filing-api](http://localhost:8080)   \n* [hmda-public-api](http://localhost:8082)   \n\n## Build hmda-platform Docker image\n\nDocker Image is build via Docker plugin utilizing [sbt-native-packager](https://sbt-native-packager.readthedocs.io/en/stable/formats/docker.html#docker-plugin)\n\n```\nsbt -batch clean hmda-platform/docker:publishLocal\n```\n\nThe image can be built without running tests using:\n\n```\nsbt \"project hmda-platform\" dockerPublishLocalSkipTests\n```\n\n## One-line Cloud Deployment to Dev/Prod\n\nThe platform and all of the related microservices explained above are deployed on [Kubernetes](https://kubernetes.io/) using [Helm](https://helm.sh/). Each deployment is a single Helm command. Below is an example for the deployment of the email-service:\n\n```shell\nhelm upgrade --install --force \\                            \n--namespace=default \\\n--values=kubernetes/hmda-platform/values.yaml \\\n--set image.repository=hmda/hmda-platform \\\n--set image.tag=\u003ctag name\u003e \\\n--set image.pullPolicy=Always \\\nhmda-platform \\\nkubernetes/hmda-platform\n```\n\n## Docker Hub\n\nAll of the containers built by the HMDA Platform are released publicly via Docker Hub: https://hub.docker.com/u/hmda\n\n## Automated Testing\n\nThe HMDA Platform takes a rigorous automated testing approach. In addtion to Travis and CodeCov, we've prepared a suite of [Newman](https://github.com/cfpb/hmda-platform/tree/master/newman) test scripts that perform end-to-end testing of the APIs on a recurring basis. The testing process for Newman is containerized and runs as a Kubernetes CronJob to act as a monitoring and alerting system. The platform and microservices are also testing for load by using [Locust](https://locust.io/).\n\n## Postman Collection\n\nIn addition to using Newman for our internal testing, we've created a [HMDA Postman](https://github.com/cfpb/hmda-platform/tree/master/newman/postman) collection that makes it easier for  users to perform a end-to-end filing of HMDA Data, including upload, parsing data, flagging edits, resolving edits, and submitting data when S/V edits are resolved.\n\n## API Documentation\n\nThe [HMDA Platform Public API Documentation](https://cfpb.github.io/hmda-platform/#hmda-api-documentation) is hosted in the [HMDA Platform API Docs repo](https://github.com/cfpb/hmda-platform-api-docs) and deployed to GitHub Pages using the [`gh-pages`](https://github.com/cfpb/hmda-platform/tree/gh-pages) branch.\n\n\n## Sprint Cadence\n\nOur team works in two week sprints. The sprints are managed as [Project Boards](https://github.com/cfpb/hmda-platform/projects). The backlog grooming happens every two weeks as part of Sprint Planning and Sprint Retrospectives.\n\n## Code Formatting\n\nOur team uses [Scalafmt](https://scalameta.org/scalafmt/) to format our codebase. \n\n## Development Process\n\nBelow are the steps the development team follows to fix issues, develop new features, etc.\n\n1. Create a fork of this repository\n2. Work in a branch of the fork\n3. Create a PR to merge into master\n4. The PR is automatically built, tested, and linted using: Travis, Snyk, and CodeCov\n5. Manual review is performed in addition to ensuring the above automatic scans are positive\n6. The PR is deployed to development servers to be checked using Newman\n7. The PR is merged only by a separate member in the dev team\n\n## Contributing\n\n[`CFPB`](https://www.consumerfinance.gov/) is developing the `HMDA Platform` in the open to maximize transparency and encourage third party contributions. If you want to contribute, please read and abide by the terms of the [License](LICENSE) for this project. [Pull Requests](https://help.github.com/articles/using-pull-requests/) are always welcome.\n\n## Issues\n\nWe use GitHub issues in this repository to track features, bugs, and enhancements to the software.\n\n## Open source licensing info\n1. [TERMS](TERMS.md)\n2. [LICENSE](LICENSE)\n3. [CFPB Source Code Policy](https://github.com/cfpb/source-code-policy/)\n\n## Credits and references\n\nRelated projects\n  - https://github.com/cfpb/hmda-combined-documentation - ReactJS Front-end with intergration DocSearch program repository powering the [HMDA Platform](http://ffiec.cfpb.gov/)\n  - https://github.com/cfpb/hmda-platform-larft - Repo for the [Public Facing LAR formatting tool](https://ffiec.cfpb.gov/tools/lar-formatting)\n  - https://github.com/cfpb/hmda-test-files - Repo for automatically generating various different test files for HMDA Data\n  - https://github.com/cfpb/hmda-census - ETL for geographic and Census data used by the HMDA Platform\n  - https://github.com/cfpb/HMDA_Data_Science_Kit - Repo for HMDA Data science work as well as Spark codebase for [Public Facing A\u0026D Reports](https://ffiec.cfpb.gov/data-publication/disclosure-reports/2018)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcfpb%2Fhmda-platform","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcfpb%2Fhmda-platform","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcfpb%2Fhmda-platform/lists"}