{"id":24538994,"url":"https://github.com/chrisammon3000/aws-permits-pipeline","last_synced_at":"2026-05-05T09:32:21.604Z","repository":{"id":106581279,"uuid":"289588657","full_name":"chrisammon3000/aws-permits-pipeline","owner":"chrisammon3000","description":"ETL pipeline for construction permits data in Los Angeles built on AWS S3, Lambda and RDS PostgreSQL.","archived":false,"fork":false,"pushed_at":"2020-10-28T21:47:40.000Z","size":153,"stargazers_count":4,"open_issues_count":4,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-09T14:05:55.537Z","etag":null,"topics":["aws","data-engineering-pipeline","postgis","postgresql","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chrisammon3000.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-23T00:24:42.000Z","updated_at":"2022-05-22T15:58:12.000Z","dependencies_parsed_at":null,"dependency_job_id":"bd906289-71a1-41f8-b083-2218ef9bc9c7","html_url":"https://github.com/chrisammon3000/aws-permits-pipeline","commit_stats":null,"previous_names":["chrisammon3000/aws-permits-pipeline"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/chrisammon3000/aws-permits-pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisammon3000%2Faws-permits-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisammon3000%2Faws-permits-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisammon3000%2Faws-permits-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisammon3000%2Faws-permits-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chrisammon3000","download_url":"https://codeload.github.com/chrisammon3000/aws-permits-pipeline/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrisammon3000%2Faws-permits-pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32643612,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-04T10:08:07.713Z","status":"online","status_checked_at":"2026-05-05T02:00:06.033Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","data-engineering-pipeline","postgis","postgresql","python"],"created_at":"2025-01-22T16:14:41.914Z","updated_at":"2026-05-05T09:32:21.592Z","avatar_url":"https://github.com/chrisammon3000.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Contributors][contributors-shield]][contributors-url]\n[![Forks][forks-shield]][forks-url]\n[![Stargazers][stars-shield]][stars-url]\n[![Issues][issues-shield]][issues-url]\n[![MIT License][license-shield]][license-url]\n[![LinkedIn][linkedin-shield]][linkedin-url]\n\naws-permits-pipeline\n==============================\n\nAn ETL pipeline for construction permits data from the [Los Angeles Open Data Portal](https://data.lacity.org/) hosted on AWS using  Lambda, RDS PostgreSQL, and S3. Once deployed the pipeline fetches fresh data once a day from the internet and loads it into an RDS instance running PostgreSQL/PostGIS. \n\n## Background\nCited from [Building and Safety Permit Information](https://data.lacity.org/A-Prosperous-City/Building-and-Safety-Permit-Information-Old/yv23-pmwf):\u003cbr\u003e\n\u003e*\"The Department of Building and Safety issues permits for the construction, remodeling, and repair of buildings and structures in the City of Los Angeles. Permits are categorized into building permits, electrical permits, and mechanical permits\"*\n\nThe raw permits data available from the [Los Angeles Open Data Portal](https://data.lacity.org/) contains information on building and construction permits for both residential and commercial properties. \n\n### Data source\nData can be downloaded directly here (~500MB):\u003cbr\u003e\nhttps://data.lacity.org/api/views/yv23-pmwf/rows.csv?accessType=DOWNLOAD\n\n## Built With\nThe pipeline is built on these frameworks and platforms:\n* AWS: Lambda, S3, RDS PostgreSQL, Parameter Store\n* CloudFormation\n* Serverless framework\n* Python\n* psycopg2\n* PostGIS\n\n## Getting Started\n\n### Prerequisites\nThe following should be installed:\n* [awscli](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) and [credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html)\n* [Serverless](https://www.serverless.com/framework/docs/getting-started/) framework\n* [Docker](https://docs.docker.com/get-docker/)\n\n### Setting up Environment\n1. Clone this repo:\n   ```\n   git clone \\\n       --branch master --single-branch --depth 1 --no-tags \\\n       https://github.com/abk7777/aws-permits-pipeline.git\n   ```\n\n2. Install serverless-python-requirements plugin:\n   ```\n   cd aws-permits-pipeline \u0026\u0026\n   npm install --save serverless-python-requirements\n   ```\n3. Run the script:\n   ```\n   bash scripts/set_parameters.sh\n   ```\n   *(optional)* Edit the file `scripts/set_parameters.sh` to set your own parameters for the database name, username and password.\n\n### Deploy AWS Infrastructure\nThese instructions will deploy a stack named `aws-permits-pipeline` or `aws-permits-pipeline-vpc` using AWS CloudFormation. Note that the `--stack-name` parameter must match the `service` variable in `serverless.yml` so that Serverless can import the CloudFormation output values:\n\n1. Deploy the RDS stack with or without VPC \u0026 read replica:\n   \n   ```\n   # RDS only (deploys fastest)\n   aws --region us-east-1 cloudformation deploy \\\n   --template-file cfn/rds.yml \\\n   --capabilities CAPABILITY_NAMED_IAM \\\n   --stack-name aws-permits-pipeline\n   ```\n   ```\n   # RDS with VPC \u0026 read replica\n   aws --region us-east-1 cloudformation deploy \\\n   --template-file cfn/rds-vpc.yml \\\n   --capabilities CAPABILITY_NAMED_IAM \\\n   --stack-name aws-permits-pipeline-vpc\n   ```\n   If deploying RDS only without the VPC you will need to update the default VPC's security group to allow connections on port 5432. This can be done through the AWS RDS and EC2 console. Deploying RDS with VPC can take a while since it provisions two instances instead of just one.\n\n2. Deploy Lambda functions with Serverless framework:\n   ```\n   cd src/functions \u0026\u0026\n   serverless deploy\n   ```\n\n### Initialize the Database\n   ```\n   serverless invoke --function initDatabase \\\n   --stage dev \\\n   --region us-east-1 \\\n   --log\n   ```\n   This installs the Postgres extensions for S3 import and PostGIS and creates a table `permits_raw`.\n\n### Running the Pipeline\n   The fetchData and loadData Lambda functions are scheduled to run once a day to fetch and load fresh data.\n   ```\n   serverless invoke --function fetchData \\\n   --stage dev \\\n   --region us-east-1 \\\n   --log\n   ```\n\n### Accessing the database\nOnce the instance is running any SQL client such as `psql` can access the database on port 5432 using the parameters specified in `set_parameters.sh`. To retrieve the endpoint:\n   \n   ```\n   aws cloudformation describe-stacks \\\n   --stack-name aws-permits-pipeline \\\n   --query \"Stacks[0].Outputs[?OutputKey=='MasterEndpointDB'].OutputValue\" \\\n   --output text\n   ```\n\n   Using `psql`:\n   ```\n   psql -h \u003cMasterEndpointDB\u003e -U postgres -p 5432 -d permitsDB\n   ```\n\n### Cleaning up\n1. Delete the Serverless stack:\n   ```\n   serverless remove\n   ```\n   \n2. Get the S3 data bucket name:\n   ```\n   aws cloudformation describe-stacks \\\n   --stack-name aws-permits-pipeline \\\n   --query \"Stacks[0].Outputs[?OutputKey=='DataBucket'].OutputValue\" \\\n   --output text\n   ```\n   \n3. Delete all data from the S3 bucket:\n   ```\n   aws s3 rm --recursive s3://\u003cyour_bucket_name\u003e\n   ```\n4. Delete the CloudFormation stack:\n   ```\n   aws cloudformation delete-stack \\\n   --stack-name aws-permits-pipeline\n   ```\n   \n## Contributors\n\n**Primary (Contact) : [Gregory Lindsey](https://github.com/gclindsey)**\n\n[contributors-shield]: https://img.shields.io/github/contributors/abk7777/aws-permits-pipeline.svg?style=flat-square\n[contributors-url]: https://github.com/abk7777/aws-permits-pipeline/graphs/contributors\n[forks-shield]: https://img.shields.io/github/forks/abk7777/aws-permits-pipeline.svg?style=flat-square\n[forks-url]: https://github.com/abk7777/aws-permits-pipeline/network/members\n[stars-shield]: https://img.shields.io/github/stars/abk7777/aws-permits-pipeline.svg?style=flat-square\n[stars-url]: https://github.com/abk7777/aws-permits-pipeline/stargazers\n[issues-shield]: https://img.shields.io/github/issues/abk7777/aws-permits-pipeline.svg?style=flat-square\n[issues-url]: https://github.com/abk7777/aws-permits-pipeline/issues\n[license-shield]: https://img.shields.io/github/license/abk7777/aws-permits-pipeline.svg?style=flat-square\n[license-url]: https://github.com/abk7777/aws-permits-pipeline/blob/master/LICENSE\n[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=flat-square\u0026logo=linkedin\u0026colorB=555\n[linkedin-url]: https://linkedin.com/in/gregory-lindsey/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrisammon3000%2Faws-permits-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchrisammon3000%2Faws-permits-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrisammon3000%2Faws-permits-pipeline/lists"}