{"id":20750317,"url":"https://github.com/bhavanachitragar/zillow-data-analytics","last_synced_at":"2026-05-27T21:31:31.755Z","repository":{"id":243195971,"uuid":"811668473","full_name":"bhavanachitragar/zillow-data-analytics","owner":"bhavanachitragar","description":" A Python script extracts data from Zillow and stores it in an initial S3 bucket. Then, Lambda functions handle the flow: copying the data to a processing bucket and transforming it from JSON to CSV format. The final CSV data resides in another S3 bucket, ready to be loaded into Amazon Redshift for in-depth analysis.  QuickSight for visualizations","archived":false,"fork":false,"pushed_at":"2024-06-10T07:10:12.000Z","size":68,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-13T02:50:54.709Z","etag":null,"topics":["airflow-dags","ec2-instance","etl-pipeline","lambda-functions","quicksight-dashboard","redshift","s3","zillow-api"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bhavanachitragar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-07T04:11:03.000Z","updated_at":"2024-06-10T07:10:16.000Z","dependencies_parsed_at":"2024-11-17T23:02:08.510Z","dependency_job_id":null,"html_url":"https://github.com/bhavanachitragar/zillow-data-analytics","commit_stats":null,"previous_names":["bhavanachitragar/zillow-data-analytics"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bhavanachitragar/zillow-data-analytics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bhavanachitragar%2Fzillow-data-analytics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bhavanachitragar%2Fzillow-data-analytics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bhavanachitragar%2Fzillow-data-analytics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bhavanachitragar%2Fzillow-data-analytics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bhavanachitragar","download_url":"https://codeload.github.com/bhavanachitragar/zillow-data-analytics/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bhavanachitragar%2Fzillow-data-analytics/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33585203,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-27T02:00:06.184Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow-dags","ec2-instance","etl-pipeline","lambda-functions","quicksight-dashboard","redshift","s3","zillow-api"],"created_at":"2024-11-17T08:26:45.365Z","updated_at":"2026-05-27T21:31:31.737Z","avatar_url":"https://github.com/bhavanachitragar.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Zillow Data Analytics using AWS\n\n![Architecture drawio (1)](https://github.com/bhavanachitragar/zillow-data-analytics/assets/91766461/d36ab26a-5f22-4866-8e1b-ddea2ed2bd98)\n\n## This architecture leverages:\n- Airflow: For scheduling and orchestration of the data pipeline tasks.\n- EC2: For running the Python scripts for data extraction and transformation.\n- Lambda Functions: For serverless, triggered processing of data transfer between S3 buckets.\n- S3: For storing data at various stages of the pipeline.\n- Redshift: For efficient data warehousing and analytics.\n- QuickSight: For data visualization and exploration.\n\n## Steps included:\n1. Python Script: Extracts data from Zillow in JSON format and stores it in an S3 bucket.\n2. S3 Bucket (Staging): Stores the initial extracted JSON data.\n3. AWS Lambda Function 1 (Data Transfer): Triggers upon new data in the staging S3 bucket and copies the JSON data to a destination S3 bucket.\n4. S3 Bucket (Processing): Holds the JSON data ready for further processing.\n5. AWS Lambda Function 2 (Data Transformation): Triggers upon new data in the processing S3 bucket, reads the JSON data, converts it to CSV format, and stores the CSV data in a designated S3 bucket.\n6. S3 Bucket (Transformed Data): Stores the final processed data in CSV format.\n7. Amazon Redshift: Stores the CSV data from the transformed data S3 bucket for efficient data warehousing and analytics.\n8. Amazon QuickSight: Connects to the Redshift data warehouse to visualize and analyze the Zillow data.\n\n\n## Airflow\n### DAG View\n![Screenshot 2024-06-10 114256](https://github.com/bhavanachitragar/zillow-data-analytics/assets/91766461/deadc3d2-cdbe-4c4e-8b70-50c6e4cdac57)\n\n\n## Redshift\n### Transformed data is loaded into Amazon Redshift\n![Screenshot 2024-06-10 105008](https://github.com/bhavanachitragar/zillow-data-analytics/assets/91766461/4c4cd696-5982-4b7c-9c9a-bb5552eb87eb)\n\n## Quicksight\n### Creating visualizations and dashboards from data sources\n![Screenshot 2024-06-10 123457](https://github.com/bhavanachitragar/zillow-data-analytics/assets/91766461/324d2a6f-8cfb-4fb7-8125-7c62ac140ab1)\n\n-----------------------------------------------------------------------------------------\n### Guided by:  Opeyemi Olanipekun\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbhavanachitragar%2Fzillow-data-analytics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbhavanachitragar%2Fzillow-data-analytics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbhavanachitragar%2Fzillow-data-analytics/lists"}