{"id":13626037,"url":"https://github.com/garystafford/tickit-data-lake-demo","last_synced_at":"2025-05-07T16:09:58.528Z","repository":{"id":37593857,"uuid":"425545707","full_name":"garystafford/tickit-data-lake-demo","owner":"garystafford","description":"Resources for video demonstrations and blog posts related to DataOps on AWS","archived":false,"fork":false,"pushed_at":"2022-01-26T03:30:20.000Z","size":44121,"stargazers_count":175,"open_issues_count":1,"forks_count":108,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-05-07T16:09:44.485Z","etag":null,"topics":["airflow","aws","data-lake","dataops","devops","redshift"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/garystafford.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-07T15:47:35.000Z","updated_at":"2025-04-16T13:20:00.000Z","dependencies_parsed_at":"2022-08-08T21:00:47.728Z","dependency_job_id":null,"html_url":"https://github.com/garystafford/tickit-data-lake-demo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garystafford%2Ftickit-data-lake-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garystafford%2Ftickit-data-lake-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garystafford%2Ftickit-data-lake-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/garystafford%2Ftickit-data-lake-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/garystafford","download_url":"https://codeload.github.com/garystafford/tickit-data-lake-demo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252912996,"owners_count":21824066,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","aws","data-lake","dataops","devops","redshift"],"created_at":"2024-08-01T21:02:08.378Z","updated_at":"2025-05-07T16:09:58.494Z","avatar_url":"https://github.com/garystafford.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# DevOps, GitOps, and DataOp for Analytics on AWS\n\n[![Test DAGs](https://github.com/garystafford/tickit-data-lake-demo/actions/workflows/test_dags.yml/badge.svg?branch=main)](https://github.com/garystafford/tickit-data-lake-demo/actions/workflows/test_dags.yml)\n\n[![Sync DAGs](https://github.com/garystafford/tickit-data-lake-demo/actions/workflows/sync_dags.yml/badge.svg?branch=main)](https://github.com/garystafford/tickit-data-lake-demo/actions/workflows/sync_dags.yml)\n\n## Source code for the following blogs and videos\n\nVideo\ndemonstration: [Building a Simple Data Lake on AWS](https://garystafford.medium.com/building-a-simple-data-lake-on-aws-df21ca092e32). Build a simple data lake on AWS using a combination of services, including Amazon MWAA, AWS Glue Data Catalog, AWS\nGlue Crawlers, AWS Glue Jobs, AWS Glue Studio, Amazon Athena, and Amazon S3.\n\nVideo\ndemonstration: [Building a Data Lake with Apache Airflow](https://garystafford.medium.com/building-a-data-lake-with-apache-airflow-b48bd953c2b). Programmatically build a simple Data Lake on AWS using Amazon Managed Workflows for Apache Airflow, AWS Glue, and\nAmazon Athena.\n\nVideo\ndemonstration: [Lakehouse Automation on AWS with Apache Airflow](https://garystafford.medium.com/data-lakehouse-automation-on-aws-1f6db2c60864). Programmatically load data into and upload data from Amazon Redshift using Apache Airflow.\n\nBlog\npost: [DevOps for DataOps: Building a CI/CD Pipeline for Apache Airflow DAGs](https://garystafford.medium.com/devops-for-dataops-building-a-ci-cd-pipeline-for-apache-airflow-dags-975e4a622f83). Build an effective CI/CD pipeline to test and deploy your Apache Airflow DAGs to Amazon MWAA using GitHub Actions.\n\n## Architectures/Workflows\n\n__DevOps for DataOps__\n\n![Data Lake Architecture](./diagram/data-science-on-aws-devops.png)\n\n__Building a Data Lake with Apache Airflow__\n\n![Data Lake Architecture](./diagram/data-science-on-aws-dl.png)\n\n__Lakehouse Automation on AWS with Apache Airflow__\n\n![Redshift Architecture](./diagram/data-science-on-aws-dw.png)\n\n## TICKIT Sample Database\n\n[Amazon Redshift TICKIT Sample Database](https://docs.aws.amazon.com/redshift/latest/dg/c_sampledb.html)\n\n## Instructions for \"Building a Simple Data Lake on AWS\"\n\n### TICKIT Tables\n\n- `tickit.saas.category`\n- `tickit.saas.event`\n- `tickit.saas.venue`\n- `tickit.crm.users`\n- `tickit.date`\n- `tickit.listing`\n- `tickit.sales`\n\n### Naming Conventions\n\n```text\n+-------------+--------------------------------------------------------------------+\n| Prefix      | Description                                                        |\n+-------------+--------------------------------------------------------------------+\n| _source     | Data Source metadata only (org. call _raw in video)                |\n| _raw        | Raw/Bronze data from data sources (org. call _converted in video)  |\n| _refined    | Refined/Silver data - raw data with initial ELT/cleansing applied  |\n| _aggregated | Gold/Aggregated data - aggregated/joined refined data              |\n+-------------+--------------------------------------------------------------------+\n```\n\n## AWS CLI Commands\n\nThere were two small changes made to the source code, as compared to the video demonstration, to help clarify the flow\nof data in the demonstration. The prefix for the (7) data source AWS Glue Data Catalog table’s prefix was switched\nfrom `raw_` from `source_`. Also, the (7) Raw/Bronze AWS Glue Data Catalog table’s prefix was switched from `converted_`\nto `raw_`. The final data flow is 1) `source_`, 2) `raw_`, 3) `refined_`, and 4) `agg_` (aggregated).\n\n```shell\nDATA_LAKE_BUCKET=\"your-data-lake-bucket\"\n\naws s3 rm \"s3://${DATA_LAKE_BUCKET}/tickit/\" --recursive\n\naws glue delete-database --name tickit_demo\n\naws glue create-database \\\n  --database-input '{\"Name\": \"tickit_demo\", \"Description\": \"Track sales activity for the fictional TICKIT web site\"}'\n\naws glue get-tables \\\n  --database-name tickit_demo \\\n  --query \"TableList[].Name\" \\\n  --output table\n\naws glue start-crawler --name tickit_postgresql\naws glue start-crawler --name tickit_mysql\naws glue start-crawler --name tickit_mssql\n\naws glue get-tables \\\n  --database-name tickit_demo \\\n  --query \"TableList[].Name\" \\\n  --expression \"source_*\"  \\\n  --output table\n\naws glue start-job-run --job-name tickit_public_category_raw\naws glue start-job-run --job-name tickit_public_date_raw\naws glue start-job-run --job-name tickit_public_event_raw\naws glue start-job-run --job-name tickit_public_listing_raw\naws glue start-job-run --job-name tickit_public_sales_raw\naws glue start-job-run --job-name tickit_public_users_raw\naws glue start-job-run --job-name tickit_public_venue_raw\n\naws glue start-job-run --job-name tickit_public_category_refine\naws glue start-job-run --job-name tickit_public_date_refine\naws glue start-job-run --job-name tickit_public_event_refine\naws glue start-job-run --job-name tickit_public_listing_refine\naws glue start-job-run --job-name tickit_public_sales_refine\naws glue start-job-run --job-name tickit_public_users_refine\naws glue start-job-run --job-name tickit_public_venue_refine\n\naws glue get-tables \\\n  --database-name tickit_demo \\\n  --query \"TableList[].Name\" \\\n  --output table\n\naws s3api list-objects-v2 \\\n  --bucket ${DATA_LAKE_BUCKET} \\\n  --prefix \"tickit/\" \\\n  --query \"Contents[].Key\" \\\n  --output table\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgarystafford%2Ftickit-data-lake-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgarystafford%2Ftickit-data-lake-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgarystafford%2Ftickit-data-lake-demo/lists"}