{"id":24788827,"url":"https://github.com/rizkipragustono/etl_data_pipelines","last_synced_at":"2026-05-02T05:04:18.203Z","repository":{"id":272124404,"uuid":"915598753","full_name":"rizkipragustono/etl_data_pipelines","owner":"rizkipragustono","description":"Practice Project: ETL and Data Pipelines with Shell, Airflow and Kafka","archived":false,"fork":false,"pushed_at":"2025-01-12T10:13:56.000Z","size":6,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-24T12:16:49.532Z","etag":null,"topics":["apache-airflow","apache-kafka","bash","data-engineering","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rizkipragustono.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-12T09:49:52.000Z","updated_at":"2025-01-12T10:16:01.000Z","dependencies_parsed_at":"2025-01-12T10:32:22.284Z","dependency_job_id":"976dad1f-28b3-466c-8fa1-bf858f391351","html_url":"https://github.com/rizkipragustono/etl_data_pipelines","commit_stats":null,"previous_names":["rizkipragustono/etl_data_pipelines"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rizkipragustono/etl_data_pipelines","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rizkipragustono%2Fetl_data_pipelines","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rizkipragustono%2Fetl_data_pipelines/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rizkipragustono%2Fetl_data_pipelines/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rizkipragustono%2Fetl_data_pipelines/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rizkipragustono","download_url":"https://codeload.github.com/rizkipragustono/etl_data_pipelines/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rizkipragustono%2Fetl_data_pipelines/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262352619,"owners_count":23297689,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-airflow","apache-kafka","bash","data-engineering","python"],"created_at":"2025-01-29T17:15:11.717Z","updated_at":"2026-05-02T05:04:18.175Z","avatar_url":"https://github.com/rizkipragustono.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ETL and Data Pipelines with Shell, Airflow and Kafka\nYou are a Data Engineer at a data analytics consulting company, assigned to a project that aims to de-congest national highways by analyzing road traffic data from different toll plazas. Each highway is operated by a different toll operator with varying IT setups and file formats. Your job is to create three data pipelines to collect, process, and store this data.\n\nPipeline 1: Batch Processing with Apache Airflow and BashOperator\nThe first pipeline uses Apache Airflow with BashOperator to automate batch data collection from different toll operators. It fetches data files in various formats (e.g., CSV, JSON, XML), processes them using bash scripts, and consolidates them into a single, unified file.\n\nPipeline 2: Data Processing with Apache Airflow and PythonOperator\nThe second pipeline uses Apache Airflow with PythonOperator to process the consolidated data. Python scripts handle data transformations, such as aggregating traffic data by toll plaza, and load it into a database for further analysis.\n\nPipeline 3: Real-Time Streaming with Kafka\nThe third pipeline collects real-time data as vehicles pass through toll plazas. Vehicle data, including vehicle_id, vehicle_type, toll_plaza_id, and timestamp, is streamed to Kafka. The data is then processed in real-time and loaded into a database for live traffic analysis.\n\nThese three pipelines together provide a solution for handling both historical and real-time traffic data to optimize highway traffic flow.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frizkipragustono%2Fetl_data_pipelines","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frizkipragustono%2Fetl_data_pipelines","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frizkipragustono%2Fetl_data_pipelines/lists"}