{"id":22190966,"url":"https://github.com/yuhexiong/airflow-dag-kafka-flink-doris-python","last_synced_at":"2025-07-23T11:06:35.518Z","repository":{"id":245963638,"uuid":"807402450","full_name":"yuhexiong/airflow-dag-kafka-flink-doris-python","owner":"yuhexiong","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-24T07:30:30.000Z","size":44,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-24T08:34:43.541Z","etag":null,"topics":["airflow","airflow-dags","airflow-docker","apache-airflow","apache-doris","apache-flink","apache-kafka","doris","flink","kafka","workflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yuhexiong.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-29T03:21:48.000Z","updated_at":"2025-04-24T07:30:34.000Z","dependencies_parsed_at":"2024-06-25T03:44:51.742Z","dependency_job_id":"2026d9ea-bebb-45d6-b29f-ba95e7ac45d8","html_url":"https://github.com/yuhexiong/airflow-dag-kafka-flink-doris-python","commit_stats":null,"previous_names":["yuhexiong/airflow-dag-kafka-flink-doris-python"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/yuhexiong/airflow-dag-kafka-flink-doris-python","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuhexiong%2Fairflow-dag-kafka-flink-doris-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuhexiong%2Fairflow-dag-kafka-flink-doris-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuhexiong%2Fairflow-dag-kafka-flink-doris-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuhexiong%2Fairflow-dag-kafka-flink-doris-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yuhexiong","download_url":"https://codeload.github.com/yuhexiong/airflow-dag-kafka-flink-doris-python/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuhexiong%2Fairflow-dag-kafka-flink-doris-python/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266665813,"owners_count":23964974,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-23T02:00:09.312Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","airflow-dags","airflow-docker","apache-airflow","apache-doris","apache-flink","apache-kafka","doris","flink","kafka","workflow"],"created_at":"2024-12-02T12:13:34.705Z","updated_at":"2025-07-23T11:06:35.507Z","avatar_url":"https://github.com/yuhexiong.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Airflow Dag Kafka/Flink/Doris\n\nuse airflow to maintain operation of pipeline and database.  \n\n## Overview\n\n- Tool: Apache Airflow v2.9.1\n- Language: Python v3.8\n\n\n## Run\n\n### ENV\nsetting config/airflow.cfg\n```\nsmtp_host = smtp.gmail.com\nsmtp_starttls = True\nsmtp_ssl = False\nsmtp_user = youremail@gmail.com\nsmtp_password = pass\nsmtp_port = 587\nsmtp_mail_from = youremail@gmail.com\n```\n\n### Run Docker\n```\ndocker compose up -d\n```\n\n### UI Page\n\nOpen http://localhost:8080 to run DAGs in UI.  \n\n\n## DAGs\n\n### 1. Check Kafka Updates\nconfirm that new data comes in every day in kafka's topic.  \n\n- setting variables\n```\nbootstrap_servers: localhost:9093\ntopic_name: topic1\nlog_file: local log file path to collect errors and send them once.\n```\n\n### 2. Check Flink Job State\ncheck all jobs on flink are running every day.  \n\n- setting variables\n```\nflink_url: localhost:8081\nlog_file: local log file path to collect errors and send them once.\n```\n\n### 3. Check Flink Job And Upload\ncheck the jobs are running normally according to the configuration file yaml every day.  \nIf there are errors, cancel and rerun them.\n\n- setting variables\n```\nlog_file: local log file path to collect errors and send them once.\n```\n\n- flink_pipeline.yaml\n```\nhost: http://localhost:8081\njars:\n  - jar_id: xxx_xxx_flink-1.0.jar\n    jobs:\n      - entry_class: com.examples.entry.KafkaToKafka\n        program_file: https://storage.googleapis.com/xxx/KafkaToKafka.yaml\n        job_name: kafka(topic-1) -\u003e kafka(topic-2)\n      - entry_class: com.examples.entry.KafkaToDoris\n        program_file: https://storage.googleapis.com/xxx/KafkaToDoris.yaml\n        job_name: kafka(topic-1) -\u003e doris(table-1)\n```\n\n### 4. Check Doris Updates\ncheck doris has new data coming in every day.  \n\n- setting variables\n```\ndoris_host: localhost\nusername: root\npassword: password\ndatabase: database\ndoris_table: doris_table\ndoris_timestamp_column: timestamp\nlog_file: local log file path to collect errors and send them once.\n```\n\n### 5. Send Exception Email\ncheck there is any data in the log file every day. If there is any data in the log file, send an email.  \n\n- setting variables\n```\nlog_file: local log file path to collect errors and send them once.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuhexiong%2Fairflow-dag-kafka-flink-doris-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyuhexiong%2Fairflow-dag-kafka-flink-doris-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuhexiong%2Fairflow-dag-kafka-flink-doris-python/lists"}