Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
Projects in Awesome Lists by akarce
A curated list of projects in awesome lists by akarce .
https://github.com/akarce/e2e-structured-streaming
End-to-end data pipeline that ingests, processes, and stores data. It uses Apache Airflow to schedule scripts that fetch data from an API, sends the data to Kafka, and processes it with Spark before writing to Cassandra. The pipeline, built with Python and Apache Zookeeper, is containerized with Docker for easy deployment and scalability.
airflow apache-airflow apache-kafka apache-spark big-data cassandra docker docker-compose kafka postgresql python spark zookeeper
Last synced: 12 Oct 2024
https://github.com/akarce/udacity-data-pipeline-with-airflow
Udacity Data Engineering Nanodegree Program, Data Pipeline with Airflow project using MinIO and Postgresql.
airflow minio postgresql pyspark spark
Last synced: 12 Oct 2024
https://github.com/akarce/e2e-otp-pipeline
End to End OTP Pipeline Project using Docker, Airflow, Kafka, KafkaUI, Cassandra, MongoDB, EmailOperator, SlackWebhookOperator and DiscordWebhookOperator
airflow cassandra discord docker kafka kafkaui mongodb slack smtp smtp-mail zookeeper
Last synced: 12 Oct 2024