https://github.com/stefen-taime/moderndataengineerpipeline

Building a Robust Data Pipeline: Integrating Proxy Rotation, Kafka, MongoDB, Redis, Logstash, Elasticsearch, and MinIO for Efficient Web Scraping
https://github.com/stefen-taime/moderndataengineerpipeline

auth0 connect dataengineering docker-compose elasticsearch fastapi kafka logstash minio mongodb proxy redis

Last synced: about 4 hours ago
JSON representation

Building a Robust Data Pipeline: Integrating Proxy Rotation, Kafka, MongoDB, Redis, Logstash, Elasticsearch, and MinIO for Efficient Web Scraping

• Host: GitHub
• URL: https://github.com/stefen-taime/moderndataengineerpipeline
• Owner: Stefen-Taime
• Created: 2023-12-20T17:57:11.000Z (11 months ago)
• Default Branch: main
• Last Pushed: 2023-12-20T21:56:16.000Z (11 months ago)
• Last Synced: 2023-12-21T21:55:32.259Z (11 months ago)
• Topics: auth0, connect, dataengineering, docker-compose, elasticsearch, fastapi, kafka, logstash, minio, mongodb, proxy, redis
• Language: Python
• Homepage: https://medium.com/@stefentaime_10958/moderndataengineering-building-a-robust-data-pipeline-integrating-proxy-rotation-kafka-mongodb-9a908d1bd94f
• Size: 2.27 MB
• Stars: 1
• Watchers: 1
• Forks: 1
• Open Issues: 0

Awesome Lists containing this project