https://github.com/stefen-taime/moderndataengineerpipeline
Building a Robust Data Pipeline: Integrating Proxy Rotation, Kafka, MongoDB, Redis, Logstash, Elasticsearch, and MinIO for Efficient Web Scraping
https://github.com/stefen-taime/moderndataengineerpipeline
auth0 connect dataengineering docker-compose elasticsearch fastapi kafka logstash minio mongodb proxy redis
Last synced: 3 months ago
JSON representation
Building a Robust Data Pipeline: Integrating Proxy Rotation, Kafka, MongoDB, Redis, Logstash, Elasticsearch, and MinIO for Efficient Web Scraping
- Host: GitHub
- URL: https://github.com/stefen-taime/moderndataengineerpipeline
- Owner: Stefen-Taime
- Created: 2023-12-20T17:57:11.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2023-12-20T21:56:16.000Z (over 1 year ago)
- Last Synced: 2024-11-16T12:13:36.388Z (5 months ago)
- Topics: auth0, connect, dataengineering, docker-compose, elasticsearch, fastapi, kafka, logstash, minio, mongodb, proxy, redis
- Language: Python
- Homepage: https://medium.com/@stefentaime_10958/moderndataengineering-building-a-robust-data-pipeline-integrating-proxy-rotation-kafka-mongodb-9a908d1bd94f
- Size: 2.27 MB
- Stars: 4
- Watchers: 1
- Forks: 1
- Open Issues: 0