{"id":20565122,"url":"https://github.com/devlucho/spark-streaming-y-kafka","last_synced_at":"2025-06-10T22:08:44.297Z","repository":{"id":260081461,"uuid":"880206601","full_name":"DevLucho/Spark-Streaming-y-Kafka","owner":"DevLucho","description":"Análisis de Datos en Tiempo Real con Spark Streaming y Kafka","archived":false,"fork":false,"pushed_at":"2024-10-29T10:50:36.000Z","size":3,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-06T08:35:38.118Z","etag":null,"topics":["analitycs","kafka","python3","spark-streaming"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DevLucho.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-29T10:11:01.000Z","updated_at":"2024-10-29T10:57:32.000Z","dependencies_parsed_at":"2024-10-29T13:05:33.623Z","dependency_job_id":null,"html_url":"https://github.com/DevLucho/Spark-Streaming-y-Kafka","commit_stats":null,"previous_names":["devlucho/spark-streaming-y-kafka"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DevLucho%2FSpark-Streaming-y-Kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DevLucho%2FSpark-Streaming-y-Kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DevLucho%2FSpark-Streaming-y-Kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DevLucho%2FSpark-Streaming-y-Kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DevLucho","download_url":"https://codeload.github.com/DevLucho/Spark-Streaming-y-Kafka/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DevLucho%2FSpark-Streaming-y-Kafka/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259159757,"owners_count":22814502,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analitycs","kafka","python3","spark-streaming"],"created_at":"2024-11-16T04:31:39.271Z","updated_at":"2025-06-10T22:08:44.249Z","avatar_url":"https://github.com/DevLucho.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Proyecto de Streaming con Kafka y Spark\n\nEste proyecto utiliza **Apache Kafka** y **Spark Streaming** en Python para simular la generación de datos de sensores, transmitirlos a través de Kafka, y procesarlos en tiempo real con Spark Streaming.\n\n## Requisitos Previos\n\n1. **Python** instalado en el sistema.\n2. **Apache Kafka** y **ZooKeeper** configurados.\n3. **Apache Spark** con soporte para **Spark Streaming y Kafka**.\n\n## Configuración y Ejecución\n\n### Paso 1: Instalación y configuración de Kafka\n\n1. **Descargar Apache Kafka**:\n   ```bash\n   wget https://downloads.apache.org/kafka/3.6.2/kafka_2.13-3.6.2.tgz\n   ```\n\n2. **Descomprimir el archivo**:\n   ```bash\n   tar -xzf kafka_2.13-3.6.2.tgz\n   ```\n\n3. **Mover Kafka a una ubicación permanente**:\n   ```bash\n   sudo mv kafka_2.13-3.6.2 /opt/Kafka\n   ```\n\n### Paso 2: Iniciar ZooKeeper y el servidor de Kafka\n\n1. **Iniciar ZooKeeper**:\n   ```bash\n   sudo /opt/Kafka/bin/zookeeper-server-start.sh /opt/Kafka/config/zookeeper.properties \u0026\n   ```\n\n2. **Iniciar el servidor de Kafka**:\n   ```bash\n   sudo /opt/Kafka/bin/kafka-server-start.sh /opt/Kafka/config/server.properties \u0026\n   ```\n\n### Paso 3: Crear el topic para el broker de mensajeria Kafka\n\n1. **Crear un topic para transmitir los datos**:\n   ```bash\n   /opt/Kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic sensor_data\n   ```\n\n### Paso 4: Ejecutar el Productor (`kafka_producer.py`)\n\n1. **Instalar la biblioteca de Kafka para Python**:\n   ```bash\n   pip install kafka-python\n   ```\n\n2. **Ejecutar el script del productor**:\n   - Este script genera datos simulados de sensores y los envía al tema `sensor_data`.\n   ```bash\n   python3 kafka_producer.py\n   ```\n\n### Paso 5: Ejecutar el Consumidor (`spark_streaming_consumer.py`)\n\n1. **Ejecutar el script del consumidor con Spark**:\n   - Asegúrate de que el productor está generando datos antes de iniciar el consumidor.\n   ```bash\n   spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.3 spark_streaming_consumer.py\n   ```\n\n## Descripción de los Scripts\n\n- **kafka_producer.py**: Este script simula datos de sensores (temperatura, humedad, y marca de tiempo) y los envía a Kafka cada segundo.\n- **spark_streaming_consumer.py**: Este script lee datos en tiempo real desde Kafka, los procesa en ventanas de tiempo de 1 minuto, y calcula el promedio de temperatura y humedad para cada sensor.\n\n## Notas Adicionales\n\n1. Es importante iniciar el productor (`kafka_producer.py`) antes de ejecutar el consumidor (`spark_streaming_consumer.py`) para asegurar que haya datos disponibles.\n2. Los resultados del procesamiento se muestran en la consola de Spark Streaming.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevlucho%2Fspark-streaming-y-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevlucho%2Fspark-streaming-y-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevlucho%2Fspark-streaming-y-kafka/lists"}