https://github.com/garybake/tfgm_feed
https://github.com/garybake/tfgm_feed
Last synced: 3 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/garybake/tfgm_feed
- Owner: garybake
- License: mit
- Created: 2020-09-17T20:39:10.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2024-05-21T06:40:13.000Z (about 2 years ago)
- Last Synced: 2024-12-06T20:38:26.984Z (over 1 year ago)
- Language: Python
- Size: 10.7 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Real time TFGM feed
Setup kafka
-----------
Set initial config files
**Windows**
`bin\windows\zookeeper-server-start.bat config\zookeeper.properties`
`bin\windows\kafka-server-start.bat config\server.properties`
**Linux**
`bin/zookeeper-server-start.sh config/zookeeper.properties`
`bin/kafka-server-start.sh config/server.properties`
**Create topic**
`bin/kafka-topics.sh --create --topic metrolink --bootstrap-server localhost:9092`
`bin/kafka-topics.sh --list --zookeeper localhost:2181`
**Listen for messages**
`bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic metrolink --from-beginning`
Setup Spark (local cluster)
-------------------
**Start the master**
`./sbin/start-master.sh`
**Add a worker**
`./sbin/start-slave.sh $(hostname):7077`
**Windows**
`bin\spark-class org.apache.spark.deploy.master.Master`
`bin\spark-class org.apache.spark.deploy.worker.Worker spark://hostname:port`
**View the webui**
http://localhost:8080/
**PySpark shell**
`./bin/pyspark --master spark://$(hostname):7077`