{"id":18632153,"url":"https://github.com/wang1365/storm-traffic","last_synced_at":"2025-04-11T07:30:51.242Z","repository":{"id":28658077,"uuid":"108352474","full_name":"wang1365/storm-traffic","owner":"wang1365","description":"使用Storm实时处理交通大数据（数据源：kafka，集群管理：zookeeper）","archived":false,"fork":false,"pushed_at":"2022-12-14T20:49:23.000Z","size":65,"stargazers_count":46,"open_issues_count":1,"forks_count":30,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-05-07T18:23:05.645Z","etag":null,"topics":["kafka","storm","traffic"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wang1365.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.TXT","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-10-26T02:39:21.000Z","updated_at":"2024-05-05T14:16:57.000Z","dependencies_parsed_at":"2023-01-14T09:16:35.413Z","dependency_job_id":null,"html_url":"https://github.com/wang1365/storm-traffic","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wang1365%2Fstorm-traffic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wang1365%2Fstorm-traffic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wang1365%2Fstorm-traffic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wang1365%2Fstorm-traffic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wang1365","download_url":"https://codeload.github.com/wang1365/storm-traffic/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223460648,"owners_count":17148760,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","storm","traffic"],"created_at":"2024-11-07T05:10:28.753Z","updated_at":"2024-11-07T05:10:29.325Z","avatar_url":"https://github.com/wang1365.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Storm-traffic\n使用storm接入交通数据，实时计算速度流量等交通模型\n\n## 功能\n* Kafka数据接入\n* 实时数据清洗和过滤\n* 实时告警\n  * 检测指定车辆是否出现\n  * 峰值告警，检测一定时间内车辆数是否超过阈值\n* 车辆信息统计\n* 工具：交通数据模拟脚本, 参考[Here](./scripts/README.md) \n\n说明：告警后的通知功能不在该项目范围内，通知功能应作为单独的微服务部署并提供相应的接口。\n\n\n## 环境\n\n* Window 7\n* Storm 1.1.1\n* Zookeeper 3.4.9\n* Kafka 2.12\n* Java 8\n* Python 3.6(官方推荐版本2.6.6，3.x也可以工作)\n\n## Zookeeper 安装\nStorm cluster 使用zookeeper来管理集群，同时由于kafka也使用zookeeper来进行分布式协调，\n简化起见，我们对storm和kafka使用同一套zookeeper，作为开发环境，我们不考虑zookeeper\n的单点故障，所以不搭建zookeeper的集群  \n* 安装方法:\n    * 参考我的另一个repo：[zookeeper-gym](https://github.com/wang1365/zookeeper-gym)\n    * 官方文档：[zookeeper](https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_singleAndDevSetup)\n\n## Storm安装\n\n* 下载 storm release并解压  \n安装文件：[downloads](http://storm.apache.org/downloads.html)  \n注意：不要下载源码，源码中在storm的根目录下缺少lib文件夹，storm命令无法运行。\n\n* 环境变量设置  \n    * 为了方便的使用storm命令行，在windows系统变量中新建STORM_HOME环境变量，并将storm的bin路径添加到系统PATH中\n    * 注意JAVA_HOME环境变量中不能包含空格！！！\n    \n* Storm配置  \nStorm配置文件为`%STORM_HOME%\\conf\\storm.yaml`, 打开该文件修改配置如下：\n```yaml\n# zookeeper集群地址列表，如果zookeeper不是使用的默认端口，\n# 还需要配置storm.zookeeper.ports\nstorm.zookeeper.servers:\n    - \"localhost\"\n\n# nimbus所在主机名称\nnimbus.host: \"localhost\"\n\n# nimbus和supervisor的数据存放路径\nstorm.local.dir: \"/storm_local\"\n\n# supervisor所在机器的所有工作进程可用的端口列表\nsupervisor.slots.ports:\n    - 6700\n    - 6701\n```\n* 启动storm  \n生产环境中，一般zookeeper、nimbus、supervisor会分别部署在不同的主机上，简化起见，我们在同一台机器\n上启动部署这些进程：\n    * 启动zookeeper：`zkServer`\n    * 启动nimbus：`storm numbus`\n    * 启动supervisor：`storm supervisor`\n    * 启动storm ui：`storm ui`\n\n## Storm概念说明\n* zookeeper  \nzookeeper本身不属于Storm的范围，它是作为一个分布式协调框架帮助storm管理集群中的节点。在启动nimbus和supervisor后，\n可以发现在zookeeper节点/storm/nimbuses和/storm/supervisor下分别多了对应的子节点，并且这些子节点都是临时节点。\n\n```\n[zk: localhost:2181(CONNECTED) 14] ls2 /storm/nimbuses\n[HIH-D-8204.hz.ntes.domain:6627]\ncZxid = 0xcb\nctime = Fri Oct 27 10:11:04 CST 2017\nmZxid = 0xcb\nmtime = Fri Oct 27 10:11:04 CST 2017\npZxid = 0x460\ncversion = 3\ndataVersion = 0\naclVersion = 0\nephemeralOwner = 0x0\ndataLength = 1\nnumChildren = 1\n\n[zk: localhost:2181(CONNECTED) 15] ls2 /storm/nimbuses/HIH-D-8204.hz.ntes.domain:6627\n[]\ncZxid = 0x460\nctime = Fri Oct 27 11:13:57 CST 2017\nmZxid = 0x460\nmtime = Fri Oct 27 11:13:57 CST 2017\npZxid = 0x460\ncversion = 0\ndataVersion = 0\naclVersion = 0\nephemeralOwner = 0x15f5b97a5af000d\ndataLength = 79\nnumChildren = 0\n```\n\n* nimbus  \nnimbus进程部署在storm集群的主控节点上，负责在集群中分发代码，对工作节点分配任务，并监控主机故障\n* supervisor\nsupervisor进程部署在storm集群的每一个工作节点，负责监听工作节点上已经分配的任务，管理工作进程，给工作进程\n分配任务\n\n## Kafka安装\n* 下载并解压：[https://kafka.apache.org/downloads](https://kafka.apache.org/downloads)\n* 环境变量配置，`KAFKA_HOME`以及将`%KAFKA_HOME%\\bin\\windows`加入到系统`PATH`\n* 根据实际情况修改kafka server的配置文件：`%KAFKA_HOME%\\config\\server.properties`,\n注意其中的zookeeper的配置如`zookeeper.connect=localhost:2181`要根据实际的情况配置。\n* 启动kafka server：`kafka-server-start.bat %KAFKA_HOME%\\config\\server.properties`\n* 测试kafka server是否正常工作\n    * 启动一个consumer client：`kafka-console-consumer.bat --zookeeper localhost --topic mytopic`\n    * 启动一个producer client：`kafka-console-producer.bat --broker-list localhost:9092 --topic mytopic`  \n    * 在producer中发送一条消息，然后观察consumer中是否可以收到\n\t\n### Intellij IDEA 调试\n* 请把storm-core和storm-core的provided注释掉，否则会出现ClassNotFound异常; 集群提交时需要保留provided\n* application.yml中把localModeEnabled设置为true，集群提交设置为false","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwang1365%2Fstorm-traffic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwang1365%2Fstorm-traffic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwang1365%2Fstorm-traffic/lists"}