{"id":15685972,"url":"https://github.com/v5tech/storm-example","last_synced_at":"2025-05-07T18:21:38.608Z","repository":{"id":149461829,"uuid":"47866178","full_name":"v5tech/storm-example","owner":"v5tech","description":"a storm kafka examples","archived":false,"fork":false,"pushed_at":"2015-12-23T09:36:10.000Z","size":39,"stargazers_count":9,"open_issues_count":0,"forks_count":6,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-31T12:57:23.129Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/v5tech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-12-12T05:45:03.000Z","updated_at":"2021-12-14T03:20:24.000Z","dependencies_parsed_at":"2023-04-16T17:15:06.598Z","dependency_job_id":null,"html_url":"https://github.com/v5tech/storm-example","commit_stats":null,"previous_names":["v5tech/storm-example"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/v5tech%2Fstorm-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/v5tech%2Fstorm-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/v5tech%2Fstorm-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/v5tech%2Fstorm-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/v5tech","download_url":"https://codeload.github.com/v5tech/storm-example/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252931819,"owners_count":21827172,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-03T17:33:51.539Z","updated_at":"2025-05-07T18:21:38.565Z","avatar_url":"https://github.com/v5tech.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# storm + kafka + hdfs\n\nstorm 0.9.6\n\nkafka_2.10 0.8.2.2\n\n涉及到的内容:\n\n    kafka + spring\n    storm + kafka\n    storm + hdfs\n    kafka + storm +hdfs\n\nstorm WordCountTopology集群环境运行方式\n\n```bash\nstorm jar net.aimeizi.example.storm.WordCountTopology s1\n```\n\n# storm + kafka 集群环境搭建\n\n拷贝kafka安装目录下lib目录下的jar到storm集群目录下的lib目录。具体拷贝清单文件如下:\n\n```\nkafka_2.10-0.8.2.2.jar\nkafka-clients-0.8.2.2.jar\nmetrics-core-2.2.0.jar\nscala-library-2.10.4.jar\nsnappy-java-1.1.1.7.jar\nzkclient-0.3.jar\nzookeeper-3.4.6.jar\nstorm-kafka-0.9.6.jar\nguava-11.0.2.jar\ncurator-framework-2.5.0.jar\ncurator-client-2.5.0.jar\nlog4j-1.2.16.jar\njopt-simple-3.2.jar\n```\n\nstorm kafka集群测试环境运行\n\n```bash\nstorm jar net.aimeizi.example.MyKafkaTopology s1\n```\n\n# flume + kafka\n\nflume整合kafka。下面的配置为flume收集nginx访问日志到kafka集群\n\nflume-1.6.0 + kafka_2.10-0.8.2.2\n\nflume-kafka.properties\n\n```\n#agent section\na.sources = r\na.sinks = k\na.channels = c\n\n#source section\na.sources.r.type = exec\na.sources.r.channels = c\na.sources.r.command = tail -f /var/log/nginx/access.log\n\n#sink section\na.sinks.k.type = org.apache.flume.sink.kafka.KafkaSink\na.sinks.k.topic = kafka-storm\na.sinks.k.brokerList = 192.168.64.128:9092,192.168.64.129:9092,192.168.64.131:9092\na.sinks.k.requiredAcks = 1\na.sinks.k.batchSize = 20\na.sinks.k.channel = c\n\n#channel section\na.channels.c.type = memory\na.channels.c.capacity = 1000\na.channels.c.transactionCapacity = 100\n\n# Bind the source and sink to the channel\na.sources.r.channels = c\na.sinks.k.channel = c\n```\n\n开始采集\n\n```bash\nbin/flume-ng agent -n a -c conf -f conf/flume-kafka.properties\n```\n\n# 参考文章\n\nhttps://github.com/spring-projects/spring-integration-kafka\n\nhttps://github.com/smallnest/spring-kafka-demo\n\nhttp://shiyanjun.cn/archives/934.html\n\nhttp://blog.csdn.net/weijonathan/article/details/18301321","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fv5tech%2Fstorm-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fv5tech%2Fstorm-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fv5tech%2Fstorm-example/lists"}