{"id":15676938,"url":"https://github.com/v5tech/kafka-log4j","last_synced_at":"2025-10-06T00:32:35.933Z","repository":{"id":149461734,"uuid":"59986456","full_name":"v5tech/kafka-log4j","owner":"v5tech","description":"使用kafka实现log4j日志集中管理","archived":false,"fork":false,"pushed_at":"2021-01-06T06:13:21.000Z","size":8,"stargazers_count":14,"open_issues_count":0,"forks_count":13,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-31T04:36:16.246Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/v5tech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2016-05-30T06:39:53.000Z","updated_at":"2024-07-15T10:31:01.000Z","dependencies_parsed_at":"2023-04-19T03:16:54.130Z","dependency_job_id":null,"html_url":"https://github.com/v5tech/kafka-log4j","commit_stats":null,"previous_names":["v5tech/kafka-log4j","ameizi/kafka-log4j"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/v5tech%2Fkafka-log4j","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/v5tech%2Fkafka-log4j/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/v5tech%2Fkafka-log4j/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/v5tech%2Fkafka-log4j/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/v5tech","download_url":"https://codeload.github.com/v5tech/kafka-log4j/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252792802,"owners_count":21805034,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-03T16:07:30.805Z","updated_at":"2025-10-06T00:32:30.883Z","avatar_url":"https://github.com/v5tech.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kafka-log4j 使用kafka实现log4j日志集中管理\n\n### 搭建Kafka环境\n\nhttps://github.com/ameizi/storm-example/blob/master/kafka集群搭建.md\n\n#### 安装zookeeper\n\n修改zookeeper配置文件\n\nconf/zoo.cfg\n\n```\ntickTime=2000\ninitLimit=10\nsyncLimit=5\ndataDir=/tmp/zookeeper\nclientPort=2181\n```\n\nzookeeper集群环境\n\n```\ntickTime=2000\ninitLimit=10\nsyncLimit=5\ndataDir=/tmp/zookeeper/data\nclientPort=2181\n\nserver.1=s1:2888:3888\nserver.2=s2:2888:3888\nserver.3=s3:2888:3888\n```\n\n分别在s1、s2、s3三台主机的dataDir目录下创建myid文件，其内容为分别为1、2、3\n\n启动zookeeper\n\n```\nzkServer.sh start\nzkServer.sh status\n```\n\n#### 安装Kafka\n\n下载：http://kafka.apache.org/downloads.html\n\n```\ntar zxf kafka-\u003cVERSION\u003e.tgz\ncd kafka-\u003cVERSION\u003e\n```\n\n修改kafka配置文件config/server.properties\n\n```\nbroker.id=0\nhost.name=localhost\nport=9092\n/tmp/kafka-logs-0\nzookeeper.connect=s1:2181,s2:2181,s3:2181\n```\n\n启动Kafka Server\n\n```\nbin/kafka-server-start.sh config/server.properties \u0026\n```\n\n创建Topic\n\n```\nbin/kafka-topics.sh --create --zookeeper s1:2181 --replication-factor 1 --partitions 1 --topic test\n```\n\n查看Topic\n\n```\nbin/kafka-topics.sh --list --zookeeper s1:2181\n```\n\n启动控制台Producer，向Kafka发送消息\n\n```\nbin/kafka-console-producer.sh --broker-list localhost:9092 --topic test\nThis is a message\nThis is another message\n^C\n```\n\n启动控制台Consumer，消费刚刚发送的消息\n\n```\nbin/kafka-console-consumer.sh --zookeeper s1:2181 --topic test --from-beginning\nThis is a message\nThis is another message\n```\n\n删除Topic\n\n```\nbin/kafka-topics.sh --delete --zookeeper s1:2181 --topic test\n```\n\n注：只有当delete.topic.enable=true时，该操作才有效\n\n\n配置Kafka集群（单台机器上）\n\n首先拷贝server.properties文件为多份（这里演示4个节点的Kafka集群，因此还需要拷贝3份配置文件）\n\n```\ncp config/server.properties config/server1.properties\ncp config/server.properties config/server2.properties\ncp config/server.properties config/server3.properties\n```\n\n修改server1.properties的以下内容：\n\n```\nbroker.id=1\nhost.name=localhost\nport=9093\nlog.dir=/tmp/kafka-logs-1\n```\n\n修改server2.properties的以下内容：\n```\nbroker.id=2\nhost.name=localhost\nport=9094\nlog.dir=/tmp/kafka-logs-2\n```\n\n修改server3.properties的以下内容：\n```\nbroker.id=3\nhost.name=localhost\nport=9095\nlog.dir=/tmp/kafka-logs-3\n```\n\n启动kafka集群\n\n```\nbin/kafka-server-start.sh config/server1.properties \u0026\nbin/kafka-server-start.sh config/server2.properties \u0026\nbin/kafka-server-start.sh config/server3.properties \u0026\n```\n\nTopic \u0026 Partition\n\nTopic在逻辑上可以被认为是一个queue，每条消费都必须指定它的Topic，可以简单理解为必须指明把这条消息放进哪个queue里。为了使得Kafka的吞吐率可以线性提高，物理上把Topic分成一个或多个Partition，每个Partition在物理上对应一个文件夹，该文件夹下存储这个Partition的所有消息和索引文件。\n\n在Kafka集群上创建备份因子为3，分区数为4的Topic\n\n```\nbin/kafka-topics.sh --create --zookeeper s1:2181 --replication-factor 3 --partitions 4 --topic kafka\n```\n\n说明：备份因子replication-factor越大，则说明集群容错性越强，就是当集群down掉后，数据恢复的可能性越大。所有的分区数里的内容共同组成了一份数据，分区数partions越大，则该topic的消息就越分散，集群中的消息分布就越均匀。\n\n然后使用kafka-topics.sh的--describe参数查看一下Topic为kafka的详情：\n\n```\nbin/kafka-topics.sh --describe --zookeeper s1:2181 --topic kafka\n```\n\n输出的第一行是所有分区的概要，接下来的每一行是一个分区的描述。可以看到Topic为kafka的消息，PartionCount=4，ReplicationFactor=3正是我们创建时指定的分区数和备份因子。\n\n另外：Leader是指负责这个分区所有读写的节点；Replicas是指这个分区所在的所有节点（不论它是否活着）；ISR是Replicas的子集，代表存有这个分区信息而且当前活着的节点。\n\n拿partition:0这个分区来说，该分区的Leader是server0，分布在id为0，1，2这三个节点上，而且这三个节点都活着。\n\n再来看下Kafka集群的日志：\n\n```\nls /tmp/kafka-logs-0\nls /tmp/kafka-logs-1\nls /tmp/kafka-logs-2\nls /tmp/kafka-logs-3\n```\n\n其中kafka-logs-0代表server0的日志，kafka-logs-1代表server1的日志，以此类推。\n\n从上面的配置可知，id为0，1，2，3的节点分别对应server0, server1, server2, server3。而上例中的partition:0分布在id为0, 1, 2这三个节点上，因此可以在server0, server1, server2这三个节点上看到有kafka-0这个文件夹。这个kafka-0就代表Topic为kafka的partion0。\n\n\n### kafka log4j appender\n\n```\nlog4j.rootLogger=INFO,console\n\nlog4j.logger.net.aimeizi.kafka=DEBUG,kafka\n\n# appender kafka\nlog4j.appender.kafka=kafka.producer.KafkaLog4jAppender\nlog4j.appender.kafka.topic=kafka\n# multiple brokers are separated by comma \",\".\nlog4j.appender.kafka.brokerList=localhost:9092, localhost:9093, localhost:9094, localhost:9095\nlog4j.appender.kafka.compressionType=none\nlog4j.appender.kafka.syncSend=true\nlog4j.appender.kafka.layout=org.apache.log4j.PatternLayout\nlog4j.appender.kafka.layout.ConversionPattern=%d [%-5p] [%t] - [%l] %m%n\n\n# appender console\nlog4j.appender.console=org.apache.log4j.ConsoleAppender\nlog4j.appender.console.target=System.out\nlog4j.appender.console.layout=org.apache.log4j.PatternLayout\nlog4j.appender.console.layout.ConversionPattern=%d [%-5p] [%t] - [%l] %m%n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fv5tech%2Fkafka-log4j","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fv5tech%2Fkafka-log4j","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fv5tech%2Fkafka-log4j/lists"}