{"id":22937634,"url":"https://github.com/komamitsu/fluency","last_synced_at":"2025-05-16T11:02:25.459Z","repository":{"id":38361687,"uuid":"43254843","full_name":"komamitsu/fluency","owner":"komamitsu","description":"High throughput data ingestion logger to Fluentd, AWS S3 and Treasure Data","archived":false,"fork":false,"pushed_at":"2025-02-26T04:45:15.000Z","size":1856,"stargazers_count":154,"open_issues_count":14,"forks_count":34,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-19T13:44:26.949Z","etag":null,"topics":["aws-s3","fluentd","logger","s3","treasuredata"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/komamitsu.png","metadata":{"files":{"readme":"README-v1.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"komamitsu"}},"created_at":"2015-09-27T16:42:26.000Z","updated_at":"2025-01-13T15:09:46.000Z","dependencies_parsed_at":"2023-12-17T05:24:51.276Z","dependency_job_id":"79077786-5c46-4570-8c86-c5c2a2b90f9e","html_url":"https://github.com/komamitsu/fluency","commit_stats":null,"previous_names":[],"tags_count":41,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/komamitsu%2Ffluency","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/komamitsu%2Ffluency/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/komamitsu%2Ffluency/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/komamitsu%2Ffluency/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/komamitsu","download_url":"https://codeload.github.com/komamitsu/fluency/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254518383,"owners_count":22084374,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-s3","fluentd","logger","s3","treasuredata"],"created_at":"2024-12-14T12:08:14.910Z","updated_at":"2025-05-16T11:02:25.409Z","avatar_url":"https://github.com/komamitsu.png","language":"Java","funding_links":["https://github.com/sponsors/komamitsu"],"categories":["项目","日志库","Projects"],"sub_categories":["联网","Networking"],"readme":"# Fluency\n[\u003cimg src=\"https://travis-ci.org/komamitsu/fluency.svg?branch=master\"/\u003e](https://travis-ci.org/komamitsu/fluency) [![Coverage Status](https://coveralls.io/repos/komamitsu/fluency/badge.svg?branch=master\u0026service=github)](https://coveralls.io/github/komamitsu/fluency?branch=master)\n\nYet another fluentd logger.\n\n## Features\n\n* Better performance ([4 times faster than fluent-logger-java](https://gist.github.com/komamitsu/c1e4045fe2ddb108cfbf12d5f014b683))\n* Asynchronous / synchronous flush to Fluentd\n* TCP / UDP heartbeat with Fluentd\n* `PackedForward` format\n* Failover with multiple Fluentds\n* Enable /disable ack response mode\n\n## Install\n\n### Gradle\n\n```groovy\ndependencies {\n    compile 'org.komamitsu:fluency:1.8.1'\n}\n```\n\n### Maven\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003eorg.komamitsu\u003c/groupId\u003e\n    \u003cartifactId\u003efluency\u003c/artifactId\u003e\n    \u003cversion\u003e1.8.1\u003c/version\u003e\n\u003c/dependency\u003e\n```\n \n## Usage\n\n### Create Fluency instance\n\n#### For single Fluentd\n\n```java\n// Single Fluentd(localhost:24224 by default)\n//   - TCP heartbeat (by default)\n//   - Asynchronous flush (by default)\n//   - Without ack response (by default)\n//   - Flush interval is 600ms (by default)\n//   - Initial chunk buffer size is 1MB (by default)\n//   - Threshold chunk buffer size to flush is 4MB (by default)\n//   - Max total buffer size is 512MB (by default)\n//   - Use off heap memory for buffer pool (by default)\n//   - Max retry of sending events is 8 (by default)\n//   - Max wait until all buffers are flushed is 10 seconds (by default)\n//   - Max wait until the flusher is terminated is 10 seconds (by default)\nFluency fluency = Fluency.defaultFluency();\n```\n\n#### For multiple Fluentd with failover\n\n```java    \n// Multiple Fluentd(localhost:24224, localhost:24225)\nFluency fluency = Fluency.defaultFluency(\n\t\t\tArrays.asList(new InetSocketAddress(24224), new InetSocketAddress(24225)));\n```\n\n#### Enable ACK response mode\n\n```java\n// Single Fluentd(localhost:24224)\n//   - With ack response\nFluency fluency = Fluency.defaultFluency(new Fluency.Config().setAckResponseMode(true));\n```\n\n#### Enable file backup mode\n\nIn this mode, Fluency takes backup of unsent memory buffers as files when closing and then resends them when restarting\n\n```java\n// Single Fluentd(localhost:24224)\n//   - Backup directory is the temporary directory\nFluency fluency = Fluency.defaultFluency(new Fluency.Config().setFileBackupDir(System.getProperty(\"java.io.tmpdir\")));\n```\n\n#### Buffer configuration\n\n```java\n// Single Fluentd(xxx.xxx.xxx.xxx:24224)\n//   - Initial chunk buffer size = 4MB\n//   - Threshold chunk buffer size to flush = 16MB\n//     Keep this value (BufferRetentionSize) between `Initial chunk buffer size` and `Max total buffer size`\n//   - Max total buffer size = 256MB\nFluency fluency = Fluency.defaultFluency(\"xxx.xxx.xxx.xxx\", 24224,\n\tnew Fluency.Config()\n\t    .setBufferChunkInitialSize(4 * 1024 * 1024)\n\t    .setBufferChunkRetentionSize(16 * 1024 * 1024)\n\t    .setMaxBufferSize(256 * 1024 * 1024L));\n```\n\n#### Waits on close sequence\n\n```java\n// Single Fluentd(localhost:24224)\n//   - Max wait until all buffers are flushed is 30 seconds\n//   - Max wait until the flusher is terminated is 40 seconds\nFluency fluency = Fluency.defaultFluency(\n\tnew Fluency.Config()\n        .setWaitUntilBufferFlushed(30)\n        .setWaitUntilFlusherTerminated(40));\n```\n\n#### Register Jackson modules\n```java\n// Single Fluentd(localhost:24224)\n//   - SimpleModule that has FooSerializer is enabled\nSender sender = new TCPSender.Config().createInstance();\n\nSimpleModule simpleModule = new SimpleModule();\nsimpleModule.addSerializer(Foo.class, new FooSerializer());\n\nPackedForwardBuffer.Config bufferConfig = new PackedForwardBuffer.Config()\n\t.setJacksonModules(Collections.\u003cModule\u003esingletonList(simpleModule));\n\nFluency fluency = new Fluency.Builder(sender).setBufferConfig(bufferConfig).build();\n```\n\n#### Set a custom error handler\n```java\nFluency fluency = Fluency.defaultFluency(\n    new Fluency.Config()\n        .setSenderErrorHandler(new SenderErrorHandler()\n        {\n            @Override\n            public void handle(Throwable e)\n            {\n                // Send a notification\n            }\n        }));\n\n\t:\n\n// If flushing events to Fluentd fails and retried out, the error handler is called back.\nfluency.emit(\"foo.bar\", event);\n```\n\n#### Send requests over SSL/TLS\n\n```java\n// Single Fluentd(localhost:24224)\n//   - Enable SSL/TLS\nFluency fluency = Fluency.defaultFluency(\n\tnew Fluency.Config()\n        .setUseSsl(true));\n```\n\nIf you want to use a custom truststore, specify the JKS file path using `-Djavax.net.ssl.trustStore` (and `-Djavax.net.ssl.trustStorePassword` if needed). You can create a custom truststore like this:\n\n```\n$ keytool -import -file server.crt -alias mytruststore -keystore truststore.jks\n```\n\nFor server side configuration, see https://docs.fluentd.org/v1.0/articles/in_forward#how-to-enable-tls/ssl-encryption .\n\n\n#### Other configurations\n\n```java\n// Multiple Fluentd(localhost:24224, localhost:24225)\n//   - Flush interval = 200ms\n//   - Max retry of sending events = 12\n//   - Use JVM heap memory for buffer pool\nFluency fluency = Fluency.defaultFluency(\n\t\t\tArrays.asList(\n\t\t\t\tnew InetSocketAddress(24224), new InetSocketAddress(24225)),\n\t\t\t\tnew Fluency.Config().\n\t\t\t\t\tsetFlushIntervalMillis(200).\n\t\t\t\t\tsetSenderMaxRetryCount(12).\n\t\t\t\t\tsetJvmHeapBufferMode(true));\n```\n\n### Emit event\n\n```java\nString tag = \"foo_db.bar_tbl\";\nMap\u003cString, Object\u003e event = new HashMap\u003cString, Object\u003e();\nevent.put(\"name\", \"komamitsu\");\nevent.put(\"age\", 42);\nevent.put(\"rate\", 3.14);\nfluency.emit(tag, event);\n```\n\nIf you want to use [EventTime](https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1#eventtime-ext-format) as a timestamp, call `Fluency#emit` with an `EventTime` object in the following way\n\n```java\nint epochSeconds;\nint nanoSeconds;\n    :\nEventTime eventTime = EventTime.fromEpoch(epochSeconds, nanoSeconds);\n\n// You can also create an EventTime object like this\n// EventTime eventTime = EventTime.fromEpochMilli(System.currentTimeMillis());\n\nfluency.emit(tag, eventTime, event);\n```\n\n### Release resources\n\n```java\nfluency.close();\n```\n\n### Know how much Fluency is allocating memory\n\n```java\nLOG.debug(\"Memory size allocated by Fluency is {}\", fluency.getAllocatedBufferSize());\n```\n\n### Know how much Fluench is buffering unsent data in memory\n\n```java\nLOG.debug(\"Unsent data size buffered by Fluency in memory is {}\", fluency.getBufferedDataSize());\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkomamitsu%2Ffluency","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkomamitsu%2Ffluency","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkomamitsu%2Ffluency/lists"}