{"id":13608149,"url":"https://github.com/elastic/logstash-forwarder","last_synced_at":"2025-12-16T20:20:11.840Z","repository":{"id":4141575,"uuid":"5254650","full_name":"elastic/logstash-forwarder","owner":"elastic","description":"An experiment to cut logs in preparation for processing elsewhere. Replaced by Filebeat: https://github.com/elastic/beats/tree/master/filebeat","archived":false,"fork":false,"pushed_at":"2024-08-08T07:17:13.000Z","size":718,"stargazers_count":1786,"open_issues_count":33,"forks_count":411,"subscribers_count":448,"default_branch":"master","last_synced_at":"2025-04-03T14:08:15.601Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elastic.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2012-08-01T03:44:25.000Z","updated_at":"2025-03-05T08:57:54.000Z","dependencies_parsed_at":"2024-11-19T21:02:05.636Z","dependency_job_id":"188e1393-ff4f-4ede-9a00-903359fb7ef8","html_url":"https://github.com/elastic/logstash-forwarder","commit_stats":{"total_commits":601,"total_committers":36,"mean_commits":"16.694444444444443","dds":0.1314475873544093,"last_synced_commit":"553e6efec0528a27a8e7cda37c545c19c2cea7eb"},"previous_names":["jordansissel/lumberjack","elasticsearch/logstash-forwarder"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Flogstash-forwarder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Flogstash-forwarder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Flogstash-forwarder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Flogstash-forwarder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elastic","download_url":"https://codeload.github.com/elastic/logstash-forwarder/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248388409,"owners_count":21095385,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T19:01:24.694Z","updated_at":"2025-12-16T20:20:11.779Z","avatar_url":"https://github.com/elastic.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# logstash-forwarder\n\n# THIS PROJECT IS REPLACED BY [FILEBEAT](https://github.com/elastic/beats/tree/master/filebeat)\n\nThe [filebeat](https://github.com/elastic/beats/tree/master/filebeat) project replaces logstash-forwarder. Please use that instead.\n\nNo further development will occur on this project. Major bug fixes or security fixes may be worked on through 2016, at which point this repository and its project will be abandoned. The replacement is [filebeat](https://github.com/elastic/beats/tree/master/filebeat) which receives new features and fixes frequently. :)\n\nHappy logging!\n\n\n---\n\n♫ I'm a lumberjack and I'm ok! I sleep when idle, then I ship logs all day! I parse your logs, I eat the JVM agent for lunch! ♫\n\n(This project was recently renamed from 'lumberjack' to 'logstash-forwarder' to\nmake its intended use clear. The 'lumberjack' name now remains as the network protocol, and 'logstash-forwarder' is the name of the program. It's still the same lovely log forwarding program you love.)\n\n## Questions and support\n\nIf you have questions and cannot find answers, please join the #logstash irc\nchannel on freenode irc or ask on the logstash-users@googlegroups.com mailing\nlist.\n\n## What is this?\n\nA tool to collect logs locally in preparation for processing elsewhere!\n\n### Resource Usage Concerns\n\nPerceived Problems: Some users view logstash releases as \"large\" or have a generalized fear of Java.\n\nActual Problems: Logstash, for right now, runs with a footprint that is not\nfriendly to underprovisioned systems such as EC2 micro instances; on other\nsystems it is fine. This project will exist until that is resolved.\n\n### Transport Problems\n\nFew log transport mechanisms provide security, low latency, and reliability.\n\nThe lumberjack protocol used by this project exists to provide a network\nprotocol for transmission that is secure, low latency, low resource usage, and\nreliable.\n\n## Configuring\n\nlogstash-forwarder is configured with a json file you specify with the -config flag:\n\n`logstash-forwarder -config yourstuff.json`\n\nHere's a sample, with comments in-line to describe the settings. Comments are\ninvalid in JSON, but logstash-forwarder will strip them out for you if they're\nthe only thing on the line:\n\n    {\n      # The network section covers network configuration :)\n      \"network\": {\n        # A list of downstream servers listening for our messages.\n        # logstash-forwarder will pick one at random and only switch if\n        # the selected one appears to be dead or unresponsive\n        \"servers\": [ \"localhost:5043\" ],\n\n        # The path to your client ssl certificate (optional)\n        \"ssl certificate\": \"./logstash-forwarder.crt\",\n        # The path to your client ssl key (optional)\n        \"ssl key\": \"./logstash-forwarder.key\",\n\n        # The path to your trusted ssl CA file. This is used\n        # to authenticate your downstream server.\n        \"ssl ca\": \"./logstash-forwarder.crt\",\n\n        # Network timeout in seconds. This is most important for\n        # logstash-forwarder determining whether to stop waiting for an\n        # acknowledgement from the downstream server. If an timeout is reached,\n        # logstash-forwarder will assume the connection or server is bad and\n        # will connect to a server chosen at random from the servers list.\n        \"timeout\": 15\n      },\n\n      # The list of files configurations\n      \"files\": [\n        # An array of hashes. Each hash tells what paths to watch and\n        # what fields to annotate on events from those paths.\n        {\n          \"paths\": [\n            # single paths are fine\n            \"/var/log/messages\",\n            # globs are fine too, they will be periodically evaluated\n            # to see if any new files match the wildcard.\n            \"/var/log/*.log\"\n          ],\n\n          # A dictionary of fields to annotate on each event.\n          \"fields\": { \"type\": \"syslog\" }\n        }, {\n          # A path of \"-\" means stdin.\n          \"paths\": [ \"-\" ],\n          \"fields\": { \"type\": \"stdin\" }\n        }, {\n          \"paths\": [\n            \"/var/log/apache/httpd-*.log\"\n          ],\n          \"fields\": { \"type\": \"apache\" }\n        }\n      ]\n    }\n\nAny part of config can use environment variables as `$VAR` or `${VAR}`. They will be evaluated before processing JSON, allowing to pass any structure.\n\nYou can also read an entire directory of JSON configs by specifying a directory instead of a file with the `-config` option.\n\n# IMPORTANT TLS/SSL CERTIFICATE NOTES\n\nThis program will reject SSL/TLS certificates which have a subject which does not match the `servers` value, for any given connection. For example, if you have `\"servers\": [ \"foobar:12345\" ]` then the 'foobar' server MUST use a certificate with subject or subject-alternative that includes `CN=foobar`. Wildcards are supported also for things like `CN=*.example.com`. If you use an IP address, such as `\"servers\": [ \"1.2.3.4:12345\" ]`, your ssl certificate MUST use an IP SAN with value \"1.2.3.4\". If you do not, the TLS handshake will FAIL and the lumberjack connection will close due to trust problems.\n\nCreating a correct SSL/TLS infrastructure is outside the scope of this document. \n\nAs a very poor example (largely due unpredictability in your system's defaults for openssl), you can try the following command as an example for creating a self-signed certificate/key pair for use with a server named \"logstash.example.com\":\n\n```\nopenssl req -x509  -batch -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.crt -subj /CN=logstash.example.com\n```\n\nThe above example will create an SSL cert for the host 'logstash.example.com'. You cannot use `/CN=1.2.3.4` to create an SSL certificate for an IP address. In order to do a certificate with an IP address, you must create a certificate with an \"IP Subject Alternative\" or often called \"IP SAN\". Creating a certificate with an IP SAN is difficult and annoying, so I highly recommend you use hostnames only. If you have no DNS available to you, it is still often easier to set hostnames in /etc/hosts than it is to create a certificate with an IP SAN.\n\nlogstash-forwarder needs the `.crt` file, and logstash will need both `.key` and `.crt` files.\n\nAgain, creating a correct SSL/TLS certificate authority or generally doing certificate management is outside the scope of this document. \n\nIf you see an error like this:\n\n```\nx509: cannot validate certificate for 1.2.3.4 because it doesn't contain any IP SANs\n```\n\nIt means you are telling logstash-forwarder to connect to a host by IP address,\nand therefore you must include an IP SAN in your certificate. Generating an SSL\ncertificate with an IP SAN is quite annoying, so I *HIGHLY* recommend you use\ndns names and set the CN in your cert to your dns name.\n\n### Goals\n\n* Minimize resource usage where possible (CPU, memory, network).\n* Secure transmission of logs.\n* Configurable event data.\n* Easy to deploy with minimal moving parts.\n* Simple inputs only:\n  * Follows files and respects rename/truncation conditions.\n  * Accepts `STDIN`, useful for things like `varnishlog | logstash-forwarder...`.\n\n## Building it\n\n1. Install [go](http://golang.org/doc/install)\n\n2. Compile logstash-forwarder\n\nNote: Do not use gccgo for this project. If you don't know what that means,\nyou're probably OK to ignore this.\n\n        git clone git://github.com/elasticsearch/logstash-forwarder.git\n        cd logstash-forwarder\n        go build -o logstash-forwarder\n\ngccgo note: Using gccgo is not recommended because it produces a binary with a\nruntime dependency on libgo. With the normal go compiler, this dependency\ndoesn't exist and, as a result, makes it easier to deploy. You can check if you\nare using gccgo by running `go version` and if it outputs something like `go\nversion xgcc`, you're probably not using gccgo, and I recommend you don't.\nYou can also check the resulting binary by doing `ldd ./logstash-forwarder` and\nseeing if `libgo` appears in the output; if it appears, then you are using gccgo,\nand I recommend you don't.\n\n## Packaging it (optional)\n\nYou can make native packages of logstash-forwarder.\n\nTo do this, a recent version of Ruby is required. At least version 2.0.0 or\nnewer. If you are using your OS distribution's version of Ruby, especially on\nRed Hat- or Debian-derived systems (Ubuntu, CentOS, etc), you will need to install\nruby and whatever the \"ruby development\" package is called for your system.\nOn Red Hat systems, you probably want `yum install ruby-devel`. On Debian systems,\nyou probably want `apt-get install ruby-dev`.\n\nPrerequisite steps to prepare ruby to build your packages are:\n\n```\ngem install bundler\nbundle install\n```\n\nThe `bundle install` will install any Ruby library dependencies that are used\nin building packages.\n\nNow build an rpm:\n\n        make rpm\n\nOr:\n\n        make deb\n\n## Installing it (via packages only)\n\nIf you don't use rpm or deb make targets as above, you can skip this section.\n\nPackages install to `/opt/logstash-forwarder`.\n\nThere are no run-time dependencies.\n\n## Running it\n\nGenerally:\n\n    logstash-forwarder -config logstash-forwarder.conf\n\nSee `logstash-forwarder -help` for all the flags. The `-config` option is required and logstash-forwrder will not run without it.\n\nThe config file is documented further up in this file.\n\nAnd also note that logstash-forwarder runs quietly when all is a-ok. If you want informational feedback, use the `verbose` flag to enable log emits to stdout.\n\nFatal errors are always sent to stderr regardless of the `-quiet` command-line option and process exits with a non-zero status.\n\n### Key points\n\n* You'll need an SSL CA to verify the server (host) with.\n* You can specify custom fields for each set of paths in the config file. Any\n  number of these may be specified. I use them to set fields like `type` and\n  other custom attributes relevant to each log.\n\n### Generating an ssl certificate\n\nLogstash supports all certificates, including self-signed certificates. To generate a certificate, you can run the following command:\n\n    $ openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout logstash-forwarder.key -out logstash-forwarder.crt -days 365\n\nThis will generate a key at `logstash-forwarder.key` and the 1-year valid certificate at `logstash-forwarder.crt`. Both the server that is running logstash-forwarder as well as the logstash instances receiving logs will require these files on disk to verify the authenticity of messages. \n\nRecommended file locations:\n\n- certificates: `/etc/pki/tls/certs/logstash-forwarder/`\n- keys: `/etc/pki/tls/private/logstash-forwarder/`\n\n## Use with logstash\n\nIn logstash, you'll want to use the [lumberjack](http://logstash.net/docs/latest/inputs/lumberjack) input, something like:\n\n    input {\n      lumberjack {\n        # The port to listen on\n        port =\u003e 12345\n\n        # The paths to your ssl cert and key\n        ssl_certificate =\u003e \"path/to/ssl.crt\"\n        ssl_key =\u003e \"path/to/ssl.key\"\n\n        # Set this to whatever you want.\n        type =\u003e \"somelogs\"\n      }\n    }\n\n## Implementation details\n\nBelow is valid as of 2012/09/19\n\n### Minimize resource usage\n\n* Sets small resource limits (memory, open files) on start up based on the\n  number of files being watched.\n* CPU: sleeps when there is nothing to do.\n* Network/CPU: sleeps if there is a network failure.\n* Network: uses zlib for compression.\n\n### Secure transmission\n\n* Uses OpenSSL to verify the server certificates (so you know who you\n  are sending to).\n* Uses OpenSSL to transport logs.\n\n### Configurable event data\n\n* The protocol supports sending a `string:string` map.\n\n## License\n\nSee LICENSE file.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felastic%2Flogstash-forwarder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felastic%2Flogstash-forwarder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felastic%2Flogstash-forwarder/lists"}