{"id":20350787,"url":"https://github.com/hartfordfive/logshipper-benchmark","last_synced_at":"2026-06-05T08:31:31.456Z","repository":{"id":73197148,"uuid":"135342884","full_name":"hartfordfive/logshipper-benchmark","owner":"hartfordfive","description":"An application to run benchmarks on a variety of log shippers.","archived":false,"fork":false,"pushed_at":"2018-06-06T12:41:10.000Z","size":2448,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-14T23:57:48.383Z","etag":null,"topics":["filebeat","fluentbit","go","golang","logging","logstash","metrics","metrics-gathering","rsyslogd"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hartfordfive.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-29T19:22:55.000Z","updated_at":"2018-06-15T05:59:53.000Z","dependencies_parsed_at":null,"dependency_job_id":"e57a4124-b607-462f-9bd0-73b9fd02b383","html_url":"https://github.com/hartfordfive/logshipper-benchmark","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hartfordfive%2Flogshipper-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hartfordfive%2Flogshipper-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hartfordfive%2Flogshipper-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hartfordfive%2Flogshipper-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hartfordfive","download_url":"https://codeload.github.com/hartfordfive/logshipper-benchmark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241880272,"owners_count":20035927,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["filebeat","fluentbit","go","golang","logging","logstash","metrics","metrics-gathering","rsyslogd"],"created_at":"2024-11-14T22:33:50.403Z","updated_at":"2026-06-05T08:31:31.428Z","avatar_url":"https://github.com/hartfordfive.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Log Shipper Benchmarks\n\n## Description\n\nThe role of this application is to provide the ability to easily benchmark the file input capabilities of various log shipper clients to a Kafka output destination. \nMetrics from the log shippers are collected via metricbeat.\n\n\n## Requirments/Dependencies\n\nGo:\n----\n- Must be running Go \u003e= 1.8.x, although was built with 1.10.0\n- Ensure `GOPATH` is set to `$HOME/go` and that the directory exists\n- Ensure `$HOME/go/bin` is part of you `PATH` environment variable\n- Require go vendoring library: `github.com/kardianos/govendor` (`go get github.com/kardianos/govendor`)\n\nFor currently included modules:\n----\n- [Fluent Bit](https://fluentbit.io/documentation/current/installation/redhat_centos.html): v0.13.0\n- [Filebeat](https://www.elastic.co/guide/en/beats/filebeat/6.1/filebeat-installation.html) : v6.1.1\n- [Logstash](https://www.elastic.co/guide/en/logstash/6.1/installing-logstash.html) : v6.1.1\n- [Rsyslogd](https://www.rsyslog.com/rhelcentos-rpms/) : v8.34.0\n\nGo Packages:\n----\n- github.com/kardianos/govendor\n- github.com/c9s/goprocinfo/linux \n- github.com/Pallinder/go-randomdata\n- github.com/mitchellh/go-ps\n\nMetrics Collection:\n----\n- [Metricbeat](https://www.elastic.co/guide/en/beats/metricbeat/6.1/metricbeat-installation.html): v6.1.1\n\n## Building from source\n\n```\nmake deps\nmake clean\nmake all\n```\n\n\n## Configuration\n\nThe config, which is in JSON format, should contain the following fields:\n- `additional_metricbeat_fields` : An object consisting of additional key/value properties to add the the metricbeat data. (Type: map[string]string, Default: \u003cempty\u003e)\n- `custom_log_entry` : If set, the this specific log entry will be written to the files instead of a randomly generated one. (Type: string, Default: \u003cempty\u003e)\n- `enable_random` : If set to true, the application will randomly choose a line size and wait time between writes. (Type: boolean, Default: false)\n- `kafka_broker_list` : List of Kafka broker hostnames (HOST:PORT) to use in the log shippers. Currently only this output destination is supported. (Type: []string, Default)\n- `log_files_base_dir` : Location where the sample log files will be created (Type: string, Default: \u003cempty\u003e)\n- `log_line_size` : The size (character length) of the log entry to be randomly generated. (Type: int, Default: 50)\n- `log_shipper_bin_path` : The path to the log shipper binary. (Type: string, Default: \u003cempty\u003e)\n- `log_shipper_flags` :  The flags to use when executing the log shipper binary. (Type: string, Default: \u003cempty\u003e)\n- `log_shipper_name` : The name of the log shipper. (Type: string, Default: \u003cempty\u003e)\n- `log_shipper_process_name` : The running process name of the log shipper. (Type: string, Default: \u003cempty\u003e)\n- `max_procs` : The max number of processors this benchmarking app should use. (Type: int, Default: \u003cempty\u003e)\n- `metrics_dir` : The directory in which the collected process metrics will be stored. (Type: string, Default: \u003cempty\u003e)\n- `module_dir` : The directory in which the `.so` shipper module is found. (Type: string, Default: \u003cempty\u003e)\n- `module_name` : The filename of the module excluding the `.so` extension. (Type: string, Default: \u003cempty\u003e)\n- `num_active_log_files` : The number of active log files that will be written to concurrently/in-parallel. (Type: int, Default: 10)\n- `random_line_size` : The MIN,MAX range for the length of the line in characters. (Type []int, Default: \u003cempty\u003e)\n- `random_write_wait` : The MIN,MAX range for period (in milliseconds) bewteen writes to each individual log files. (Type []int, Default: \u003cempty\u003e)\n- `total_run_time_seconds` : The total time (in seconds) to run the benchmark. (Type int, Default: \u003cempty\u003e)\n- `working_dir` :  The working directory in which the module will be running. (Type: string, Default: \u003cempty\u003e)\n- `write_wait_period_ms` : The period (in milliseconds) bewteen writes to the each individual log files.  (Type int, Default: \u003cempty\u003e)\n\nSamples can be found in the [_sample_configs](_sample_configs/) directory.\n\n## Implementing additional shippers\n\nThe individual shippers work with a plugin based system.  In order to benchmark a new shipper, you must create a plugin that respects the following interface\n```\ntype Shipper interface {\n        Name()\n        Run(binPath string, cmdArgs []string, workingDir string, filesToMonitor []string, kafkaBrokers []string, filebeatExec chan *exec.Cmd, terminate chan bool, wg *sync.WaitGroup)\n        CleanupFiles()\n        BuildConfig(confDestPath string, filesToMonitor []string, kafkTopicName string, kafkaBrokersList []string)\n        GetVersion() string\n}\n```\n\nTo compile the plugin manually, run the following command:\n```\ngo build -a -v -buildmode=plugin -o output/path/to/plugin.so source/path/to/plugin.go\n```\n\n\n## Running the benchmarks:\n\nThe benchmarks for a given log shipper are executed as follows:\n```\n./logshipper-benchmark [PATH_TO_CONFIG]\n```\n\nTo terminate the benchmark, simply hit `Ctrl+C`.  If the process is backgrounded, to initial a clean shutdown, you must kill the `logshipper-benchmark` with a `SIGINT` or `SIGTERM` signal.\nFor example:\n```\nkill -INT [PID]\n```\n\n## Considerations\n\nThis application was developed quickly to have the ability to efficiently create benchmarks for a variety of log shippers.  Having said that, it's certain this code isn't optimal and may likely contain bugs.  If you find any bugs/performance improvements to this application, feel free to create a bug report or issue a PR.  If you have any additional log shippers you'd like to test, or even different version of them, a PR for thew new log shipper would be much appreciated.\n\n\n## Author\n\nAlain Lefebvre \u003chartfordfive@gmail.com\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhartfordfive%2Flogshipper-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhartfordfive%2Flogshipper-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhartfordfive%2Flogshipper-benchmark/lists"}