{"id":17931123,"url":"https://github.com/rdner/load-log-vs-filestream","last_synced_at":"2025-06-10T10:39:27.851Z","repository":{"id":119127727,"uuid":"610241486","full_name":"rdner/load-log-vs-filestream","owner":"rdner","description":"Performance test suite for `log` input and `filestream` input in Filebeat","archived":false,"fork":false,"pushed_at":"2023-03-06T13:50:07.000Z","size":6,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-03T10:30:58.871Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rdner.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-06T11:34:38.000Z","updated_at":"2023-03-06T15:02:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"fad60e37-63f0-46ce-a160-8f55650accab","html_url":"https://github.com/rdner/load-log-vs-filestream","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rdner%2Fload-log-vs-filestream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rdner%2Fload-log-vs-filestream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rdner%2Fload-log-vs-filestream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rdner%2Fload-log-vs-filestream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rdner","download_url":"https://codeload.github.com/rdner/load-log-vs-filestream/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rdner%2Fload-log-vs-filestream/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259057178,"owners_count":22799106,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-28T21:19:45.731Z","updated_at":"2025-06-10T10:39:27.756Z","avatar_url":"https://github.com/rdner.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Performance Testing of `log` input vs `filestream` input\n\n## Dependencies\n\nIt requires [`jq`](https://stedolan.github.io/jq/) and [Go](https://go.dev) to be installed.\n\n## Prepare the data set\n\nIn order to start using the test suite, first you need to generate the data set using the `./logs/generate` script:\n\n```sh\ncd ./logs\n./generate 100 10000\n```\n\nIn this example it's 100 files, 10,000 lines per file. Keep in mind that the script must be run in the `./logs` directory.\n\n## Run and get results\n\nBefore the `run` script you always need to set the `FILEBEAT_CMD` environment variable. This can be a path to a Filebeat binary or simply a command if you have it globally installed. It's `filebeat` by default.\n\nIn order to run the suite:\n\n1. Run `./clean` – deletes the registry and the output files for both `log` and `filestream` inputs for a clean start. Use again when necessary.\n2. Run `FILEBEAT_CMD=../bin/filebeat ./run log` – runs the test for the `log` input\n3. Observe the CPU load, once it's down press `CTRL+C` to stop Filebeat. Unfortunately, this is the only way for now. The `--once` flag on Filebeat is [unstable](https://github.com/elastic/beats/issues/33718).\n4. You'll see the test results on your terminal as a JSON object and it gets saved as `./result-log.json` as well\n5. Run `FILEBEAT_CMD=../bin/filebeat ./run fs` – runs the test for the `filestream` input.\n6. Observe the CPU load, once it's down press `CTRL+C` to stop Filebeat.\n7. You'll see the test results on your terminal as a JSON object and it gets saved as `./result-fs.json` as well\n8. Run `go run main.go ./result-log.json ./result-fs.json`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frdner%2Fload-log-vs-filestream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frdner%2Fload-log-vs-filestream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frdner%2Fload-log-vs-filestream/lists"}