{"id":18929953,"url":"https://github.com/redhatcre/beatshipper","last_synced_at":"2025-06-10T16:32:14.780Z","repository":{"id":60974441,"uuid":"546301681","full_name":"RedHatCRE/beatshipper","owner":"RedHatCRE","description":null,"archived":false,"fork":false,"pushed_at":"2023-01-08T17:21:19.000Z","size":61,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-05-25T02:44:32.007Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RedHatCRE.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-10-05T21:44:03.000Z","updated_at":"2023-05-24T14:40:17.000Z","dependencies_parsed_at":"2023-02-08T06:45:16.766Z","dependency_job_id":null,"html_url":"https://github.com/RedHatCRE/beatshipper","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedHatCRE%2Fbeatshipper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedHatCRE%2Fbeatshipper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedHatCRE%2Fbeatshipper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedHatCRE%2Fbeatshipper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RedHatCRE","download_url":"https://codeload.github.com/RedHatCRE/beatshipper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedHatCRE%2Fbeatshipper/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259110031,"owners_count":22806638,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T11:35:57.296Z","updated_at":"2025-06-10T16:32:14.734Z","avatar_url":"https://github.com/RedHatCRE.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# beatshipper\n\nSince there’s no way to send GNU zip files through filebeat, this service will be responsible for checking if there are new .gz files based on a path that we’ll explode using globbing, decompress them, send them using the filebeat service with the provided configuration and store them in a local file registry.\n\nIt can also be used for non-compressed files.\n\n[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n\n## Requirements\n\n\u003e 🔔 Go Version 1.18 or higher\n\n## Installation\n\n- Move to the cloned repository and install with:\n\n```\n$ go install\n```\n\n```\n$ beatshipper -h\nSends data based on paths that will be exploded using GLOB\nwith the possibility of passing GNU Zip files also.\n\nUsage:\n  beatshipper [command]\n\nAvailable Commands:\n  completion  Generate the autocompletion script for the specified shell\n  help        Help about any command\n  send        Send the message as a beat\n\nFlags:\n  -h, --help      help for beatshipper\n  -v, --version   version for beatshipper\n\nUse \"beatshipper [command] --help\" for more information about a command.\n\n$ beatshipper --version\nbeatshipper version 0.0.1\n```\n\n## Configuration\n\nWe should create the configuration in any of the following both locations:\n\n- `/etc/beatshipper/beatshipper-conf.yml`\n- `$HOME/.config/beatshipper-conf.yml`\n\nAn example is located in the root of the project. We should add the following fields:\n\n- `host`\n- `port`\n- `path` array of paths that will be exploded using `GLOB`\n- `registry`: the name of the file where we'll store the processed files.\n- `recheck` time of recheck (re-process) if there are new pending files to send\n- `logsource` very useful in logstash to handle conditions based on the name of the source\n\n# Configuration through SystemD\n\nWe have created two SystemD files:\n\n- `service` file: it specifies the binary that will execute the service\n- `timer` file: we are gonna execute the service as timer.\n\nBy default the timer will launch the service in one shoot type every 30 minutes (`*:0/30`). If we wanna change the time we should change the `OnCalendar` directime in the `timer` file.\n\nThese files are being coppied to the following directory `/lib/systemd/system/` if we install the generated RPM package.\n\nWe can enable and start the `timer` with the following commands:\n\n```\n$ systemctl enable beatshipper.timer\n$ systemctl start beatshipper.timer\n```\n\nWe can check if the timer has been activated:\n\n```\n$ systemctl list-timers | grep beatshipper -B1\nNEXT                        LEFT          LAST                        PASSED       UNIT                           ACTIVATES\nMon 2022-12-26 16:00:00 CET 27min left    Mon 2022-12-26 15:30:32 CET 2min 12s ago beatshipper.timer              beatshipper.service\n\n$ journalctl -u beatshipper.timer\ndic 26 15:30:32 user systemd[1]: Started Beatshipper execution schedule.\n```\n\n## Usage:\n\nIf we have the right configuration, we can start the program using the following command - It will read the configuration, start checking if there are pending packages that aren't in the registry and send them to the server that is listening using `beats`- e.g. a logstash instance with the `beats` module -:\n\n```\n$ beatshipper send\n2022/11/18 21:04:53 shipper.go:77: Conexion successful with: localhost:5044\n2022/11/18 21:04:53 shipper.go:82: Processing: /jenkinst-test\njenkinst-test/jenkins-logs/rcj/test/sub-test/sub-sub-test/var/log/extra/rpm-list.txt.gz\n2022/11/18 21:04:53 shipper.go:82: Processing: /jenkinst-test\njenkinst-test/jenkins-logs/rcj/test/sub-test/sub-sub-test10/var/log/extra/rpm-list.txt.gz\n2022/11/18 21:04:53 shipper.go:118: Chunk slice into: 5 slices\n2022/11/18 21:04:53 shipper.go:96: Sending batch of data...\n2022/11/18 21:04:53 shipper.go:96: Sending batch of data...\n2022/11/18 21:04:53 shipper.go:96: Sending batch of data...\n2022/11/18 21:04:53 shipper.go:101: Conexion closed\n2022/11/18 21:04:53 registry.go:75: Added to registry: /jenkinst-test\njenkinst-test/jenkins-logs/rcj/test/sub-test/sub-sub-test/var/log/extra/rpm-list.txt.gz\n2022/11/18 21:04:53 registry.go:75: Added to registry: /jenkinst-test\njenkinst-test/jenkins-logs/rcj/test/sub-test/sub-sub-test10/var/log/extra/rpm-list.txt.gz\n```\n\n## Registry\n\nFiles will be stored in the following way:\n\n```\n$ jq \u003c shipper_registry.json\n{\n  \"ParsedFiles\": [\n    {\n      \"name\": \"/jenkinst-test/jenkins-logs/rcj/test/sub-test/sub-sub-test/var/log/extra/rpm-list.txt.gz\",\n      \"date\": \"2022-11-18T21:04:53.773777134+01:00\"\n    },\n    {\n      \"name\": \"/jenkinst-test/jenkins-logs/rcj/test/sub-test/sub-sub-test10/var/log/extra/rpm-list.txt.gz\",\n      \"date\": \"2022-11-18T21:04:53.773781056+01:00\"\n    },\n  ]\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredhatcre%2Fbeatshipper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fredhatcre%2Fbeatshipper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredhatcre%2Fbeatshipper/lists"}