{"id":13811506,"url":"https://github.com/streamnative/pulsar-beat-output","last_synced_at":"2025-03-23T14:10:50.048Z","repository":{"id":35445054,"uuid":"164271205","full_name":"streamnative/pulsar-beat-output","owner":"streamnative","description":"Elastic Beats Output to Apache Pulsar","archived":false,"fork":false,"pushed_at":"2024-09-20T08:19:53.000Z","size":201808,"stargazers_count":56,"open_issues_count":16,"forks_count":24,"subscribers_count":30,"default_branch":"master","last_synced_at":"2025-03-15T11:06:37.946Z","etag":null,"topics":["apache-pulsar","elasticbeats","filebeat","pulsar"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/streamnative.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":"auditbeat/.gitignore","citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-06T03:15:35.000Z","updated_at":"2024-10-22T22:38:36.000Z","dependencies_parsed_at":"2024-01-15T14:14:58.951Z","dependency_job_id":"a4177560-a51a-4df1-887e-f25f95cfd918","html_url":"https://github.com/streamnative/pulsar-beat-output","commit_stats":{"total_commits":54,"total_committers":16,"mean_commits":3.375,"dds":0.6296296296296297,"last_synced_commit":"6fb3ebe7c1bec7b421370f1d2a0461076c976baa"},"previous_names":["amateurevents/filebeat-ouput-pulsar"],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fpulsar-beat-output","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fpulsar-beat-output/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fpulsar-beat-output/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamnative%2Fpulsar-beat-output/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/streamnative","download_url":"https://codeload.github.com/streamnative/pulsar-beat-output/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245111952,"owners_count":20562512,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-pulsar","elasticbeats","filebeat","pulsar"],"created_at":"2024-08-04T04:00:22.395Z","updated_at":"2025-03-23T14:10:50.017Z","avatar_url":"https://github.com/streamnative.png","language":"Go","funding_links":[],"categories":["Logging"],"sub_categories":[],"readme":"## Beat Output Pulsar\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0)\n[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fstreamnative%2Fpulsar-beat-output.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2Fstreamnative%2Fpulsar-beat-output?ref=badge_shield)\n\nThis is an output implementation of [elastic beats](https://github.com/elastic/beats) for support [Filebeat](https://github.com/elastic/beats/tree/master/filebeat), [Metricbeat](https://github.com/elastic/beats/tree/master/metricbeat), [Functionbeat](https://github.com/elastic/beats/tree/master/x-pack/functionbeat), [Winlogbeat](https://github.com/elastic/beats/tree/master/winlogbeat), [Auditbeat](https://github.com/elastic/beats/tree/master/auditbeat) to [Apache Pulsar](https://github.com/apache/pulsar)\n\n### Compatibility\nThis output is developed and tested using Apache Pulsar Client 2.4.0 and Beats 7.3.1\n\n### Download pulsar-beat-output\n\n```\nmkdir -p $GOPATH/src/github.com/streamnative/\ncd $GOPATH/src/github.com/streamnative/\ngit clone https://github.com/streamnative/pulsar-beat-output\ncd pulsar-beat-output\n```\n\n### Build\n\n#### Build beats\n\n```\ngo build -o filebeat filebeat/filebeat.go\ngo build -o functionbeat functionbeat/functionbeat.go\ngo build -o winlogbeat winlogbeat/winlogbeat.go\ngo build -o packetbeat packetbeat/packetbeat.go\n```\n\n### Usage\n\nIn this section, you can use the sample config file in the directory [./sample/config/], or you can create it as follow steps.\n\n#### example\n#### Add following configuration to beat.yml \n```yml\noutput.pulsar:\n  url: \"pulsar://localhost:6650\"\n  topic: my_topic\n  name: test123\n```\n#### Start filebeat\n```\n./filebeat modules enable system\n./filebeat modules list\n./filebeat -c filebeat.yml -e\n```\n\n### Build and test with docker\n\n#### Requirements\n\n- [Docker](https://docs.docker.com/docker-for-mac/install/)\n\n#### Build Beat images\n\n```\ndocker build -t pulsar-beat .\n```\n\n#### Create network\n\n```\ndocker network create pulsar-beat\n```\n\n#### Start Pulsar service\n```\ndocker run -d -it --network pulsar-beat -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-beat-standalone apachepulsar/pulsar:2.7.0 bin/pulsar standalone\n```\n\n#### Add following configuration to filebeat.yml\n```yml\noutput.pulsar:\n  url: \"pulsar://pulsar-beat-standalone:6650\"\n  topic: my_topic\n  name: test123\n```\n\n#### Start Filebeat\n```\ndocker pull golang:1.17\ndocker run -it --network pulsar-beat --name filebeat golang:1.17 /bin/bash\ngit clone https://github.com/streamnative/pulsar-beat-output\ncd pulsar-beat-output\ngo build -o filebeat filebeat/filebeat.go\nchown -R root:root filebeat.yml test_module/modules.d/system.yml test_module/module/system\ncp test_module/module/system/auth/test/test.log /var/log/messages.log\ncp filebeat/filebeat filebeat.yml test_module\ncd test_module\n./filebeat modules enable system\n./filebeat -c filebeat.yml -e\n```\n\n#### New open a window for consumer message\n```\ndocker cp pulsar-client.py pulsar-beat-standalone:/pulsar\ndocker exec -it pulsar-beat-standalone /bin/bash\npython pulsar-client.py\n```\nNow you can see the information collected from filebeat.\n\n### Configurations\n\n#### Client\n| Name              | Description                                                                                                                                       | Default                 |\n|-------------------|---------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------|\n| url               | Configure the service URL for the Pulsar service                                                                                                  | pulsar://localhost:6650 |\n| certificate_path  | path of tls cert file                                                                                                                             | \"\"                      |\n| private_key_path  | path of tls key file                                                                                                                              | \"\"                      |\n| use_tls           | Whether to turn on TLS, if to start, use protocol pulsar+ssl                                                                                      | false                   |\n| token             | Access token information of cluster                                                                                                               | \"\"                      |\n| token_file_path   | The file path where token is saved                                                                                                                | \"\"                      |\n| log_level         | Setting the log level, available options(panic, fatal, error, warn, info, debug, trace)                                                           | info                    |\n| oauth2.enabled    | Enabled or disabled oauth2 authentication                                                                                                         | false                   |\n| oauth2.clientId   | client ID                                                                                                                                         | \"\"                      |\n| oauth2.issuerUrl  | URL of the authentication provider which allows the Pulsar client to obtain an access token                                                       | \"\"                      |\n| oauth2.privateKey | URL of a JSON credentials file                                                                                                                    | \"\"                      |\n| oauth2.audience   | The audience value is either the application (`Client ID`) for an ID Token or the API that is being called (`API Identifier`) for an Access Token | \"\"                      |\n| oauth2.scope      | Scope is a mechanism in OAuth 2.0 to limit an application's access to a user's account                                                            | \"\"                      |\n\n\n\n#### Producer\n| Name                       | Description                                                                                                                                                                       | Default |\n|----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|\n| topic                      | Specify the topic this producer will be publishing on. You can set the topic dynamically by using a format string to access any event field. For example `%{[fields.log_topic]}`. | \"\"      |\n| partition_key              | Specify the message key. You can set the message key dynamically by using a format string to access any event field. For example `%{[fields.partition_key]}`                      | \"\"      |\n| name                       | Specify a name for the producer                                                                                                                                                   | \"\"      |\n| send_timeout               | Set the send timeout                                                                                                                                                              | 30s     |\n| block_if_queue_full        | Set whether the send and sendAsync operations should block when the outgoing message queue is full.                                                                               | false   |\n| batching_max_messages      | maximum number of messages in a batch                                                                                                                                             | 1000    |\n| batching_max_publish_delay | the batch delay                                                                                                                                                                   | 1ms     |\n| message_routing_mode       | the message routing mode, SinglePartition,RoundRobinPartition, CustomPartition(0,1,2)                                                                                             | 1       |\n| hashing_schema             | JavaStringHash,Murmur3_32Hash(0,1)                                                                                                                                                | 0       |\n| compression_type           | NONE,LZ4,ZLIB,ZSTD(0,1,2,3)                                                                                                                                                       | 0       |\n| max_cache_producers        | Specify the maximun cache(lru) producers of dynamic topic.                                                                                                                        | 8       |\n\n### FAQ\n\n#### case-insensitive import collision: \"github.com/datadog/zstd\" and \"github.com/DataDog/zstd\"\n\n```\n/root/go/pkg/mod/github.com/apache/pulsar-client-go@v0.3.0/pulsar/internal/compression/zstd_cgo.go:27:2: case-insensitive import collision: \"github.com/datadog/zstd\" and \"github.com/DataDog/zstd\"\n```\n\nReplace zstd_cgo.go file\n```\ncp zstd_cgo.go /root/go/pkg/mod/github.com/apache/pulsar-client-go@v0.3.0/pulsar/internal/compression/zstd_cgo.go\n```\n\n#### Install Pulsar Go Client\nReference https://pulsar.apache.org/docs/en/client-libraries-go/ .\n\nIf you encounter problems with dynamic libraries，please reference:https://pulsar.apache.org/docs/en/client-libraries-cpp/.\n\n#### Build Packetbeat\nReference https://github.com/elastic/beats/issues/11054.\n\n#### Build auditbeat.go\n```\nvendor/github.com/elastic/beats/x-pack/auditbeat/module/system/package/rpm_linux.go:23:24: fatal error: rpm/rpmlib.h: No such file or directory\n```\n\n```\naapt-get install librpm-dev\n```\n\n#### Start beat\n```\nExiting: error loading config file: config file (\"filebeat.yml\") must be owned by the user identifier (uid=0) or root\n```\n```\nchown -R root:root filebeat.yml\n```\n\n## License\n[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fstreamnative%2Fpulsar-beat-output.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fstreamnative%2Fpulsar-beat-output?ref=badge_large)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamnative%2Fpulsar-beat-output","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstreamnative%2Fpulsar-beat-output","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamnative%2Fpulsar-beat-output/lists"}