{"id":20226867,"url":"https://github.com/qihoo360/qbusbridge","last_synced_at":"2025-04-05T18:09:15.274Z","repository":{"id":45237387,"uuid":"152210224","full_name":"Qihoo360/qbusbridge","owner":"Qihoo360","description":"The Apache Kafka Client SDK","archived":false,"fork":false,"pushed_at":"2024-05-07T11:47:45.000Z","size":626,"stargazers_count":290,"open_issues_count":2,"forks_count":73,"subscribers_count":29,"default_branch":"master","last_synced_at":"2025-03-29T17:08:32.062Z","etag":null,"topics":["consumer","cplusplus","golang","kafka","kafka-consumer","kafka-producer","kafkabridge","kafkaclient","kafkasdk","php","producer","python"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Qihoo360.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-09T07:50:23.000Z","updated_at":"2025-01-16T10:27:49.000Z","dependencies_parsed_at":"2024-06-18T20:10:17.685Z","dependency_job_id":"ba51d68a-03da-4cee-afe3-189a202d656e","html_url":"https://github.com/Qihoo360/qbusbridge","commit_stats":null,"previous_names":["qihoo360/kafkabridge"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Qihoo360%2Fqbusbridge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Qihoo360%2Fqbusbridge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Qihoo360%2Fqbusbridge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Qihoo360%2Fqbusbridge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Qihoo360","download_url":"https://codeload.github.com/Qihoo360/qbusbridge/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247378149,"owners_count":20929297,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consumer","cplusplus","golang","kafka","kafka-consumer","kafka-producer","kafkabridge","kafkaclient","kafkasdk","php","producer","python"],"created_at":"2024-11-14T07:20:55.836Z","updated_at":"2025-04-05T18:09:15.247Z","avatar_url":"https://github.com/Qihoo360.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n## Introduction [中文](https://github.com/Qihoo360/kafkabridge/blob/master/README_ZH.md)\n* Qbusbridge is a client SDK for pub-sub messaging systems. Currently it supports:\n\n  * [Apache Kafka](http://kafka.apache.org/)\n  * [Apache Pulsar](https://pulsar.apache.org/)\n\n  User could switch to any pub-sub messaging system by changing the configuration file. The default config is accessing Kafka, if you want to change it to Pulsar, change the config to:\n\n  ```ini\n  mq.type=pulsar\n  # Other configs for pulsar...\n  ```\n\n  See [config](config/) for more details.\n\n  \u003e  TODO: English config docs is missed currently.\n\n* Qbusbridge-Kafka is based on the [librdkafka](https://github.com/edenhill/librdkafka) under the hook. A mass of details related to how to use has been hidden, that making QBus more simple and easy-to-use than [librdkafka](https://github.com/edenhill/librdkafka). For producing and consuming messages, the only thing need the users to do is to invoke a few APIs, for these  they don't need to understand too much about Kafka.\n\n* The reliability of messages producing, that is may be the biggest concerns of the users, has been considerably improved.\n\n## Features\n* Multiple programming languages are supported, includes C++, PHP, Python, Golong, with very consistent APIs.\n* Few interfaces, so it is easy to use.\n* For advanced users, adapting [librdkafka](https://github.com/edenhill/librdkafka)'s configration by profiles is also supported.\n* In the case of writing data not by keys, the SDK will do the best to guarantee the messages being written successfully .\n* Two writing modes, synchronous and asynchronous, are supported.\n* As for messages consuming, the offset could be submited automatically, or by manual  configurating.\n* For the case of using php-fpm, the connection is keeping-alived for reproduce messages uninterruptedly, saving the cost caused by recreating connections.\n\n## Compiling\n\nEnsure your system has g++ (\u003e= 4.8.5), boost (\u003e= 1.41), cmake (\u003e= 3.1) and swig (\u003e= 3.0.12) installed.\n\nIn addition, qbus SDK is linking libstdc++ statically, so you must ensure that `libstdc++.a` exists. For CentOS users, run:\n\n```bash\nsudo yum install -y glibc-static libstdc++-static\n```\n\n#### git clone:\n\n```shell\ngit clone --recursive https://github.com/ntt360/qbusbridge.git\n```\n\n\n#### SASL Support\n\nIf you need `librdkafa` to support `kafka` SASL authentication, you also need to install:\n\n```shell\nsudo yum install -y cyrus-sasl-devel\n```\n\nIf you also use GSSAPI authentication, you need to compile the corresponding plugin:\n\n```shell\nsudo yum install -y cyrus-sasl-gssapi\n```\n\n### 1. Install submodules\n\nRun `./build_dependencies.sh`.\n\nIt will automatically download submodules and install them to `cxx/thirdparts/local` where `CMakeLists.txt` finds headers and libraries.\n\nSee `./cxx/thirdparts/local`:\n\n```\ninclude/\n  librdkafka/\n    rdkafka.h\n  log4cplus/\n    logger.h\nlib/\n  librdkafka.a\n  liblog4cplus.a\n```\n\nIf you want to support `SASL` functionality, after compiling, you can go to the `cxx/thirdparts/librdkafka/examples/` directory and execute the following command to test whether the `SASL` component has been successfully compiled:\n\n```shell\ncd cxx/thirdparts/librdkafka/examples/\n./rdkafka_example -X builtin.features\n# builtin.features = gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,sasl_oauthbearer,http,oidc\n```\nMake sure the output of `builtin.features` has compiled the `SASL` related authentication modules!\n\n### 2. Build SDK\n\n#### C++\n\nNavigate to the `cxx` directory and run `./build.sh`, following files will be generated:\n\n```\ninclude/\n  qbus_consumer.h\n  qbus_producer.h\nlib/\n  debug/libQBus.so\n  release/libQBus.so\n```\n\n\u003e Though building C++ SDK requires C++11 support, the SDK could be used with older g++. eg. build qbus SDK with g++ 4.8.5 and use qbus SDK with g++ 4.4.7.\n\n#### Go\nNavigate to the `golang` directory and run `./build.sh`, following files will be generated:\n\n```\ngopath/\n  src/\n    qbus/\n      qbus.go\n      libQBus_go.so\n```\n\nYou can enable go module for examples by running `USE_GO_MOD=1 ./build.sh`. Then following files will be generated:\n\n```\nexamples/\n  go.mod\n  qbus/\n    qbus.go\n    go.mod\n    libQBus_go.so\n```\n\n#### Python\nNavigate to the `python` directory and run `./build.sh`, following files will be generated:\n\n```\nexamples/\n  qbus.py\n  _qbus.so\n```\n\n#### PHP\nNavigate to the `php` directory and run `build.sh`, following files will be generated:\n\n```\nexamples/\n  qbus.php\n  qbus.so\n```\n\n### 3. Build examples\n\n#### C++\n\nNavigate to `examples` subdirectory and run `./build.sh [debug|release]` to generate executable files. `debug` is using `libQBus.so` in `lib/debug` subdirectory, `release` is using `libQBus.so` in `lib/release` subdirectory. Run `make clean` to delete them.\n\nIf you want to build your own programs, see how `Makefile` does.\n\n#### Go\n\nNavigate to `examples` subdirectory and run `./build.sh` to generate executable files, run `./clean.sh` to delete them.\n\nAdd path of `libQBus_go.so` to env `LD_LIBRARY_PATH`, eg.\n\n```bash\nexport LD_LIBRARY_PATH=$PWD/gopath/src/qbus:$LD_LIBRARY_PATH\n```\n\nIf you want to build your own programs, add generated `gopath` directory to env `GOPATH`, or move `gopath/src/qbus` directory to `$GOPATH/src`.\n\n#### Python\n\nCopy generated `qbus.py` and `_qbus.so` to the path of the Python scripts to run.\n\n#### PHP\n\nEdit `php.ini` and add `extension=\u003cmodule-path\u003e`, `\u003cmodule-path\u003e` is the path of `qbus.so`.\n\n\n## Usage\n\n#### Data Producing\n* In the case of writing data not by keys, the SDK will do it's best to submit every single message. As long as there is one broker in the Kafka cluster behaving normally, it will try to resend.\n* Writing data only need to invoke the `produce` interface, and in the asynchronous mode, by checking the return value, you could know whether the sending queue is full or not.\n* In the synchronous writing mode, `produce` interface will return value directively that indicate whether the current message has been written succuessfully. But that is at the expense of some extra performance loss and CPU usage. So asynchronous mode is recommended.\n* The following is a C++ example demonstrating how to invoke the `produce` interface:\n~~~c++\nbool QbusProducer::init(const string\u0026 broker_list,\n                        const string\u0026 log_path,\n                        const string\u0026 config_path,\n                        const string\u0026 topic_name);\nbool QbusProducer::produce(const char* data,\n                           size_t data_len,\n                           const std::string\u0026 key);\nvoid QbusProducer::uninit();\n~~~\n\n* C++ SDK use example:\n\n~~~c++\n#include \u003cstring\u003e\n#include \u003ciostream\u003e\n#include \"qbus_producer.h\"\n\nint main(int argc, const char* argv[]) {\n    qbus::QbusProducer qbus_producer;\n    if (!qbus_producer.init(\"127.0.0.1:9092\",\n                    \"./log\",\n                    \"./config\",\n                    \"topic_test\")) {\n        std::cout \u003c\u003c \"Failed to init\" \u003c\u003c std::endl;\n        return 0;\n    }\n\n    std::string msg(\"test\\n\");\n    if (!qbus_producer.produce(msg.c_str(), msg.length(), \"key\")) {\n        std::cout \u003c\u003c \"Failed to produce\" \u003c\u003c std::endl;\n    }\n\n    qbus_producer.uninit();\n\n    return 0;\n}\n\n~~~\n\n#### Data Consuming\n* Consuming data only need to invoke the `subscribeOne` to subscribe the 'topic' (also support subscribing multiple topics). The current process is not blocked, every message will send back to the user through the callback.\n* The SDK also supports submit offset manually, users can submit the offset in the code of the message body that returned by through callbacks.\n* The following is an example of C++, that demonstrate the usage of the consuming interface:\n~~~c++\nbool QbusConsumer::init(const std::string\u0026 broker_list,\n                        const std::string\u0026 log_path,\n                        const std::string\u0026 config_path,\n                        const QbusConsumerCallback\u0026 callback);\nbool QbusConsumer::subscribeOne(const std::string\u0026 group, const std::string\u0026 topic);\nbool QbusConsumer::subscribe(const std::string\u0026 group,\n                             const std::vector\u003cstd::string\u003e\u0026 topics);\nbool QbusConsumer::start();\nvoid QbusConsumer::stop();\nbool QbusConsumer::pause(const std::vector\u003cstd::string\u003e\u0026 topics);\nbool QbusConsumer::resume(const std::vector\u003cstd::string\u003e\u0026 topics);\n~~~\n\n* C++ SDK use example: \n\n~~~c++\n#include \u003ciostream\u003e\n#include \"qbus_consumer.h\"\n\nqbus::QbusConsumer qbus_consumer;\nclass MyCallback: public qbus::QbusConsumerCallback {\n    public:\n        virtual void deliveryMsg(const std::string\u0026 topic,\n                    const char* msg,\n                    const size_t msg_len) const {\n            std::cout \u003c\u003c \"topic: \" \u003c\u003c topic \u003c\u003c \" | msg: \" \u003c\u003c std::string(msg, msg_len) \u003c\u003c std::endl;\n        }\n\n};\n\nint main(int argc, char* argv[]) {\n    MyCallback my_callback;\n    if (qbus_consumer.init(\"127.0.0.1:9092\",\n                    \"log\",\n                    \"config\",\n                    my_callback)) {\n        if (qbus_consumer.subscribeOne(\"groupid_test\", \"topic_test\")) {\n            if (!qbus_consumer.start()) {\n                std::cout \u003c\u003c \"Failed to start\" \u003c\u003c std::endl;\n                return NULL;\n            }\n\n            while (1) sleep(1);  //other operations can appear here\n\n            qbus_consumer.stop();\n        } else {\n            std::cout \u003c\u003c \"Failed subscribe\" \u003c\u003c std::endl;\n        }\n    } else {\n        std::cout \u003c\u003c \"Failed init\" \u003c\u003c std::endl;\n    }\n    return 0;\n}\n\n~~~\n\nYou can use `pause()` and `resume()` methods to pause or resume consuming some topics, see [qbus_pause_resume_example.cc](./cxx/examples/consumer/qbus_pause_resume_example.cc)\n\nSee examples in [C examples](c/examples/)，[C++ examples](cxx/examples/)，[Go examples](golang/examples/)，[Python examples](python/examples/)，[PHP examples](php/examples/) for more usage.\n\n## CONFIGURATION\n\nThe configuration file is in [INI](https://en.wikipedia.org/wiki/INI_file) format:\n\n```ini\n[global]\n\n[topic]\n\n[sdk]\n```\n\nSee [rdkafka 1.0.x configuration](https://github.com/edenhill/librdkafka/blob/1.0.x/CONFIGURATION.md) for *global* and *topic* configurations, and [sdk configuration](https://github.com/Qihoo360/kafkabridge/blob/master/CONFIGURATION.md) for *sdk* configuration.\n\nNormally kafkabridge works with an empty configuration file, but if your broker version \u003c 0.10.0.0, you must specify api.version-related configuration parameters, see [broker version compatibility](https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#broker-version-compatibility).\n\neg. for broker 0.9.0.1, following configurations are necessary:\n\n```ini\n[global]\napi.version.request=false\nbroker.version.fallback=0.9.0.1\n```\n\n**The default config is now compatible with broker 0.9.0.1. Therefore, if higher version broker is used, `api.version.request` should be set true. Otherwise, the message protocol would be older version, e.g. no timestamp field.**\n\n## Contact\n\n QQ group: 876834263\n\n![](https://github.com/Qihoo360/qbusbridge/blob/master/kafkabridge.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqihoo360%2Fqbusbridge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqihoo360%2Fqbusbridge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqihoo360%2Fqbusbridge/lists"}