{"id":13486647,"url":"https://github.com/brokercap/Bifrost","last_synced_at":"2025-03-27T21:30:47.628Z","repository":{"id":37910298,"uuid":"129269428","full_name":"brokercap/Bifrost","owner":"brokercap","description":"Bifrost ---- 面向生产环境的 MySQL,MariaDB,kafka 同步到Redis,MongoDB,ClickHouse,StarRocks,Doris,Kafka等服务的异构中间件","archived":false,"fork":false,"pushed_at":"2024-10-16T12:53:09.000Z","size":4321,"stargazers_count":1943,"open_issues_count":81,"forks_count":330,"subscribers_count":49,"default_branch":"master","last_synced_at":"2024-10-29T15:40:50.496Z","etag":null,"topics":["cdc-go","clickhouse","datalink","datax","dts","go","go-canal","go-mysql","go-mysqlbinlog","go-otter","gobinlog","mongodb","mysql","mysql-kafka","mysql-mongodb","mysql-redis","mysqlbinlog","rabbitmq"],"latest_commit_sha":null,"homepage":"https://www.xbifrost.com","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brokercap.png","metadata":{"files":{"readme":"README.EN.MD","changelog":"changelog.txt","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-12T15:05:46.000Z","updated_at":"2024-10-28T03:38:44.000Z","dependencies_parsed_at":"2023-10-14T14:06:39.514Z","dependency_job_id":"b3288d1d-3f23-4415-85bc-494fbc0635c0","html_url":"https://github.com/brokercap/Bifrost","commit_stats":{"total_commits":976,"total_committers":12,"mean_commits":81.33333333333333,"dds":0.2551229508196722,"last_synced_commit":"8eef6f7b14e3dc414168e557e84378c867c70da6"},"previous_names":["jc3wish/bifrost"],"tags_count":94,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brokercap%2FBifrost","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brokercap%2FBifrost/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brokercap%2FBifrost/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brokercap%2FBifrost/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brokercap","download_url":"https://codeload.github.com/brokercap/Bifrost/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222314063,"owners_count":16965428,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cdc-go","clickhouse","datalink","datax","dts","go","go-canal","go-mysql","go-mysqlbinlog","go-otter","gobinlog","mongodb","mysql","mysql-kafka","mysql-mongodb","mysql-redis","mysqlbinlog","rabbitmq"],"created_at":"2024-07-31T18:00:49.446Z","updated_at":"2025-03-27T21:30:47.608Z","avatar_url":"https://github.com/brokercap.png","language":"Go","funding_links":[],"categories":["开源类库","StandAlone Projects","Open source library","Go","数据管道","Repositories"],"sub_categories":["数据库","Database"],"readme":"\r\n## Bifrost ---- A heterogeneous middleware which can synchronize MySQL/MariaDB binlog data To Redis,MongoDB,ClickHouse,Elasticsearch...\r\n\r\n##### *Real time data synchronization, only use mouse .....*\r\n\r\n[![Gitter](https://badges.gitter.im/brokercap-Bifrost/Bifrost.svg)](https://gitter.im/brokercap-Bifrost/Bifrost?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge)\r\n[![Build Status](https://app.travis-ci.com/brokercap/Bifrost.svg?branch=v1.8.x)](https://app.travis-ci.com/github/brokercap/Bifrost)\r\n[![License](https://img.shields.io/github/license/jc3wish/Bifrost.svg)](https://opensource.org/licenses/apache2.0)\r\n\r\n[中文](https://github.com/brokercap/Bifrost/blob/v1.8.x/README.MD)\r\n\r\nBinary [Linux](https://github.com/brokercap/Bifrost/releases) , [mac](https://github.com/brokercap/Bifrost/releases) , [windows](https://github.com/brokercap/Bifrost/releases) , [freebsd](https://github.com/brokercap/Bifrost/releases)\r\n\r\nSource: [Github](https://github.com/brokercap/Bifrost/)  , [Gitee](https://gitee.com/jc3wish/Bifrost)\r\n\r\nYour star is the biggest support for Bifrost!!!\r\n\r\n##### TO\r\n\r\n| ToServer                                                                                                | DDL | Notes                                           |\r\n|---------------------------------------------------------------------------------------------------------|-----|-------------------------------------------------|\r\n| Redis                                                                                                   | NO  |                                                 |\r\n| MongoDB                                                                                                 | NO  |                                                 |\r\n| MySQL                                                                                                   | YES |                                                 |\r\n| ClickHouse                                                                                              | YES |                                                 |\r\n| Memcache                                                                                                | NO  |                                                 |\r\n| RabbitMQ                                                                                                | YES |                                                 |\r\n| Kafka                                                                                                   | YES | support canal json,debezium json, string data   |\r\n| ActiveMQ                                                                                                | YES |                                                 |\r\n| [Http](https://github.com/brokercap/Bifrost/blob/v1.8.x/plugin/http/example/http_server/http_server.go) | YES  |                                                 |\r\n| [Hprose RPC](https://github.com/brokercap/Bifrost/blob/v1.8.x/hprose_server/tcp_server.go)              | YES  |                                                 |\r\n| [Hprose RPC](https://github.com/brokercap/Bifrost/blob/v1.8.x/hprose_server/tcp_server.go)              | YES  |                                                 |\r\n| StarRocks                                                                                               | YES | use mysql protocol                              |\r\n| Doris                                                                                                   | YES | use mysql protocol                              |\r\n\r\n\r\n##### FROM\r\n\r\n| 源端      | 全量  | 增量  | 格式               | 备注                                |\r\n|---------|-----|-----|------------------|-----------------------------------|\r\n| MySQL   | YES | YES | Binglog \u0026 Select | MySQL Slave                       |\r\n| MariaDB | YES | YES | Binglog \u0026 Select | MySQL Slave                       |\r\n| Percona | YES | YES | Binglog \u0026 Select | MySQL Slave                       |\r\n| Kafka   | NO  | YES | string           | kafka data to field \"queue_data\"  |\r\n| Kafka   | NO  | YES | canal            | canal json to bifrost data        |\r\n| kafka   | NO  | YES | bifrost          | bifrost to kafka,kafka to bifrost |\r\n| kafka   | NO  | YES | debezium         | debezium for mysql data           |\r\n| Mongo   | NO  | YES | oplog            |                                   |\r\n| kafka   | NO  | YES | customer json    | customer json to bifrost struct   |\r\n| Mock    | NO  | YES | MySQL Binlog     | use for outut plugin develop      |\r\n\r\n---\r\n\r\n#### **WIKI** : [https://wiki.xbifrost.com/en](https://wiki.xbifrost.com/en)\r\n\r\n----\r\n\r\n#### Online experience\r\n\r\nhttp://www.xbifrost.com/demo/\r\n\r\n![image](https://github.com/brokercap/Bifrost/blob/v1.8.x/bifrost.png)\r\n\r\n\r\n#### *Bifrsot* Diagram\r\n\r\n![image](https://github.com/brokercap/Bifrost/blob/v1.8.x/diagram.png)\r\n\r\n\r\n### Points\r\n\r\n\r\n* Support all save types of MySQL/MariaDB\r\n\r\n* Interface **dynamics** can dispose different data sheet and target database flexibly\r\n\r\n* Supported by various data sources and target databases\r\n\r\n* Supported **FULL** AND **Increment**\r\n\r\n* Supported Email AND Wechat **warning**\r\n\r\n* Supported **Flow** display\r\n\r\n* Plugin MySQL and ClickHouse suppoted DDL\r\n\r\n\r\n### MySQL version(after test)\r\n\r\n* 5.1.73\r\n\r\n* 5.5.42\r\n\r\n* 5.6.36\r\n\r\n* 5.7.18\r\n\r\n* 8.0.11\r\n\r\n\r\n### Install\r\n\r\n##### Compile\r\n\r\ndownload Bifrost Source\r\n\r\n```\r\ngit clone git clone https://github.com/brokercap/Bifrost.git\r\n\r\n./build.sh linux\r\n\r\n./build.sh windows\r\n\r\n./build.sh freebsd\r\n\r\n./build.sh darwin\r\n```\r\n\r\nAfter compiling, the corresponding platform name folder will be created in the tags directory,\r\n\r\n##### Binary\r\n\r\n```\r\nwget https://github.com/brokercap/Bifrost/releases/download/v2.3.12-beta/bifrost_v2.3.12-beta_Linux-amd64-bin.tar.gz\r\n\r\ntar -zxvf bifrost_v2.3.12-beta_Linux-amd64-bin.tar.gz\r\n\r\ncd bifrost_v2.3.12-beta_Linux-amd64-bin/bin \u0026\u0026 chmod a+x ./Bifrost*\r\n\r\n```\r\n\r\n\r\n##### Start\r\n\r\n`````shell\r\n./Bifrost-server start\r\n\r\n`````\r\n\r\n##### Stop\r\n\r\n`````shell\r\n./Bifrost-server stop\r\n\r\n`````\r\n\r\n\r\n##### manager\r\n\r\nhttps://127.0.0.1:21036\r\n\r\nDefault user name : Bifrost\r\n\r\n\r\nDefault user password : Bifrost123\r\n\r\n\r\n##### Docker\r\n\r\n`````shell\r\ndocker pull jc3wish/bifrost\r\n\r\nmkdir -p /data/BifrostData\r\n\r\ndocker run -d -p21036:21036 -v /data/BifrostData:/linux/data jc3wish/bifrost\r\n\r\n`````\r\n\r\n\r\n##### Configuration\r\n`````sh\r\n\r\nvim Bifrost.ini\r\n\r\n[user]\r\n#登入用户名和密码\r\n#用户名在前，= 号后面是密码\r\nBifrost=Bifrost123\r\nBifrostMonitor=Bifrost123\r\n\r\n[groups]\r\n#administrator 管理人员可以添加删除等操作\r\n#monitor 只能查看数据\r\nBifrost=administrator\r\nBifrostMonitor=monitor\r\n\r\n[Bifrostd]\r\n#log写到入的文件夹，默认为启动文件所在文件夹下的logs目录\r\n#log_dir=/data/log/\r\n\r\n#监听端口,默认 0.0.0.0:21036\r\nlisten=0.0.0.0:21036\r\n\r\n#通过界面配置的信息保存文件夹,默认为启动文件所在文件夹下的data目录\r\ndata_dir=./\r\n\r\n#设置同步给第三方服务的队列大小，默认5000\r\ntoserver_queue_size=5000\r\n\r\n#通道队列的大小，默认1000\r\nchannel_queue_size=1000\r\n\r\n#是否支持动态加plugin so 插件，默认为false\r\n#这个参数只在 linux 下有效，其他平台全部为false\r\ndynamic_plugin=false\r\n\r\n#是否支持https\r\ntls=true\r\n\r\n#server.key 及 server.crt 相对于 Bifrost 的路径，也可以填绝对路径\r\ntls_key_file=./etc/server.key\r\ntls_crt_file=./etc/server.crt\r\n\r\n#采用什么方式存储,默认文件方式\r\nmeta_storage_type=redis\r\n\r\n#存储介质的连接地址\r\nmeta_storage_path=127.0.0.1:6379\r\n\r\n#用于区别实例的名字\r\ncluster_name=bifrostTestClusterName\r\n\r\n`````\r\n\r\n##### build https certificate\r\n\r\n[example](https://github.com/brokercap/Bifrost/wiki/HTTPS%E8%AF%81%E4%B9%A6)\r\n\r\n\r\n\r\n### Manager doc\r\n\r\n[Bifrost](http://www.xbifrost.com/)\r\n\r\n### Thank you to the relevant Elaine package provider\r\n\r\n- github.com/gmallard/stompngo\r\n- github.com/hprose/hprose-golang\r\n- github.com/Shopify/sarama\r\n- github.com/bradfitz/gomemcache/memcache\r\n- gopkg.in/mgo.v2\r\n- github.com/streadway/amqp\r\n- github.com/go-redis/redis/v8\r\n- github.com/syndtr/goleveldb/leveldb\r\n- github.com/ClickHouse/clickhouse-go\r\n- github.com/olivere/elastic/v7\r\n\r\n### QQ群号： 70331601\r\n\r\n\u003cimg src=\"/images/qq.jpg\" width=\"250\" alt=\"qq群\"/\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrokercap%2FBifrost","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrokercap%2FBifrost","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrokercap%2FBifrost/lists"}