{"id":15109601,"url":"https://github.com/zuoyebang/bitalostored","last_synced_at":"2026-04-03T04:03:12.428Z","repository":{"id":224506907,"uuid":"763428598","full_name":"zuoyebang/bitalostored","owner":"zuoyebang","description":"Bitalostored is a high-performance distributed storage system, core engine based on bitalosdb(self-developed), compatible with Redis protocol.","archived":false,"fork":false,"pushed_at":"2026-03-23T08:13:26.000Z","size":23156,"stargazers_count":2163,"open_issues_count":4,"forks_count":153,"subscribers_count":150,"default_branch":"main","last_synced_at":"2026-03-23T23:52:39.834Z","etag":null,"topics":["bitalosdb","database","distributed-storage","high-performance","kvstore","nosql","redis","storage-engine"],"latest_commit_sha":null,"homepage":"https://bitalos.zuoyebang.com","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zuoyebang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-02-26T09:24:45.000Z","updated_at":"2026-03-22T14:41:11.000Z","dependencies_parsed_at":"2024-03-13T03:31:54.456Z","dependency_job_id":"18a65897-d815-4df3-8c6d-593cf3a86148","html_url":"https://github.com/zuoyebang/bitalostored","commit_stats":null,"previous_names":["zuoyebang/bitalostored"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/zuoyebang/bitalostored","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zuoyebang%2Fbitalostored","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zuoyebang%2Fbitalostored/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zuoyebang%2Fbitalostored/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zuoyebang%2Fbitalostored/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zuoyebang","download_url":"https://codeload.github.com/zuoyebang/bitalostored/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zuoyebang%2Fbitalostored/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31333229,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-03T03:20:36.090Z","status":"ssl_error","status_checked_at":"2026-04-03T03:20:35.133Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitalosdb","database","distributed-storage","high-performance","kvstore","nosql","redis","storage-engine"],"created_at":"2024-09-25T23:05:32.264Z","updated_at":"2026-04-03T04:03:12.420Z","avatar_url":"https://github.com/zuoyebang.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"![bitalos](./docs/bitalos.png)\n\n### Bitalostored is a high-performance distributed storage system, compatible with Redis protocol. [中文版](./README_CN.md)\n\n## Introduction\n\n- Bitalostored is a high-performance distributed storage system, core engine based on [bitalosdb](https://github.com/zuoyebang/bitalosdb/blob/main/README.md), compatible with Redis protocol. As an alternative to Redis, it stores data with low-cost hard disk instead of expensive memory, takes full advantage of multi-core and provides excellent single-core performance, which can significantly reduce service costs.\n\n- Bitalostored contains three main projects: dashboard (visual management platform), stored (storage service), and proxy (proxy service). Current open-source version is stable, and provides a complete industrial grade solution.\n\n## Main Creator\n\n- Author: Xu Ruibo(hustxurb@163.com), joined Zuoyebang in December 2018 (working till now), is responsible for live class middle platform and Zuoyebang platform, and leads the storage technology team to develop Bitalos from 0 to 1.\n\n- Contributors: Xing Fu(wzxingfu@gmail.com), Lu Wenwei(422213023@qq.com), Liu Fang(killcode13@sina.com), Li Jingchen(cokin.lee@outlook.com)\n\n# Practice (Zuoyebang)\n\n- Application: supports Zuoyebang's online \u0026 offline KV storage\n\n- Peak traffic: QPS(30 Million), Network(8000Gbps)\n\n- Storage data: 1PB\n\n- Average reading time: 100μs, average writing time: 150μs\n\n- Availability: 99.999%\n\n- Stability: Since V1.0 was released in 2019, there have been zero online incidents\n\n## Key Technology\n\n- Compatible with Redis protocol, low integration cost. Supports most commands, including LUA, distributed transactions.\n\n- Efficient horizontal scaling, supporting up to 1024 shards and enabling shard expansion in seconds.\n\n- Multi-cloud disaster recovery, supporting multi-data room or multi-cloud deployment \u0026 management, with a complete downgrade and disaster recovery solution.\n\n- Multi-master architecture (supported in Enterprise Edition), based on CRDT, optimized data synchronization and consistency strategies, ensuring adaptive conflict resolution during multi-master writes and achieving eventual consistency.\n\n- High-performance data consistency architecture, based on Bitalos-Raft, with deeply optimized Raft protocol, which greatly improves write performance, and features more stable election strategy and data synchronization process.\n    - High-performance log engine: bitaloslog, featuring log indexing based on bitalostree, supporting high-throughput write operations.\n\n- High-performance kernel equipped with self-developed KV engine: bitalosdb.\n    - High-performance compressed indexing technology: bitalostree, a B+ tree based on ultra-large pages, featuring innovative index compression that eliminates write amplification in B+ trees and maximizes read performance.\n\n    - High-performance KV index, vector computation implemented based on ASM assembly, with significantly improved performance.\n\n    - High-performance K-KV index, multi-level vector index based on bitalostree, balancing index compression ratio and retrieval performance.\n\n    - High-performance KV separation technology: bithash, based on a compact index structure, with O(1) retrieval efficiency and capable of independent GC.\n\n    - High-performance storage structure that compresses Redis composite data types, significantly reducing I/O costs and improving system throughput.\n\n## Quick deployment\n\n- Applicable scenarios: Deploy a test cluster on a single machine(machine needs to be connected to the Internet), experience the functions of all components(dashboard, proxy, and stored), and cluster operation and maintenance\n\n- Deployment script: install.sh, follow the prompts to enter the number of shards (group), the number of slave nodes (slave), and the number of witness nodes (witness); the default number: proxy * 1, group * 2 (master * 2, slave * 2 , witness * 2)\n\n- Admin web: 127.0.0.1:8080, both of default user\u0026password are demo\n\n- Service address: 127.0.0.1:8790, use command: redis-cli -h 127.0.0.1 -p 8790\n\n- Uninstall script: uninstall.sh\n\n## Performance\n\n- Currently, there are several well-known open-source storage systems (Redis-compatible) that serve as alternatives to Redis. We have selected two high-performance products (t\\*d\\* and \\*i\\*a) for performance comparison.\n\n### Hardware\n\n```\nCPU:    Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz\nMemory: 384GB\nDisk:   2*3.5TB NVMe SSD\n```\n\n### Program\n\n- Benchmark: memtier_benchmark (redis official tool)\n\n- NoSQL Program: thread number(8), cgroup cpu(8 core)\n\n- Command args: 2 data spec\n\n```\n--data-size=1024 --key-maximum=40672038 -t 16 -c 32 -n 81920\n--data-size=128 --key-maximum=335544320 -t 16 -c 32 -n 655360\n```\n\n- Command (e.g., --data-size=1024)\n\n```\n./memtier_benchmark -t 16 -c 32 -s 127.0.0.1 -p xxxx --distinct-client-seed --command=\"set __key__ __data__\" --key-prefix=\"performance_test_key_prefix_\" --key-minimum=1 --key-maximum=40672038 --random-data --data-size=1024 -n 81920\n./memtier_benchmark -t 16 -c 32 -s 127.0.0.1 -p xxxx --distinct-client-seed --command=\"get __key__\" --key-prefix=\"performance_test_key_prefix_\" --key-minimum=1 --key-maximum=40672038 --test-time=300\n./memtier_benchmark -t 16 -c 32 -s 127.0.0.1 -p xxxx --distinct-client-seed --command=\"incr __key__\" --key-prefix=\"int_\" --key-minimum=1 --key-maximum=40672038 --random-data --data-size=1024 -n 81920\n./memtier_benchmark -t 16 -c 32 -s 127.0.0.1 -p xxxx --distinct-client-seed --command=\"lpush __key__ __data__\" --key-prefix=\"list_\" --key-minimum=1 --key-maximum=40672038 --random-data --data-size=1024 -n 81920\n./memtier_benchmark -t 16 -c 32 -s 127.0.0.1 -p xxxx --distinct-client-seed --command=\"sadd __key__ __data__\" --key-prefix=\"set_\" --key-minimum=1 --key-maximum=40672038 --random-data --data-size=1024 -n 81920\n./memtier_benchmark -t 16 -c 32 -s 127.0.0.1 -p xxxx --distinct-client-seed --command=\"zadd __key__ __key__ __data__\" --key-prefix=\"\" --key-minimum=1 --key-maximum=40672038 --random-data --data-size=1024 -n 81920\n./memtier_benchmark -t 16 -c 32 -s 127.0.0.1 -p xxxx --distinct-client-seed --command=\"hset __key__ __data__ __key__\" --key-prefix=\"hash_\" --key-minimum=1 --key-maximum=40672038 --random-data --data-size=1024 -n 81920\n```\n\nincr is irrelevant to data size, only needs to be tested once.\n\n\n### Data\n\n- Total data size：40GB\n\n- Comparison dimensions： comand（SET、GET、LPUSH、SADD、ZADD、HSET） x value-size\u0026count（1KB \u0026 40,672,000、128B \u0026 335,544,320）, INCR\n\n### Config\n\n- t\\*d\\* \u0026 \\*i\\*a\n\n```\nThreads:8\nMemtable：1GB\nWAL：disable\nBinlog：disable\nCache：40GB\n\nOther parameters are set as same as the official recommended benchmark configuration\n```\n\n- bitalostored\n\n```\nThreads:8\nMemtable：1GB\nWAL：disable\nRaftlog：disable\n```\n\n### Result\n\n- QPS\n\n![benchmark](./docs/benchmark-bitalostored-qps.png)\n\n## Document\n\nTechnical architecture and documentation, refer to the official website: bitalos.zuoyebang.com\n\n## Technology accumulation(bitalosearch)\n\n- High performance distributed search \u0026 analysis engine, SQL protocol, focusing on AP scenarios, and has certain TP capabilities. It is being practiced internally, and the open source plan is to be determined\n\n- Compared to elasticsearch, bitalosearch has significant cost advantages. Hard disk consumption is saved 30%; data writing performance is improved by 25%; for complex analysis logic, query performance is improved by 20% to 500%","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzuoyebang%2Fbitalostored","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzuoyebang%2Fbitalostored","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzuoyebang%2Fbitalostored/lists"}