{"id":48239119,"url":"https://github.com/qoollo/bob","last_synced_at":"2026-04-04T20:13:26.202Z","repository":{"id":37456756,"uuid":"174350948","full_name":"qoollo/bob","owner":"qoollo","description":"Distributed BLOB storage","archived":false,"fork":false,"pushed_at":"2024-11-27T12:03:12.000Z","size":5893,"stargazers_count":31,"open_issues_count":105,"forks_count":3,"subscribers_count":11,"default_branch":"master","last_synced_at":"2024-11-27T13:19:05.187Z","etag":null,"topics":["big-data","blob-storage","distributed","high-performance","key-value","storage"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/qoollo.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-07T13:40:08.000Z","updated_at":"2024-11-01T03:06:42.000Z","dependencies_parsed_at":"2023-09-26T20:19:58.865Z","dependency_job_id":"d4a26771-9cf6-409d-b3f8-923d210226dc","html_url":"https://github.com/qoollo/bob","commit_stats":null,"previous_names":[],"tags_count":85,"template":false,"template_full_name":null,"purl":"pkg:github/qoollo/bob","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qoollo%2Fbob","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qoollo%2Fbob/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qoollo%2Fbob/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qoollo%2Fbob/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/qoollo","download_url":"https://codeload.github.com/qoollo/bob/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qoollo%2Fbob/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31412144,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T20:09:54.854Z","status":"ssl_error","status_checked_at":"2026-04-04T20:09:44.350Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","blob-storage","distributed","high-performance","key-value","storage"],"created_at":"2026-04-04T20:13:22.389Z","updated_at":"2026-04-04T20:13:26.193Z","avatar_url":"https://github.com/qoollo.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bob\n[![build](https://github.com/qoollo/bob/actions/workflows/build.yml/badge.svg)](https://github.com/qoollo/bob/actions/workflows/build.yml) [![Docker Pulls](https://img.shields.io/docker/pulls/qoollo/bob?color=brightgreen)](https://hub.docker.com/r/qoollo/bob/)\n\n![Bob](logo/bob-git-docker-logo.svg)\n\nBob is a distributed storage system designed for byte data such as photos. It has decentralized architecture where each node can handle user calls. [Pearl](https://github.com/qoollo/pearl) is used as a backend storage.\n\nMore information can be found on [wiki](https://github.com/qoollo/bob/wiki).\n\n# Installation\nFor Linux x86_64, you can use RPM or DEB packages from the [releases](https://github.com/qoollo/bob/releases) page.\nThere is also a ZIP archive with compiled x64 binaries.\n\nDocker images can be found on DockerHub [qoollo/bob](https://hub.docker.com/r/qoollo/bob/).\n\nAdditionally, you can build Bob from sources. Instruction can be found [here](https://github.com/qoollo/bob/wiki/Build-from-source).\n\n# API\nBob uses gRPC API with the following proto file: [bob.proto](bob-grpc/proto/bob.proto).\nThis means that you can automatically generate a protocol implementation for any platform. \nDetailed gRPC API description can be found on [wiki](https://github.com/qoollo/bob/wiki/gRPC-API).\n\nThere is also a rich client for .NET: [https://github.com/qoollo/bob-client-net](https://github.com/qoollo/bob-client-net).\n\n# Tools\nYou can use [bobc](https://github.com/qoollo/bob/blob/master/bob-apps/bin/bobc.rs) and [bobp](https://github.com/qoollo/bob/wiki/Tools#bob-benchmark-tool-bobp) to access data on a cluster. `brt` can be used to check and recover corrupted BLOBs.\n\nThere are also tools for removing old partitions, recovering aliens, expanding cluster and more, that can be found in [bob-tools](https://github.com/qoollo/bob-tools) repository.\n\n# Performance\n\nBob can handle more than 10k RPS. \nDetailed tests and hardware configuration can be found on [wiki](https://github.com/qoollo/bob/wiki/Performance-tests).\n\n\n# Architecture\n[cluster.yaml](https://github.com/qoollo/bob/wiki/Cluster-config) file describes the addresses of all nodes in the cluster and a set of directories/physical disks on each node. All data is logically distributed across virtual disks (vdisks). You should create a cluster with more virtual disks than physical ones (it would be reasonable to place 3..10 vdisks on 1 physical disk). Destination vdisk determs like this: `vdisk_id = data_id % vdisks_count`. Cluster writes data to all nodes that contain target vdisk.\n\nExample config with 2 nodes and 3 vdisks:\n```YAML\n- nodes:\n  - name: node1\n    address: 127.0.0.1:20000\n    disks:\n      - name: disk1\n        path: /tmp/d1\n  - name: node2\n    address: 127.0.0.1:20001\n    disks:\n      - name: disk1\n        path: /tmp/d1\n      - name: disk2\n        path: /tmp/d2\n\n- vdisks:\n  - id: 0\n    replicas:\n      - node: node1\n        disk: disk1\n  - id: 1\n    replicas:\n      - node: node2\n        disk: disk1\n  - id: 2\n    replicas:\n      - node: node1\n        disk: disk1\n      - node: node2\n        disk: disk2\n```\n\nAs you can see, there are no replicated nodes, instead Bob replicates vdisks. Vdisks are distributed across nodes, which provides more flexibility. Some nodes with more powerful hardware can have more vdisks and therefore store more data.\n\n`cluster.yaml` can be generated semi-automatically with [CCG](https://github.com/qoollo/bob/wiki/Tools#cluster-config-generator-ccg).\n\n# Backend\nBackend store data in 2 groups: \"normal\" and \"alien\" folders. \"Normal\" are vdisk folders described in cluster config . \"Alien\" is folder for data that cannot be written to its node due to node unavailability. It has the same structure that \"normal\" folder but also has \"node name\" in folder hierarchy. \nUnder vdisk info in folder it has timestamp folder info. \n```\n- disk 1\n    - vdisk id 1\n...\n    - vdisk id N\n        - timestamp 1\n        ...\n        - timestamp N\n            - blob 1\n            ...\n            - blob N\n...\n- disk N\n    - Alien\n        - node name\n            - vdisk id 1\n                - timestamp 1\n                    - blob 1\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqoollo%2Fbob","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqoollo%2Fbob","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqoollo%2Fbob/lists"}