{"id":20331231,"url":"https://github.com/comcast/bynar","last_synced_at":"2025-04-22T20:21:16.656Z","repository":{"id":53551539,"uuid":"108432847","full_name":"Comcast/Bynar","owner":"Comcast","description":"Server remediation as a service","archived":false,"fork":false,"pushed_at":"2024-07-22T17:40:05.000Z","size":3458,"stargazers_count":58,"open_issues_count":27,"forks_count":13,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-03-29T18:36:29.148Z","etag":null,"topics":["ceph","protobuf","rust","zeromq"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Comcast.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"Contributing.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-26T15:50:51.000Z","updated_at":"2024-10-26T10:27:50.000Z","dependencies_parsed_at":"2025-01-16T15:07:56.020Z","dependency_job_id":"1818a462-3798-462b-a214-eaa99121fb70","html_url":"https://github.com/Comcast/Bynar","commit_stats":{"total_commits":463,"total_committers":13,"mean_commits":35.61538461538461,"dds":0.593952483801296,"last_synced_commit":"cb8d76028fc0e6202cbba61677a131863532b659"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Comcast%2FBynar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Comcast%2FBynar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Comcast%2FBynar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Comcast%2FBynar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Comcast","download_url":"https://codeload.github.com/Comcast/Bynar/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250316138,"owners_count":21410483,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ceph","protobuf","rust","zeromq"],"created_at":"2024-11-14T20:19:19.885Z","updated_at":"2025-04-22T20:21:16.638Z","avatar_url":"https://github.com/Comcast.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bynar\n\u003cimg src=\"bynar.png\" width=100\u003e\n\n[![Build Status](https://travis-ci.org/Comcast/Bynar.svg?branch=master)](https://travis-ci.org/Comcast/Bynar)\n\nWarehouse scale server repair, more benign than borg.\n\n----\n\nBynar is an open source system for automating server maintenance\nacross the datacenter.  Bynar builds upon many years of experience\nautomating the drudgery of server repair. The goal is to have the\ndatacenter maintain itself.  Large clusters these days require\nlots of maintenance.  [Cassandra], [Ceph], [Gluster], [Hadoop] and others\nall require quick replacement of server parts as they break down or the cluster\ncan become degraded.  As your cluster grows, you generally need to have more\npeople to maintain them.  Bynar hopes to break this cycle and\nfree up your time so that your clusters can scale to ever greater sizes\nwithout requiring more people to maintain them.  \n\nThe project is divided into different binaries that all communicate over protobuf:\n1. disk-manager: This program handles adding and the removal of disks from a server\n2. bynar:  This program handles detection of failed hard drives, files a ticket\nfor a datacenter technician to replace the drive, waits for the resolution of the ticket and\nthen makes an API call to `disk-manager` to add the new disk back into the server.\n3. bynar-client: Enables you to manually make API calls against `disk-manager` and `bynar`\n\n\n----\n\n## To start using Bynar\n\n### Infrastructure:\nBynar requires a Postgres database to be setup.  Setting up a production ready Postgres is outside the scope of this document.  For testing Bynar a [docker postgres](https://hub.docker.com/_/postgres) container is quick to setup.  The database maintains information about hardware status and ongoing operations.\n\n### Configuration:\n1. Create your configuration file.  The utility takes json config\ninformation.  Edit the `/etc/bynar/bynar.json` file to configure it.\nThe slack_* fields are optional.  They will allow Bynar to send alerts to a\nchannel while it's performing maintenance. The daemon_* fields are optional.  \nThey will allow the user to choose the output files if Bynar is run as a daemon.\nJIRA is the only currently supported back end ticketing system.  A plugin system allows for more back end support.  \nAn optional proxy field can be configured to send JIRA REST API requests through.\nFor extra security we highly recommend that you enable the vault integration.\nThe disk-manager sits on a port and if an attacker gains access to it they can\nquickly wipe out your disks.  If you don't wish to enable vault integration\nset the disk-manager up to only listen on a loopback port.\nFields for this file are listed below. A sample file can also be found under\nconfig/bynar.json.\n\n```\n{\n \"proxy\": \"https://my.proxy\",\n \"manager_host\": \"localhost\",\n \"manager_port\": 5555,\n \"slack_webhook\": \"https://hooks.slack.com/services/ID\",\n \"slack_channel\": \"#my-channel\",\n \"slack_botname\": \"my-bot\",\n \"jira_user\": \"test_user\",\n \"jira_password\": \"user_password\",\n \"jira_host\": \"https://tickets.jira.com\",\n \"jira_issue_type\": \"3\",\n \"jira_priority\": \"4\",\n \"jira_project_id\": \"MyProject\",\n \"jira_ticket_assignee\": \"assignee_username\",\n \"vault_endpoint\": \"https://my_vault.com\",\n \"vault_token\": \"token_98706420\",\n \"database\": {\n     \"username\": \"postgres_user\",\n     \"password\": \"postgres_passwd\",\n     \"port\": \"8888\",\n     \"dbname\": \"database_name\",\n     \"endpoint\": \"some.endpoint\"\n },\n \"daemon_output\": \"bynar_daemon.out\",\n \"daemon_error\" : \"bynar_daemon.err\",\n \"daemon_pid\" : \"bynar_daemon.pid\"\n}\n```\n\n## Disk Manager\nThis binary handles adding and removing disks from a server.  It uses\nprotobuf serialization to allow RPC usage. Please check the\n[api crate](https://github.com/Comcast/Bynar/tree/master/api) for more information or the [bynar-client](https://github.com/Comcast/Bynar/tree/master/src/client.rs).\n\n## Configuration:\n1. Create your configuration file.  The utility takes json config\n`/etc/bynar/disk-manager.json` file. This file should be deployed  \nwhen the Bynar package is installed. The vault_* options are optional\nbut recommended.  When enabled the disk-manager upon starting will save\nthe generated public key to vault under `/bynar/{hostname}.pem`.  Any clients\nwanting to connect to it will need to contact vault first.  If vault is\nnot enabled it will save the public key to /etc/bynar/.\n```\n{\n  \"backend\": \"ceph\",\n  \"vault_endpoint\": \"https://my_vault:8888\",\n  \"vault_token\": \"token_98706420\"\n}\n```\nBynar that runs on Ceph, should have a ceph.json file to describe it. This tells \nwhere to look for ceph configuration, user details etc.\n`/etc/bynar/ceph.json` file:\n```\n{\n  \"config_file\": \"/etc/ceph/ceph.conf\",\n  \"user_id\": \"admin\",\n  \"pool_name\": \"pool_name\",\n  \"target_weight\": 1.0,\n  \"system_disks\": [\n    {\n      \"device\": \"/dev/sdc\"\n    }\n  ],\n  \"journal_devices\": [\n\t\t{\n\t\t\t\"device\": \"/dev/sda\"\n\t\t},\n\t\t{\n\t\t\t\"device\": \"/dev/sdb\",\n\t\t\t\"partition_id\": 1\n\t\t}\n\t],\n  \"osd_config\": [\n    {\n      \"is_lvm\": false,\n      \"dev_path\": \"/dev/sdx\",\n      \"journal_path\" : \"/dev/sdxY\",\n      \"rdb_path': \"dev/sdxZ\",\n    }\n  ]\n\t\"udev_rule_path\": \"/etc/udev/rules.d\"\n}\n```\nThe pool_name is the name of the pool used to measure latency in the cluster,\ntarget_weight the desired weight of OSDs in the cluster. \n\nSystem Disks must be specified for ceph to filter out.  \nThis is a list of all disks that Ceph should not run on.  \nA disk with the root or boot partition, as wellas the device path of the root and boot (/boot, /boot/efi) partitions must be provided for Bynar to filter out.  \nBynar needs to be able to distinguish the disks so it does not try to wipe a boot partition.  \nIf not provided ceph will attempt to add/remove the disk/partition as an OSD. \n\nOptionally, latency_cap, backfill_cap, and increment can be specified for ceph to use. \nBynar will gradually weight in an osd that is added to the cluster so as not to introduce\ntoo much latency to the cluster or cause issues with pgs stuck in backfill.  \nBynar has its own defaults to use however explicit parameters can be set.\nPlease note that the latency_cap is in ms\n\nJournal devices can optionally be specified for ceph to use.  Bynar will attempt\nto balance the number of partitions across the devices given.  If an explict \n`partition_id` is also given Bynar will make use of that.  If no `partition_id`\nis given Bynar will create new partitions when disks are added.  The partition \nsize will be equal to the ceph.conf `osd journal size` configuration setting \nwhich is given in megabytes.\n\nOsd Configs should be specified for ceph to use for each OSD device on the server.  \nThis lets Bynar know whether to add an osd device manually or through LVM. \nWhen configuring for a Bluestore device that will not be added as an LVM, \nyou can also specify the journal path and the RocksDB path (the \nblock.wal and block.db symlinks respectively), though they should not point to the same location.\n\nThe udev_rules_path is needed when adding an osd device manually, as the kernel needs to recognize that the device is owned by ceph:ceph\n### Directory layout:\n1. Top level is the dead disk detector aka bynar\n2. api is the protobuf api create\n3. disk-manager is the service that handles the adding and removal of disks\n\n### Launch the program\n1. After building Bynar from source or downloading prebuilt packages\nlaunch the `disk-manager`, `bynar` service on every server you want\nmaintained.\n\n## To start developing Bynar\n\nThis [community repository] hosts all information about\nbuilding Bynar from source, how to contribute code\nand documentation, who to contact about what, etc.\n\n### Dependencies for Ubuntu 18.04:\nEnsure there is enough space on the root partition of your development system.\nTypical recommendation is that the root partition should be atleast 25GB.\nThe following packages are required. Install using:\n``` \nsudo apt install \u003cpackage_name\u003e \n```\n\n1.  libzmq3-dev  4.1 or higher\n2.  libprotobuf-dev 2.5 or higher\n3.  librados2  # ceph jewel or higher\n4.  libatasmart-dev\n5.  libssl-dev\n6.  libblkid-dev\n8.  libudev-dev # for building\n9.  librados-dev # for building\n10. pkg-config # for building libudev\n11. libclang-dev\n12. libzmq5\n13. llvm\n14. libdevmapper-dev\n15. liblvm2-dev\n16. liblvm2app2.2\n17. gcc\n18. clang\n19. smartmontools\n20. parted\n\nCLI command to install all the dependencies:\n```\nsudo apt install libzmq3-dev libprotobuf-dev librados2 libatasmart-dev libssl-dev libblkid-dev libudev-dev librados-dev pkg-config libclang-dev llvm libdevmapper-dev liblvm2-dev liblvm2app2.2 gcc clang smartmontools parted\n```\n\n\n### Working Rust environment\n\nInstall Rust and point it to the nightly build. The stable version will not be\nsufficient to run the test cases it needs a feature only available on nightly build. \n \n```\n$ curl https://sh.rustup.rs -sSf | sh\n$ rustup override set nightly\n\n```\n\n### Retrieving source\nLogin to your github account, and checkout the latest source code from \nthis repository. Then, to create executable binary\n\nRun:\n```\n$ cargo build --release\n```\n\nTo check your code without building the binary:\n```\n$ cargo check\n```\n\n## Bynar Workflow\nHardware issues crop up all the time as part of the regular cycle of things in servers.  Bynar\ncan nearly completely automate that maintenance of hard drive failure except for the actual replacing of\nthe drive.  The typical workflow by a human would look something like this:\n1. Receive an alert about a drive failing\n2. SSH over to the server to investigate.  Try to rule out obvious things\n3. Conclude drive is dead and file a support ticket with the datacenter tech to remove it\n   * Or file a ticket with HP/Dell/Cisco/Etc to replace the drive\n4. Depending on the software running on top of this drive I may have to:\n   * Inform the cluster that the drive is dead\n   * Rebalance the data in the cluster\n5. Wait for a replacement\n6. After the drive is replaced inform the clusters that the drive is now back\nin service and rebalance the data back onto the drive.\n\nSo how can Bynar help?  Well it can handle steps 1,2,3,4 and 6.  Nearly everything!\nWhile it is replacing your drives it can also inform you over slack or other channels\nto keep you in the loop.\nThe time saved here multplies with each piece of hardware replaced and now you \ncan focus your time and energy on other things.  It's a positive snowball effect!\n\n\n## Testing\n\nNote that root permissions are required for integration testing.  The reason\nis that the test functions will attempt to create loopback devices, mount them,\ncheck their filesystems etc and all that requires root. The nightly compiler\nis also required for testing because mocktopus makes use of features that \nhaven't landed in stable yet.  Run: `sudo ~/.cargo/bin/cargo test -- --nocapture` to test.\n\n## Support and Contributions\n\nIf you need support, start by checking the [issues] page.\nIf that doesn't answer your questions, or if you think you found a bug,\nplease [file an issue].\n\nThat said, if you have questions, reach out to us\n[communication].\n\nWant to contribute to Bynar? Awesome! Check out the [contributing](https://github.com/Comcast/Bynar/blob/master/Contributing.md) guide.\n\n[Cassandra]: http://cassandra.apache.org/\n[Ceph]: http://docs.ceph.com/docs/master/\n[Hadoop]: http://hadoop.apache.org/\n[Gluster]: https://www.gluster.org/\n[communication]: https://github.com/Comcast/Bynar/issues/new\n[community repository]: https://github.com/Comcast/Bynar\n[file an issue]: https://github.com/Comcast/Bynar/issues/new\n[issues]: https://github.com/Comcast/Bynar/issues\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcomcast%2Fbynar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcomcast%2Fbynar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcomcast%2Fbynar/lists"}