{"id":21888147,"url":"https://github.com/chorusone/blazar","last_synced_at":"2025-07-15T15:39:19.621Z","repository":{"id":262404988,"uuid":"877769302","full_name":"ChorusOne/blazar","owner":"ChorusOne","description":"Automatic Cosmos SDK Network Upgrades","archived":false,"fork":false,"pushed_at":"2025-04-14T10:51:18.000Z","size":48412,"stargazers_count":25,"open_issues_count":13,"forks_count":3,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-15T10:16:38.525Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ChorusOne.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-24T08:00:37.000Z","updated_at":"2025-04-10T12:22:59.000Z","dependencies_parsed_at":"2025-03-28T14:37:05.424Z","dependency_job_id":null,"html_url":"https://github.com/ChorusOne/blazar","commit_stats":null,"previous_names":["chorusone/blazar"],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChorusOne%2Fblazar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChorusOne%2Fblazar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChorusOne%2Fblazar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChorusOne%2Fblazar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ChorusOne","download_url":"https://codeload.github.com/ChorusOne/blazar/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249048742,"owners_count":21204306,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-28T11:14:22.222Z","updated_at":"2025-04-15T10:17:19.999Z","avatar_url":"https://github.com/ChorusOne.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"#\"\u003e\n    \u003cimg src=\"https://github.com/user-attachments/assets/f5a48a09-b3ce-41d8-8fe9-3937da45038f\" alt=\"Logo\" width=\"192\" height=\"192\"\u003e\n  \u003c/a\u003e\n\n  \u003ch3 align=\"center\"\u003eBlazar: Automatic Cosmos SDK Network Upgrades\u003c/h3\u003e\n\n  \u003cp align=\"center\"\u003e\n    Life is too short to wait for the upgrade block height!\n    \u003cbr /\u003e\n    \u003cbr /\u003e\n    \u003ca href=\"#getting-started\"\u003eGetting Started\u003c/a\u003e\n    ·\n    \u003ca href=\"#cli--rest-interface\"\u003eCLI\u003c/a\u003e\n    ·\n    \u003ca href=\"#what-is-blazar\"\u003eWeb UI\u003c/a\u003e\n    ·\n    \u003ca href=\"#proxy-ui\"\u003eProxy UI\u003c/a\u003e\n    ·\n    \u003ca href=\"#slack-integration\"\u003eSlack\u003c/a\u003e\n    ·\n    \u003ca href=\"#frequently-asked-questions\"\u003eFAQ\u003c/a\u003e\n  \u003c/p\u003e\n\u003c/div\u003e\n\n## What is Blazar?\nBlazar is a standalone application designed to automate network upgrades for [Cosmos SDK](https://github.com/cosmos/cosmos-sdk) based blockchain networks.\n\n![Web UI](https://github.com/user-attachments/assets/3250a2d6-2bb7-4c1e-bc89-e8d15736300c)\n\n## The Need for Blazar\nAt [Chorus One](https://chorus.one), we manage over 60 blockchain networks, many of which are part of the Cosmos Ecosystem. Each network has its own upgrade schedule, which can vary from monthly to bi-weekly, depending on the urgency of the upgrade and Cosmos SDK releases. Our 24/7 on-call team handles multiple upgrades weekly.\n\nThe upgrade process is generally straightforward but can be time-consuming. Here's how it typically works:\n1. An upgrade is announced via a governance proposal or other communication channels (Discord, Telegram, Slack, Email, etc.).\n2. The upgrade details specify the block height and the version network operators should use.\n3. At the specified block height, the node halts, and operators must upgrade the binary and restart the node(s).\n4. While waiting for consensus, operators often engage in progress updates on Discord.\n5. Once the upgrade is successful, operators return to their regular tasks.\n\nBlazar was created to automate this process, allowing our team to use their time more productively. It currently handles the majority of network upgrades for Cosmos Networks at Chorus One.\n\n## Key Features\n- **Upgrade Feeds:** Fetch upgrade information from multiple sources like \"Governance\", \"Database\", and \"Local\".\n- **Upgrade Strategies:** Supports various upgrade scenarios, including height-specific and manually coordinated upgrades.\n- **Pre and Post Upgrade Checks:** Automate checks like docker image existence, node and consensus status.\n- **Stateful Execution:** Tracks upgrade stages to ensure consistent execution flow.\n- **Cosmos SDK Gov/Upgrade Module Compliance:** Understands and respects the Cosmos SDK governance module upgrades.\n- **Slack Integration:** Optional Slack notifications for every action Blazar performs.\n- **Modern Stack:** Includes CLI, UI, REST, gRPC, Prometheus metrics, and Protobuf.\n- **Built by Ops Team:** Developed by individuals with firsthand experience in node operations.\n\n## Comparison to Cosmovisor\nWhile many operators use [Cosmovisor](https://docs.cosmos.network/main/build/tooling/cosmovisor) with systemd services, this setup doesn't meet our specific needs. Instead of relying on GitHub releases, we [build our own binaries](https://handbook.chorus.one/node-software/build-process.html), ensuring a consistent build environment with Docker. This approach allows us to use exact software versions and generate precise build artifacts (e.g., libwasmvm.so).\n\nCosmovisor is designed to run as the parent process of a validator node, replacing node binaries at the upgrade height. However, this model isn't compatible with Docker Compose managed services. To address this, we developed Blazar as a more effective solution tailored to our setup.\n\n**Note:** If you'd like Blazar to work with systemd services, contributions are welcome!\n\n|                   \t| Blazar                                     \t| Cosmovisor                      \t|\n|-------------------\t|--------------------------------------------\t|---------------------------------\t|\n| Control plane     \t| Docker                                     \t| [Fork/Exec](https://docs.cosmos.network/main/build/tooling/cosmovisor#design)                       \t|\n| Upgrade mechanism \t| Image Tag Update                           \t| Replace Binary                  \t|\n| Configuration     \t| TOML (Blazar) + YAML (docker-compose.yml)  \t| [Custom directory structure](https://docs.cosmos.network/main/build/tooling/cosmovisor#folder-layout) \t|\n| Upgrade strategy  \t| Gov, Coordinated, Uncoordinated           \t| [Gov, Coordinated, Uncoordinated](https://docs.cosmos.network/main/build/tooling/cosmovisor#adding-upgrade-binary)   |\n| Upgrade scope     \t| Single, Multi-node*                        \t| Single node                      \t|\n| Pre checks        \t| :heavy_check_mark:                          | :heavy_check_mark: (preupgrade.sh)              \t|\n| Post checks       \t| :heavy_check_mark:                          | :x:                              \t|\n| Metrics            \t| :heavy_check_mark:                        \t| :x:                              \t|\n| Notifications     \t| :heavy_check_mark: (Slack)                 \t| :x:                              \t|\n| UI + REST + RPC   \t| :heavy_check_mark:                         \t| :x:                              \t|\n| CLI               \t| :heavy_check_mark:                          | :heavy_check_mark:               \t|\n| Upgrade Feeds     \t| Governance, Database, Local                \t| Governance, Local**                |\n\n\\* `DATABASE` registered upgrades are executed by multiple nodes feeding from the provider\n\n\\** For Cosmovisor everything looks as [if it was scheduled through governance](https://docs.cosmos.network/main/build/tooling/cosmovisor#detecting-upgrades)\n\n## How Blazar Works\n![Blazar Under the Hood](https://github.com/user-attachments/assets/1ecb0629-24bc-472f-a9cf-4d8d5e271a86)\n\nBlazar constructs a prioritized list of upgrades from multiple providers and takes appropriate actions based on the most recent state. It retrieves block heights from WSS endpoints or periodic gRPC polls and triggers Docker components when the upgrade height is reached. Notifications are sent to logs and Slack (if configured).\n\nIn simple terms, Blazar performs the following steps:\n1. **Upgrade List Construction:** Blazar compiles a unified list of upgrades from various providers (database, local, chain), resolving priorities based on the highest precedence.\n2. **State Evaluation \u0026 Action:** The Blazar daemon reads this list in conjunction with the most recent state, taking relevant actions, such as performing a pre-upgrade check or finalizing the upgrade process.\n3. **Block Height Detection:** The daemon tracks block heights via WSS endpoints or periodic gRPC polls.\n4. **Upgrade Execution:** When the upgrade height is reached, the corresponding Docker components are executed.\n5. **Notification Delivery:** Blazar sends notifications to logs and Slack (if configured).\n\nWhile the logic is simple, it's important to understand the differences between the types of upgrades:\n1. **Governance:** A coordinated upgrade initiated by chain governance, expected to be executed by all validators at a specified block height.\n2. **Non Governance Coordinated:** An upgrade initiated by operators, not by the chain, but it is expected to occur at the same block height across all validators.\n3. **Non Governance Uncoordinated:** An operator-initiated upgrade, independent of chain governance, that can be executed at any time.\n\nNOTE: Blazar does one job and does it well, meaning you need one Blazar instance per Cosmos-SDK node.\n\nNOTE: You are free to choose your upgrade proposal providers. An SQL database is not mandatory - you can opt to use the \"LOCAL\" provider or both simultaneously, depending on your needs.\n\n## Getting Started\nTo use Blazar, first build the binary with the Go compiler, then deploy it on a host with Docker Compose installed.\n```sh\n$ apt-get install golang\n$ apt-get install docker-compose\n```\n\nConfigure and run Blazar:\n```sh\n$ cp blazar.sample.toml blazar.toml\n$ make build\n$ ./blazar run --config blazar.toml\n```\n\n### Requirements: Docker \u0026 Docker Compose\nBlazar is designed to work with nodes configured and spawned via Docker Compose.\n\n### CLI \u0026 REST Interface\nRegister or list upgrades using the CLI:\n```sh\n$ ./blazar upgrades list --host 127.0.0.1 --port 5678\n... table with upgrades ...\n\n$ ./blazar upgrades register --height \"13261400\" --tag '4.2.0' --type NON_GOVERNANCE_COORDINATED --source DATABASE --host 127.0.0.1 --port 5678 --name 'security upgrade'\n```\n\nOr use the REST interface:\n```\ncurl -s http://127.0.0.1:1234/v1/upgrades/list\n```\n\n### Slack Integration\nTrack the upgrade process in a single Slack thread 🧵.\n\n![Slack Notifications](https://github.com/user-attachments/assets/f59139e8-cdf9-4cd1-87bf-e5b1c0a667a7)\n\n### Proxy UI\nBlazar Proxy consolidates the latest updates from all Blazar instances. Here's how you can run it:\n```\n$ cp proxy.sample.toml proxy.toml\n$ ./blazar proxy --config proxy.toml\n```\n\n![Proxy UI](https://github.com/user-attachments/assets/2b80e96e-5e9a-4d59-ae9e-8479c8fdee81)\n\n\n## Frequently Asked Questions\n\u003cdetails\u003e\n  \u003csummary\u003eWhy do I need to register a version tag separately?\u003c/summary\u003e\n  \nCosmos-SDK Software Upgrade Proposals don't explicitly specify the version you must upgrade to. It can be derived from the rich text data within the proposal, such as:\n1. A link to the binary release (if present).\n2. The proposal title.\n3. The human-written text.\n\nCurrently, Blazar does not infer which version should be used. As a network operator, you must provide a version tag; otherwise, Blazar will skip the upgrade.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eWhat are the upgrade priorities, and why do I need them?\u003c/summary\u003e\n\nConsider a scenario where a network operator runs three nodes. The first node uses an image with a patch (e.g., PebbleDB support), while the other two run vanilla upstream images.\n\nIn this configuration, Blazar uses three upgrade sources:\n* CHAIN (priority 1)\n* DATABASE (priority 2)\n* LOCAL (priority 3)\n\nAll three Blazar instances detect a new upgrade from CHAIN. The operator registers a new version in the DATABASE so that every instance knows what to pick up. However, one node requires a patched version. The network operator must register a new version in the LOCAL provider.\n\nNow, the first node sees two different versions from two providers (DATABASE \u0026 LOCAL). Which one should it use?\n**The one with the higher priority**\n\nThe end state on each Blazar node is:\n1. Node 1 - v1.0.0-patched, priority 3\n2. Nodes 2 \u0026 3 - v1.0.0, priority 2\n\nThe same logic applies to upgrade entries and versions.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eWhat happens if I don't register a version tag for an upgrade?\u003c/summary\u003e\n\nBlazar will skip the upgrade.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eBlazar doesn't display any upgrades?\u003c/summary\u003e\n\nBlazar maintains its own state of all upgrades, which is periodically refreshed at the interval specified in your configuration. If you don't see the upgrades, it is likely that you need to wait for the given interval for Blazar to update the state.\n\nNOTE: Adding a new version or upgrade via CLI/UI will trigger a state update.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eThe upgrade governance proposal passed, but the upgrade is still in the 'SCHEDULED' state?\u003c/summary\u003e\n\nBlazar will change the upgrade state from 'SCHEDULED' to 'ACTIVE' when the voting period is over.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eWhat is the purpose of the 'force cancel' flag?\u003c/summary\u003e\n\nThere are two ways to cancel an upgrade in Blazar. The standard method creates a `cancellation entry` in the provider storage, such as an SQL database, if no upgrade is registered. Otherwise, it updates the upgrade status field to `CANCELLED` for the upgrade with the highest priority.\n\nBlazar periodically fetches and updates the list of upgrades at the interval specified in your configuration. But what if you need to cancel the upgrade immediately and can't wait for the next fetch? For such uncommon scenarios, you can use the `force cancel` mode, which sets the `CANCELLED` status directly in the Blazar state machine.\n\nThe force mode works per Blazar instance, so if you have, say, 3 nodes, you would need to force cancel all three via CLI/UI/RPC calls. If you use the `DATABASE` provider, you can simply cancel the upgrade for everyone, but you need to wait for Blazar to pick it up.\n\nTo simplify, think of the `force cancel` as the last line of defense. It is unlikely that you will need it, but it's there just in case.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eI registered a new upgrade, but only one node is 'up to date'?\u003c/summary\u003e\n\nRemember that Blazar refreshes its internal state periodically. If you registered a new upgrade on one instance with the 'DATABASE' provider and the other node doesn't see it, you have two options:\n1. Wait for Blazar to sync (see 'Time to next sync' in the UI).\n2. Force sync via UI/CLI/RPC call.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eDoes Blazar work with chains with non-standard gov module (e.g., Neutron)?\u003c/summary\u003e\n\nYes, but you'll need to register manually a `GOVERNANCE` type upgrade in `LOCAL` or `DATABASE` provider.\n\nNeutron is a smart contract chain that implements its own governance (DAO DAO) via an on-chain contract. Blazar currently doesn't understand the custom smart contract logic, therefore the operator cannot use the `CHAIN` provider.\nHowever, the Neutron governance is integrated with Cosmos SDK upgrades module and will output the `upgrade-info.json` at the upgrade height. Therefore from Blazar perspective, the `GOVERNANCE` type is valid, but the source provide must be different.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eWhat is the difference between 'compose-file' and 'env-file' upgrade mode?\u003c/summary\u003e\n\nWhen performing a node upgrade, Blazar updates the docker version tag (e.g., `v1.0.0` to `v2.0.0`). That version is stored in the `docker-compose.yaml` file in the following form:\n```\n$ cat docker-compose.yaml | grep 'image'\nimage: \u003cclient_id\u003e.dkr.ecr.us-east-1.amazonaws.com/chorusone/archway:v1.0.0\n```\n\nor in the `.env` file:\n```\n$ cat docker-compose.yaml | grep 'image'\nimage: \u003cclient_id\u003e.dkr.ecr.us-east-1.amazonaws.com/chorusone/archway:${VERSION_archway}\n\n$ cat .env\nVERSION_archway=1.0.0\n```\n\n\u003e Why do we support both upgrade modes and which one is better?\n\nThe `compose-file` is simpler, but we highly recommend the `env-file` mode. If the version tag is stored in the `.env` file, the blast radius of possible mistakes is very low, unlike editing the whole `docker-compose.yaml` to replace one single variable.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eWhat is the purpose of SQL migration files?\u003c/summary\u003e\n\nThis question is relevant for anyone who wants to use the DATABASE provider.\n\nBlazar leverages [GORM](https://gorm.io) to manage SQL databases. If you enable automatic migrations by setting:\n```toml\n[upgrade-registry.provider.database]\nauto-migrate = true\n```\n\nyou can disregard the `migrations` files since [GORM](https://gorm.io) will automatically initialize all necessary SQL tables.\n\nHowever, if `auto-migrate` is disabled, you'll need to manually apply the migration SQL statements.\n\u003c/details\u003e\n\n## License\nBlazar is licensed under the Apache 2.0 License. For more detailed information, please refer to the LICENSE file in the repository.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchorusone%2Fblazar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchorusone%2Fblazar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchorusone%2Fblazar/lists"}