{"id":13647725,"url":"https://github.com/bottlerocket-os/bottlerocket-update-operator","last_synced_at":"2025-11-12T21:47:23.780Z","repository":{"id":37178109,"uuid":"240619080","full_name":"bottlerocket-os/bottlerocket-update-operator","owner":"bottlerocket-os","description":"A Kubernetes operator for automated updates to Bottlerocket","archived":false,"fork":false,"pushed_at":"2025-11-05T17:58:27.000Z","size":2186,"stargazers_count":206,"open_issues_count":52,"forks_count":49,"subscribers_count":24,"default_branch":"develop","last_synced_at":"2025-11-12T21:47:17.506Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bottlerocket-os.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE-APACHE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":"COPYRIGHT","agents":null,"dco":null,"cla":null}},"created_at":"2020-02-14T23:57:54.000Z","updated_at":"2025-11-05T17:58:33.000Z","dependencies_parsed_at":"2023-10-13T08:42:35.748Z","dependency_job_id":"d6264d70-e7d6-45d3-be62-28ec92070ba0","html_url":"https://github.com/bottlerocket-os/bottlerocket-update-operator","commit_stats":{"total_commits":577,"total_committers":31,"mean_commits":"18.612903225806452","dds":0.7244367417677643,"last_synced_commit":"2fcbeb120679afa7e53f0cc3a33ada6d00e20f3c"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":"amazon-archives/__template_Custom","purl":"pkg:github/bottlerocket-os/bottlerocket-update-operator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bottlerocket-os%2Fbottlerocket-update-operator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bottlerocket-os%2Fbottlerocket-update-operator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bottlerocket-os%2Fbottlerocket-update-operator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bottlerocket-os%2Fbottlerocket-update-operator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bottlerocket-os","download_url":"https://codeload.github.com/bottlerocket-os/bottlerocket-update-operator/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bottlerocket-os%2Fbottlerocket-update-operator/sbom","scorecard":{"id":248952,"data":{"date":"2025-08-11","repo":{"name":"github.com/bottlerocket-os/bottlerocket-update-operator","commit":"acbf3ab23809fdbd30a9e7a85c97729aa1d6e297"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":5.3,"checks":[{"name":"Security-Policy","score":10,"reason":"security policy file detected","details":["Info: security policy file detected: SECURITY.md:1","Info: Found linked content: SECURITY.md:1","Info: Found disclosure, vulnerability, and/or timelines in security policy: SECURITY.md:1","Info: Found text in security policy: SECURITY.md:1"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Code-Review","score":10,"reason":"all changesets reviewed","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/ci.yml:1","Warn: topLevel 'contents' permission set to 'write': .github/workflows/deploy-helm-charts.yaml:13","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":9,"reason":"11 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 9","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:29: update your workflow using https://app.stepsecurity.io/secureworkflow/bottlerocket-os/bottlerocket-update-operator/ci.yml/develop?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:44: update your workflow using https://app.stepsecurity.io/secureworkflow/bottlerocket-os/bottlerocket-update-operator/ci.yml/develop?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/deploy-helm-charts.yaml:20: update your workflow using https://app.stepsecurity.io/secureworkflow/bottlerocket-os/bottlerocket-update-operator/deploy-helm-charts.yaml/develop?enable=pin","Warn: containerImage not pinned by hash: Dockerfile:2","Info:   0 out of   3 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   1 containerImage dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":9,"reason":"license file detected","details":["Info: project has a license file: LICENSE-APACHE:0","Warn: project license file does not contain an FSF or OSI license."],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":0,"reason":"Project has not signed or included provenance with any releases.","details":["Warn: release artifact v1.6.0 not signed: https://api.github.com/repos/bottlerocket-os/bottlerocket-update-operator/releases/221911232","Warn: release artifact v1.5.0 not signed: https://api.github.com/repos/bottlerocket-os/bottlerocket-update-operator/releases/202127096","Warn: release artifact v1.4.0 not signed: https://api.github.com/repos/bottlerocket-os/bottlerocket-update-operator/releases/163048504","Warn: release artifact v1.3.0 not signed: https://api.github.com/repos/bottlerocket-os/bottlerocket-update-operator/releases/117695309","Warn: release artifact v1.2.0 not signed: https://api.github.com/repos/bottlerocket-os/bottlerocket-update-operator/releases/112563535","Warn: release artifact v1.6.0 does not have provenance: https://api.github.com/repos/bottlerocket-os/bottlerocket-update-operator/releases/221911232","Warn: release artifact v1.5.0 does not have provenance: https://api.github.com/repos/bottlerocket-os/bottlerocket-update-operator/releases/202127096","Warn: release artifact v1.4.0 does not have provenance: https://api.github.com/repos/bottlerocket-os/bottlerocket-update-operator/releases/163048504","Warn: release artifact v1.3.0 does not have provenance: https://api.github.com/repos/bottlerocket-os/bottlerocket-update-operator/releases/117695309","Warn: release artifact v1.2.0 does not have provenance: https://api.github.com/repos/bottlerocket-os/bottlerocket-update-operator/releases/112563535"],"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 30 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":5,"reason":"5 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: RUSTSEC-2025-0012","Warn: Project is vulnerable to: RUSTSEC-2024-0388","Warn: Project is vulnerable to: RUSTSEC-2024-0384","Warn: Project is vulnerable to: GHSA-2gh3-rmm4-6rq5","Warn: Project is vulnerable to: RUSTSEC-2024-0437"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-17T08:07:01.581Z","repository_id":37178109,"created_at":"2025-08-17T08:07:01.581Z","updated_at":"2025-08-17T08:07:01.581Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":284115869,"owners_count":26949957,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-12T02:00:06.336Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T01:03:44.545Z","updated_at":"2025-11-12T21:47:23.773Z","avatar_url":"https://github.com/bottlerocket-os.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# Bottlerocket Update Operator\n\nThe Bottlerocket update operator (or, for short, Brupop) is a [Kubernetes operator](https://Kubernetes.io/docs/concepts/extend-Kubernetes/operator/) that coordinates Bottlerocket updates on hosts in a cluster.\nWhen installed, the Bottlerocket update operator starts a controller deployment on one node, an agent daemon set on every Bottlerocket node, and an Update Operator API Server deployment.\nThe controller orchestrates updates across your cluster, while the agent is responsible for periodically querying for Bottlerocket updates, draining the node, and performing the update when asked by the controller.\nThe agent performs all cluster object mutation operations via the API Server, which performs additional authorization using the Kubernetes TokenReview API -- ensuring that any request associated with a node is being made by the agent pod running on that node.\nFurther, `cert-manager` is required in order for the API server to use a CA certificate to communicate over SSL with the agents.\nUpdates to Bottlerocket are rolled out in [waves](https://github.com/bottlerocket-os/twoliter/tree/develop/twoliter/embedded/waves) to reduce the impact of issues; the nodes in your cluster may not all see updates at the same time.\n\nFor a deep dive on installing Brupop, how it works, and its integration with Bottlerocket, [check out this design deep dive document!](./design/1.0.0-release.md)\n\n## Getting Started\n\n### Version Support Policy\n\nAs per our policy, Brupop follows the semantic versioning (semver) principles, ensuring that any updates in minor versions do not introduce any breaking or backward-incompatible changes. However, please note that we only provide security patches for the latest minor version. Therefore, it is highly recommended to always keep your Brupop installation up to date with the latest available version.\n\nFor example: If `v1.3.0` is the latest Brupop release, then, `v1.3` (latest minor version) will be considered as supported and `v1.3.0` (latest available version) will be the recommended version of Brupop to be installed. When `v1.3.1` is released, then that version will be considered as recommended.\n\n### Installation\n\n1. Brupop uses [cert-manager](https://cert-manager.io) to manage self-signed certificates used by the Bottlerocket update operator. To install cert-manager:\n\n```sh\nkubectl apply -f \\\n  https://github.com/cert-manager/cert-manager/releases/download/v1.8.2/cert-manager.yaml\n```\n\nOr if you're using helm:\n\n```sh\n# Add the cert-manager helm chart\nhelm repo add jetstack https://charts.jetstack.io\n\n# Update your local chart cache with the latest\nhelm repo update\n\n# Install the cert-manager (including its CRDs)\nhelm install \\\n  cert-manager jetstack/cert-manager \\\n  --namespace cert-manager \\\n  --create-namespace \\\n  --version v1.8.2 \\\n  --set installCRDs=true\n```\n\n2. The Bottlerocket update operator can then be installed using helm:\n\n```sh\n# Add the bottlerocket-update-operator chart\nhelm repo add brupop https://bottlerocket-os.github.io/bottlerocket-update-operator\n\n# Update your local chart cache with the latest updates\nhelm repo update\n\n# Create a namespace\nkubectl create namespace brupop-bottlerocket-aws\n\n# Install the brupop CRD\nhelm install brupop-crd brupop/bottlerocket-shadow\n\n# Install the brupop operator\nhelm install brupop-operator brupop/bottlerocket-update-operator\n```\n\nThis will create the custom resource definition, roles, deployments, etc., and use the latest update operator image available in [Amazon ECR Public](https://gallery.ecr.aws/bottlerocket/bottlerocket-update-operator).\n\nAlternatively, you can use the pre-baked manifest, with all the default values, [found at the root of this repository (named `bottlerocket-update-operator.yaml`).](./bottlerocket-update-operator.yaml)\nThis YAML manifest file is also attached to each release [found in the GitHub releases page.](https://github.com/bottlerocket-os/bottlerocket-update-operator/releases)\n\nNote: The generated manifests points to the latest version of the Update Operator, `v1.1.0`.\nBe sure to use the release for the Update Operator release that you plan to deploy.\n\n3. Label nodes with `bottlerocket.aws/updater-interface-version=2.0.0` to indicate they should be automatically updated.\nOnly bottlerocket nodes with this label will be updated. For more information on labeling, refer to the [Label nodes section of this readme.](https://github.com/bottlerocket-os/bottlerocket-update-operator#label-nodes)\n\n```sh\nkubectl label node MY_NODE_NAME bottlerocket.aws/updater-interface-version=2.0.0\n```\n\n### Configuration\n\n#### Configure via Helm values yaml file\n\nYou can use a values file when installing brupop with helm (via the `--values / -f` flag) to configure how brupop functions:\n\n```yaml\n# Default values for bottlerocket-update-operator.\n\n# The namespace to deploy the update operator into\nnamespace: \"brupop-bottlerocket-aws\"\n\n# The image to use for brupop\n# This defaults to the image built alongside a particular helm chart, but you can override it by uncommenting this line:\n# image: \"public.ecr.aws/bottlerocket/bottlerocket-update-operator:v1.8.0\"\n\n# Placement controls\n# See the Kubernetes documentation about placement controls for more details:\n# * https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/\n# * https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector\n# * https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity\nplacement:\n  agent:\n    # The agent is a daemonset, so the only controls that apply to it are tolerations.\n    tolerations: []\n\n  controller:\n    tolerations: []\n    nodeSelector: {}\n    podAffinity: {}\n    podAntiAffinity: {}\n\n  apiserver:\n    tolerations: []\n    nodeSelector: {}\n    podAffinity: {}\n    # By default, apiserver pods prefer not to be scheduled to the same node.\n    podAntiAffinity:\n      preferredDuringSchedulingIgnoredDuringExecution:\n        - weight: 1\n          podAffinityTerm:\n            labelSelector:\n              matchExpressions:\n              - key: brupop.bottlerocket.aws/component\n                operator: In\n                values:\n                - apiserver\n            topologyKey: kubernetes.io/hostname\n\n# If testing against a private image registry, you can set the pull secret to fetch images.\n# This can likely remain as `brupop` so long as you run something like the following:\n# kubectl create secret docker-registry brupop \\\n#  --docker-server 109276217309.dkr.ecr.us-west-2.amazonaws.com \\\n#  --docker-username=AWS \\\n#  --docker-password=$(aws --region us-west-2 ecr get-login-password) \\\n#  --namespace=brupop-bottlerocket-aws\n#imagePullSecrets:\n#  - name: \"brupop\"\n\n# External load balancer setting.\n# When `exclude_from_lb_wait_time_in_sec` is set to a positive value\n# brupop will exclude the node from load balancing\n# and will wait for `exclude_from_lb_wait_time_in_sec` seconds before draining nodes.\n# Under the hood, this uses the `node.kubernetes.io/exclude-from-external-load-balancers` label\n# to exclude those nodes from load balancing.\nexclude_from_lb_wait_time_in_sec: \"0\"\n\n# Concurrent update nodes setting.\n# When `max_concurrent_updates` is set to a positive integer value,\n# brupop will concurrently update max `max_concurrent_updates` nodes.\n# When `max_concurrent_updates` is set to \"unlimited\",\n# brupop will concurrently update all nodes with respecting `PodDisruptionBudgets`\n# Note: the \"unlimited\" option does not work well with `exclude_from_lb_wait_time_in_sec`\n# option, which could potential exclude all nodes from load balancer at the same time.\nmax_concurrent_updates: \"1\"\n\n# DEPRECATED: use the scheduler settings\n# Start and stop times for update window\n# Brupop will operate node updates within update time window.\n# when you set up time window start and stop time, you should use UTC (24-hour time notation).\nupdate_window_start: \"0:0:0\"\nupdate_window_stop: \"0:0:0\"\n\n# Scheduler setting\n# Brupop will operate node updates on scheduled maintenance windows by using cron expressions.\n# When you set up the scheduler, you should follow cron expression rules.\n# ┌───────────── seconds (0 - 59)\n# │ ┌───────────── minute (0 - 59)\n# │ │ ┌───────────── hour (0 - 23)\n# │ │ │ ┌───────────── day of the month (1 - 31)\n# │ │ │ │ ┌───────────── month (Jan, Feb, Mar, Apr, Jun, Jul, Aug, Sep, Oct, Nov, Dec)\n# │ │ │ │ │ ┌───────────── day of the week (Mon, Tue, Wed, Thu, Fri, Sat, Sun)\n# │ │ │ │ │ │ ┌───────────── year (formatted as YYYY)\n# │ │ │ │ │ │ │\n# │ │ │ │ │ │ │\n# * * * * * * *\nscheduler_cron_expression: \"* * * * * * *\"\n\n# API server ports\n# The port the API server uses for its own operations. This is accessed by the controller,\n# the bottlerocket-shadow daemonset, etc.\napiserver_internal_port: \"8443\"\n# API server internal address where the CRD version conversion webhook is served\napiserver_service_port: \"443\"\n\nlogging:\n  # Formatter for the logs emitted by brupop.\n  # Options are:\n  # * full - Human-readable, single-line logs\n  # * compact - A variant of full optimized for shorter line lengths\n  # * pretty - \"Excessively pretty\" logs optimized for human-readable terminal output.\n  # * json - Newline-delimited JSON-formatted logs.\n  formatter: \"pretty\"\n  # Whether or not to enable ANSI colors on log messages.\n  # Makes the output \"pretty\" in terminals, but may add noise to web-based log utilities.\n  ansi_enabled: \"true\"\n\n  # Controls the filter for tracing/log messages.\n  # This can be as simple as a log-level (e.g. \"info\", \"debug\", \"error\"), but also supports more complex directives.\n  # See https://docs.rs/tracing-subscriber/0.3.17/tracing_subscriber/filter/struct.EnvFilter.html#directives\n  controller:\n    tracing_filter: \"info\"\n  agent:\n    tracing_filter: \"info\"\n  apiserver:\n    tracing_filter: \"info\"\n\n# Provide pod level labels for the brupop resources.\npodLabels: {}\n```\n\n#### Configuration via Kubernetes yaml\n\n##### Configure API server ports\n\nIf you'd like to configure what ports the API server uses,\nadjust the value that is consumed in the container environment:\n\n```yaml\n      ...\n      containers:\n        - command:\n            - \"./api-server\"\n          env:\n            - name: APISERVER_INTERNAL_PORT\n              value: 999\n```\n\nYou'll then also need to adjust the various \"port\" entries in the YAML manifest\nto correctly reflect what port the API server starts on and expects for its service port:\n\n```yaml\n    ...\n    webhook:\n      clientConfig:\n        service:\n          name: brupop-apiserver\n          namespace: brupop-bottlerocket-aws\n          path: /crdconvert\n          port: 123\n```\n\nThe default values are generated from the [default values yaml file](https://github.com/bottlerocket-os/bottlerocket-update-operator/blob/develop/deploy/bottlerocket-update-operator/values.yaml) file:\n- `apiserver_internal_port: \"8443\"` - This is the container port the API server starts on.\n  If this environment variable is _not_ found, the Brupop API server will fail to start.\n- `apiserver_service_port: \"443\"` - This is the port Brupop's Kubernetes Service uses to target the internal API Server port.\n  It is used by the node agents to access the API server.\n  If this environment variable is _not_ found, the Brupop agents will fail to start.\n\n##### Resource Requests \u0026 Limits\n\nThe `bottlerocket-update-operator.yaml` manifest makes several default recommendations for\nKubernetes resource requests and limits. In general, the update operator and its components are lite-weight\nand shouldn't consume more than 10m CPU (which is roughly equivalent to 1/100th of a CPU core)\nand 50Mi (which is roughly equivalent to 0.05 GB of memory).\nIf this limit is breached, the Kubernetes API will restart the faulting container.\n\nNote that your mileage with these resource requests and limits may vary.\nAny number of factors may contribute to varying results in resource utilization (different compute instance types, workload utilization, API ingress/egress, etc).\n[The Kubernetes documentation for Resource Management of Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)\nis an excellent resource for understanding how various compute resources are utilized\nand how Kubernetes manages these resources.\n\nIf resource utilization by the brupop components is not a concern,\nremoving the `resources` fields in the manifest will not affect the functionality of any components.\n\n##### Exclude Nodes from Load Balancers Before Draining\nThis configuration uses Kubernetes [ServiceNodeExclusion](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) feature.\n`EXCLUDE_FROM_LB_WAIT_TIME_IN_SEC` can be used to enable the feature to exclude the node from load balancer before draining.\nWhen `EXCLUDE_FROM_LB_WAIT_TIME_IN_SEC` is 0 (default), the feature is disabled.\nWhen `EXCLUDE_FROM_LB_WAIT_TIME_IN_SEC` is set to a positive integer, bottlerocket update operator will exclude the node from\nload balancer and then wait `EXCLUDE_FROM_LB_WAIT_TIME_IN_SEC` seconds before draining the pods on the node.\n\nTo enable this feature, set the `exclude_from_lb_wait_time_in_sec` value in your helm values yaml file to a positive integer. For example,\n`exclude_from_lb_wait_time_in_sec: \"100\"`.\n\nOtherwise, go to `bottlerocket-update-operator.yaml` and change `EXCLUDE_FROM_LB_WAIT_TIME_IN_SEC` to a positive integer value.\nFor example:\n```yaml\n      ...\n      containers:\n        - command:\n            - \"./agent\"\n          env:\n            - name: MY_NODE_NAME\n              valueFrom:\n                fieldRef:\n                  fieldPath: spec.nodeName\n            - name: EXCLUDE_FROM_LB_WAIT_TIME_IN_SEC\n              value: \"180\"\n      ...\n```\n\n##### Set Up Max Concurrent Update\n\n`MAX_CONCURRENT_UPDATE` can be used to specify the max concurrent updates during updating.\nWhen `MAX_CONCURRENT_UPDATE` is a positive integer, the bottlerocket update operator\nwill concurrently update up to `MAX_CONCURRENT_UPDATE` nodes respecting [PodDisruptionBudgets](https://kubernetes.io/docs/tasks/run-application/configure-pdb/).\nWhen `MAX_CONCURRENT_UPDATE` is set to `unlimited`, bottlerocket update operator\nwill concurrently update all nodes respecting [PodDisruptionBudgets](https://kubernetes.io/docs/tasks/run-application/configure-pdb/).\n\nNote: The `MAX_CONCURRENT_UPDATE` configuration does not work well with `EXCLUDE_FROM_LB_WAIT_TIME_IN_SEC`\nconfiguration, especially when `MAX_CONCURRENT_UPDATE` is set to `unlimited`, it could potentially exclude all\nnodes from load balancer at the same time.\n\nTo enable this feature, set the `max_concurrent_updates` value in your helm values yaml file to a positive integer value or `unlimited`. For example,\n`max_concurrent_updates: \"1\"` or `max_concurrent_updates: \"unlimited\"`.\n\nOtherwise, go to `bottlerocket-update-operator.yaml` and change `MAX_CONCURRENT_UPDATE` to a positive integer value or `unlimited`.\nFor example:\n```yaml\n      containers:\n        - command:\n            - \"./controller\"\n          env:\n            - name: MY_NODE_NAME\n              valueFrom:\n                fieldRef:\n                  fieldPath: spec.nodeName\n            - name: MAX_CONCURRENT_UPDATE\n              value: \"1\"\n```\n\n##### Set scheduler\n`SCHEDULER_CRON_EXPRESSION` can be used to specify the scheduler in which updates are permitted.\nWhen `SCHEDULER_CRON_EXPRESSION` is \"* * * * * * *\" (default), the feature is disabled.\n\nTo enable this feature, set the `scheduler_cron_expression` value in your helm values yaml file.\nThis value should be a valid cron expression.\nA cron expression can be configured to a time window or a specific trigger time.\nWhen users specify cron expressions as a time window, the bottlerocket update operator will operate node updates within that update time window.\nWhen users specify cron expression as a specific trigger time, brupop will update and complete all waitingForUpdate nodes on the cluster when that time occurs.\n```\n# ┌───────────── seconds (0 - 59)\n# | ┌───────────── minute (0 - 59)\n# | │ ┌───────────── hour (0 - 23)\n# | │ │ ┌───────────── day of the month (1 - 31)\n# | │ │ │ ┌───────────── month (Jan, Feb, Mar, Apr, Jun, Jul, Aug, Sep, Oct, Nov, Dec)\n# | │ │ │ │ ┌───────────── day of the week (Mon, Tue, Wed, Thu, Fri, Sat, Sun)\n# | │ │ │ │ │ ┌───────────── year (formatted as YYYY)\n# | │ │ │ │ │ |\n# | │ │ │ │ │ |\n# * * * * * * *\n```\n\nNote: brupop uses Coordinated Universal Time(UTC), please convert your local time to Coordinated Universal Time (UTC). This tool [Time Zone Converter](https://www.timeanddate.com/worldclock/converter.html) can help you find your desired time window on UTC.\nFor example (schedule to run update operator at 03:00 PM on Monday ):\n```yaml\n      containers:\n        - command:\n            - \"./controller\"\n          env:\n            - name: MY_NODE_NAME\n              valueFrom:\n                fieldRef:\n                  fieldPath: spec.nodeName\n            - name: SCHEDULER_CRON_EXPRESSION\n              value: \"* * * * * * *\"\n```\n\n##### Set an Update Time Window - DEPRECATED\n\n**Note**: these settings are deprecated and will be removed in a future release.\nTime window settings cannot be used in combination with the preferred cron expression format and will be ignored.\n\nIf you still decide to use these settings, please use \"hour:00:00\" format only instead of \"HH:MM:SS\".\n\n`UPDATE_WINDOW_START` and `UPDATE_WINDOW_STOP` can be used to specify the time window in which updates are permitted.\n\nTo enable this feature, set the `update_window_start` and `update_window_stop` values in your helm values yaml file to a `hour:minute:second` formatted value (UTCE 24-hour time notation).\nFor example: `update_window_start: \"08:0:0\"` and `update_window_stop: \"12:30:0\"`.\n\nOtherwise, go to `bottlerocket-update-operator.yaml` and change `UPDATE_WINDOW_START` and `UPDATE_WINDOW_STOP` to a `hour:minute:second` formatted value (UTC (24-hour time notation)).\n\nNote that `UPDATE_WINDOW_START` is inclusive and `UPDATE_WINDOW_STOP` is exclusive.\n\nNote: brupop uses UTC (24-hour time notation), please convert your local time to UTC.\nFor example:\n```yaml\n      containers:\n        - command:\n            - \"./controller\"\n          env:\n            - name: MY_NODE_NAME\n              valueFrom:\n                fieldRef:\n                  fieldPath: spec.nodeName\n            - name: UPDATE_WINDOW_START\n              value: \"09:00:00\"\n            - name: UPDATE_WINDOW_STOP\n              value: \"21:00:00\"\n```\n\n### Label nodes\n\nBy default, each Workload resource constrains scheduling of the update operator by limiting pods to Bottlerocket nodes based on their labels.\nThese labels are not applied on nodes automatically and will need to be set on each desired node using `kubectl`.\nThe agent relies on each node's updater components and schedules its pods based on their interface supported.\nThe node indicates its updater interface version in a label called `bottlerocket.aws/updater-interface-version`.\nAgent deployments, respective to the interface version, are scheduled using this label and target only a single version in each.\n\nFor versions \u003e `0.2.0` of the Bottlerocket update operator, only `update-interface-version` `2.0.0` is supported, which uses Bottlerocket's [update API](https://github.com/bottlerocket-os/bottlerocket/blob/develop/sources/updater/README.md#update-api) to dispatch updates.\nFor this reason, only Bottlerocket OS versions \u003e `v0.4.1` are supported.\n\nFor the `2.0.0` `updater-interface-version`, this label looks like:\n\n``` text\nbottlerocket.aws/updater-interface-version=2.0.0\n```\n\nWith [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/) configured for the desired cluster, you can use the below command to get all nodes:\n\n```sh\nkubectl get nodes\n```\nMake a note of all the node names that you would like the Bottlerocket update operator to manage.\n\nNext, add the `updater-interface-version` label to the nodes.\nFor each node, use this command to add `updater-interface-version` label.\nMake sure to change `NODE_NAME` with the name collected from the previous command:\n\n```sh\nkubectl label node NODE_NAME bottlerocket.aws/updater-interface-version=2.0.0\n```\n\nIf all nodes in the cluster are running Bottlerocket and require the same `updater-interface-version`, you can label all at the same time by running this:\n```sh\nkubectl label node $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}') bottlerocket.aws/updater-interface-version=2.0.0\n```\n\n#### Automatic labeling via Bottlerocket user-data\n\nYou can automatically [add Kubernetes labels to your Bottlerocket nodes via the settings provided in user data](https://github.com/bottlerocket-os/bottlerocket#kubernetes-settings)\nwhen your nodes are provisioned:\n\n```toml\n# Configure the node-labels Bottlerocket setting\n[settings.kubernetes.node-labels]\n\"bottlerocket.aws/updater-interface-version\" = \"2.0.0\"\n```\n\n#### Automatic labeling via `eksctl`\n\nIf you're using `eksctl`, you can automatically add labels to nodegroups. For example:\n\n```yaml\n---\napiVersion: eksctl.io/v1alpha5\nkind: ClusterConfig\n\nmetadata:\n  name: bottlerocket-cluster\n  region: us-west-2\n  version: '1.17'\n\nnodeGroups:\n  - name: ng-bottlerocket\n    labels: { bottlerocket.aws/updater-interface-version: 2.0.0 }\n    instanceType: m5.large\n    desiredCapacity: 3\n    amiFamily: Bottlerocket\n```\n\n### Uninstalling\n\nIf you remove the `bottlerocket.aws/updater-interface-version=2.0.0` label from a node,\nthe Brupop controller will remove the resources from that node (including the `bottlerocketshadow` CRD associated with that node).\n\nDeleting the custom resources, definitions, and deployments will then fully remove the bottlerocket update operator from your cluster:\n\n```sh\nhelm uninstall brupop\nhelm uninstall brupop-crd\n```\n\n## Operation\n\n### Overview\n\nThe update operator controller and agent processes communicate using Kubernetes [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/), with one being created for each node managed by the operator.\nThe Custom Resource created by the update operator is called a BottlerocketShadow resource, or otherwise shortened to `brs`.\nThe Custom Resource's Spec is configured by the controller to indicate a desired state, which guides the agent components.\nThe update operator's agent component keeps the Custom Resource Status updated with the current state of the node.\nMore about Spec and Status can be found in the [Kubernetes documentation](https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/#object-spec-and-status).\n\nAdditionally, the update operator's controller and apiserver components expose metrics which can be configured to be [collected by Prometheus](#monitoring-cluster-history-and-metrics-with-prometheus).\n\n### Observing State\n\n#### Monitoring Custom Resources\n\nThe current state of the cluster from the perspective of the update operator can be summarized by querying the Kubernetes API for BottlerocketShadow objects.\nThis view will inform you of the current Bottlerocket version of each node managed by the update operator, as well as the current ongoing update status of any node with an update in-progress.\nThe following command requires `kubectl` to be configured for the desired cluster to be monitored:\n\n``` sh\nkubectl get bottlerocketshadows --namespace brupop-bottlerocket-aws\n```\n\nYou can shorten this with:\n\n``` sh\nkubectl get brs --namespace brupop-bottlerocket-aws\n```\n\nYou should see output akin to the following:\n\n```\n$ kubectl get brs --namespace brupop-bottlerocket-aws\nNAME                                               STATE   VERSION   TARGET STATE   TARGET VERSION\nbrs-node-1                                         Idle    1.5.2     Idle\nbrs-node-2                                         Idle    1.5.1     StagedUpdate   1.5.2\n```\n\n#### Monitoring Cluster History and Metrics with Prometheus\n\nThe update operator provides metrics endpoints which can be scraped by [Prometheus](https://prometheus.io/).\nThis allows you to monitor the history of update operations using popular metrics analysis and visualization tools.\n\nWe provide a [sample configuration](./deploy/examples/prometheus-resources.yaml) which demonstrates a Prometheus deployment into the cluster that is configured to gather metrics data from the update operator.\n\nTo deploy the sample configuration, you can use `kubectl`:\n\n```sh\nkubectl apply -f ./deploy/telemetry/prometheus-resources.yaml\n```\n\nNow that Prometheus is running in the cluster, you can use the UI provided to visualize the cluster's history.\nGet the Prometheus pod name (e.g. `prometheus-deployment-5554fd6fb5-8rm25`):\n\n```sh\nkubectl get pods --namespace brupop-bottlerocket-aws\n```\n\nSet up port forwarding to access Prometheus on the cluster:\n\n```sh\nkubectl port-forward $prometheus-pod-name 9090:9090 --namespace brupop-bottlerocket-aws\n```\n\nPoint your browser to `localhost:9090/graph` to access the sample Prometheus UI.\n\nSearch for:\n* `brupop_hosts_state` to check how many hosts are in each state.\n* `brupop_hosts_version` to check how many hosts are in each Bottlerocket version.\n\n\n### Image Region\n\n`bottlerocket-update-operator.yaml` pulls operator images from Amazon ECR Public.\nYou may also choose to pull from regional Amazon ECR repositories such as the following.\n\n  - `917644944286.dkr.ecr.af-south-1.amazonaws.com`\n  - `375569722642.dkr.ecr.ap-east-1.amazonaws.com`\n  - `328549459982.dkr.ecr.ap-northeast-1.amazonaws.com`\n  - `328549459982.dkr.ecr.ap-northeast-2.amazonaws.com`\n  - `328549459982.dkr.ecr.ap-northeast-3.amazonaws.com`\n  - `328549459982.dkr.ecr.ap-south-1.amazonaws.com`\n  - `328549459982.dkr.ecr.ap-southeast-1.amazonaws.com`\n  - `328549459982.dkr.ecr.ap-southeast-2.amazonaws.com`\n  - `386774335080.dkr.ecr.ap-southeast-3.amazonaws.com`\n  - `328549459982.dkr.ecr.ca-central-1.amazonaws.com`\n  - `328549459982.dkr.ecr.eu-central-1.amazonaws.com`\n  - `328549459982.dkr.ecr.eu-north-1.amazonaws.com`\n  - `586180183710.dkr.ecr.eu-south-1.amazonaws.com`\n  - `328549459982.dkr.ecr.eu-west-1.amazonaws.com`\n  - `328549459982.dkr.ecr.eu-west-2.amazonaws.com`\n  - `328549459982.dkr.ecr.eu-west-3.amazonaws.com`\n  - `553577323255.dkr.ecr.me-central-1.amazonaws.com`\n  - `509306038620.dkr.ecr.me-south-1.amazonaws.com`\n  - `328549459982.dkr.ecr.sa-east-1.amazonaws.com`\n  - `328549459982.dkr.ecr.us-east-1.amazonaws.com`\n  - `328549459982.dkr.ecr.us-east-2.amazonaws.com`\n  - `328549459982.dkr.ecr.us-west-1.amazonaws.com`\n  - `328549459982.dkr.ecr.us-west-2.amazonaws.com`\n  - `388230364387.dkr.ecr.us-gov-east-1.amazonaws.com`\n  - `347163068887.dkr.ecr.us-gov-west-1.amazonaws.com`\n  - `183470599484.dkr.ecr.cn-north-1.amazonaws.com.cn`\n  - `183901325759.dkr.ecr.cn-northwest-1.amazonaws.com.cn`\n\nExample regional image URI:\n```\n328549459982.dkr.ecr.us-west-2.amazonaws.com/bottlerocket-update-operator:v1.1.0\n```\n\n### Current Limitations\n\n- Monitoring on newly-rebooted nodes is limited.\n  We are considering an approach in which custom health checks can be configured to run after reboots. (https://github.com/bottlerocket-os/bottlerocket/issues/503)\n- Node labels are not automatically applied to allow scheduling (https://github.com/bottlerocket-os/bottlerocket/issues/504)\n\n## Troubleshooting\n\nWhen installed with the [default deployment](./bottlerocket-update-operator.yaml), the logs can be fetched through Kubernetes deployment logs.\nBecause mutations to a node are orchestrated through the API server component, searching those deployment logs for a node ID can be useful.\nTo get logs for the API server, run the following:\n\n```sh\nkubectl logs deployment/brupop-apiserver --namespace brupop-bottlerocket-aws\n```\n\nThe controller logs will usually not help troubleshoot issues about the state of updates in a cluster, but they can similarly be fetched:\n\n```sh\nkubectl logs deployment/brupop-controller-deployment --namespace brupop-bottlerocket-aws\n```\n\n### Why are updates stuck in my cluster?\nThe bottlerocket update operator only installs updates on one node at a time.\nIf a node's update becomes stuck, it can prevent the operator from proceeding with updates across the cluster.\n\nThe update operator uses the [Kubernetes Eviction API](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/) to safely drain pods from a node.\nThe eviction API will respect [PodDisruptionBudgets](https://kubernetes.io/docs/tasks/run-application/configure-pdb/), refusing to remove a pod from a node if it would cause a PDB not to be satisfied.\nIt is possible to mistakenly configure a Kubernetes cluster in such a way that a Pod can never be deleted while still maintaining the conditions of a PDB.\nIn this case, the operator may become stuck waiting for the PDB to allow an eviction to proceed.\n\nSimilarly, if the Node in question is repeatedly encountering issues while updating, it may cause updates across the cluster to become stuck.\nSuch issues can be troubleshooted by requesting the update operator agent logs from the node.\nFirst, list the agent pods and select the pod residing on the node in question:\n\n```sh\nkubectl get pods --selector=brupop.bottlerocket.aws/component=agent -o wide --namespace brupop-bottlerocket-aws\n```\n\nThen fetch the logs for that agent:\n\n```sh\nkubectl logs brupop-agent-podname --namespace brupop-bottlerocket-aws\n```\n\n### Why are my bottlerocket nodes egressing to `https://updates.bottlerocket.aws`?\n\nThe [Bottlerocket updater API](https://github.com/bottlerocket-os/bottlerocket/blob/develop/sources/updater/README.md)\nutilizes [TUF repository signed metadata](https://theupdateframework.io/metadata/)\nserved from a public URL to query and check for updates.\nThe URL is `https://updates.bottlerocket.aws` and Bottlerocket's updater system requires\naccess to this endpoint in order to perform updates.\n\nCluster nodes running in production environments with limited network access\nmay not be able to reach this metadata endpoint\nand you may see failures when updates are available or an update statuses that appears \"stuck\".\n\nTo troubleshoot and validate this, [access one of your Bottlerocket nodes on your cluster](https://github.com/bottlerocket-os/bottlerocket/blob/9adbe40c3fced559b2d05829e62902060c5ecc9c/README.md#exploration),\nand manually execute `apiclient update check`. If you see errors that indicate `Failed to check for updates`\nand network timeouts, ensure that your nodes on cluster have access to `https://updates.bottlerocket.aws`.\n\nIf it is unacceptable to give your Bottlerocket nodes outside network access,\nyou may consider creating a self managed proxy within the cluster\nthat can periodically scrape the TUF repository from `https://updates.bottlerocket.aws`.\nThis would also require updating [the `settings.updates.metadata-base-url` and `settings.updates.targets-base-url` settings](https://github.com/bottlerocket-os/bottlerocket#updates-settings)\nto point to your proxy.\n[Typically, this is done via `tuftool download` or `tuftool clone`.](https://github.com/awslabs/tough/tree/develop/tuftool#download-tuf-repo)\n\nUsers who are building _their own Bottlerocket variants and TUF repositories_\nwill also need to update their Bottlerocket `settings.updates` settings to point to their custom TUF repository.\nBut since Brupop simply interfaces with the node's `apiclient` via the daemonset,\nno further configurations or changes are required in Brupop.\nAn in depth discussion on [building your own TUF repos can be found in the core Bottlerocket repo.](https://github.com/bottlerocket-os/bottlerocket/blob/develop/PUBLISHING.md#build-a-repo)\n\n### Why do only some of my Bottlerocket instances have an update available?\n\nUpdates to Bottlerocket are rolled out in [waves](https://github.com/bottlerocket-os/twoliter/tree/develop/twoliter/embedded/waves) to reduce the impact of issues; the container instances in your cluster may not all see updates at the same time.\nYou can check whether an update is available on your instance by running the `apiclient update check` command from within the [control](https://github.com/bottlerocket-os/bottlerocket#control-container) or [admin](https://github.com/bottlerocket-os/bottlerocket#admin-container) container.\n\n### Why do new container instances launch with older Bottlerocket versions?\n\nThe Bottlerocket update operator performs in-place updates for instances in your Kubernetes cluster.\nThe operator does not influence how those instances are launched.\nIf you use an [auto-scaling group](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html) to launch your instances, you can update the AMI ID in your [launch configuration](https://docs.aws.amazon.com/autoscaling/ec2/userguide/LaunchConfiguration.html) or [launch template](https://docs.aws.amazon.com/autoscaling/ec2/userguide/LaunchTemplates.html) to use a newer version of Bottlerocket.\n\n## How to Contribute and Develop Changes\n\nWorking on the update operator requires a fully configured and working Kubernetes cluster.\nBecause the agent component relies on the Bottlerocket API to properly function, we suggest a cluster which is running Bottlerocket nodes.\nThe `integ` crate can currently be used to launch Bottlerocket nodes into an [Amazon EKS](https://aws.amazon.com/eks/) cluster to observe update-operator behavior.\n\nHave a look at the [design](./design/DESIGN.md) to learn more about how the update operator functions.\nPlease feel free to open an issue with an questions!\n\n### Building and Deploying a Development Image\n\nTo re-generate the yaml manifest found at the root of this reposiroy, simply run:\n```\nmake manifest\n```\nNote: this requires `helm` to be installed.\n\nTargets in the `Makefile` can assist in creating an image.\nThe following command will build and tag an image using the local Docker daemon:\n\n```sh\nmake brupop-image\n```\n\nOnce this image is pushed to a container registry, you can set environment variables to regenerate a `.yaml` file suitable for deploying the image to Kubernetes.\nFirstly, modify the `.env` file to contain the desired image name, as well as a secret for pulling the image if necessary.\nThen run the following to regenerate the `.yaml` resource definitions:\n\n```sh\ncargo build -p deploy\n```\n\nThese can of course be deployed using `kubectl apply` or the automatic integration testing tool [integ](https://github.com/bottlerocket-os/bottlerocket-update-operator/tree/develop/integ).\n\n## Security\n\nSee [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.\n\n## License\n\nThis project is dual licensed under either the Apache-2.0 License or the MIT license, your choice.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbottlerocket-os%2Fbottlerocket-update-operator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbottlerocket-os%2Fbottlerocket-update-operator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbottlerocket-os%2Fbottlerocket-update-operator/lists"}