{"id":18244321,"url":"https://github.com/stackb/buildkube","last_synced_at":"2025-04-04T13:31:21.705Z","repository":{"id":50277284,"uuid":"137157179","full_name":"stackb/buildkube","owner":"stackb","description":"Bazel Remote Cache + Remote Execution in Kubernetes","archived":false,"fork":false,"pushed_at":"2018-12-10T20:41:24.000Z","size":67,"stargazers_count":63,"open_issues_count":1,"forks_count":8,"subscribers_count":5,"default_branch":"master","last_synced_at":"2023-02-28T12:32:44.078Z","etag":null,"topics":["bazel","kubernetes","remote-execution"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stackb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-06-13T03:23:50.000Z","updated_at":"2023-02-09T17:04:07.000Z","dependencies_parsed_at":"2022-08-12T21:01:41.123Z","dependency_job_id":null,"html_url":"https://github.com/stackb/buildkube","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackb%2Fbuildkube","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackb%2Fbuildkube/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackb%2Fbuildkube/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stackb%2Fbuildkube/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stackb","download_url":"https://codeload.github.com/stackb/buildkube/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223143828,"owners_count":17094722,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bazel","kubernetes","remote-execution"],"created_at":"2024-11-05T09:16:14.055Z","updated_at":"2024-11-05T09:16:14.642Z","avatar_url":"https://github.com/stackb.png","language":"Python","readme":"# buildkube\n\n\u003ctable\u003e\u003ctr\u003e\n\u003ctd\u003e\n  \u003ca href=\"https://bazel.build\"\u003e\n    \u003cimg src=\"https://github.com/bazelbuild.png\" height=\"120\"/\u003e\n  \u003c/a\u003e\n\u003c/td\u003e\n\u003ctd\u003e\n  \u003ca href=\"https://github.com/bazelbuild/remote-apis\"\u003e\n    \u003cimg src=\"http://laplace.us.es/wiki/images/0/09/Conduccion-metal.gif\"\n         height=\"120\"/\u003e\n  \u003c/a\u003e\n\u003c/td\u003e\n\u003ctd\u003e\n  \u003ca href=\"https://kubernetes.io\"\u003e\n    \u003cimg src=\"https://github.com/kubernetes.png\"\n         height=\"120\"/\u003e\n  \u003c/a\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctd\u003eBazel\u003c/td\u003e\n\u003ctd\u003eREAPI\u003c/td\u003e\n\u003ctd\u003eKubernetes\u003c/td\u003e\n\u003c/table\u003e\n\nbuildkube uses [rules_docker] and [rules_k8s] to build and deploy\n[bazel-buildfarm] (java), [bazel-buildbarn] (golang) and/or [buildgrid] (python)\ninto an existing kubernetes cluster.  These are the 3 known open-source\nserver-side implementations of the [remote-execution-api] (REAPI), plus the\nclosed source google Remote Build Execution\n([RBE](https://groups.google.com/forum/#!forum/rbe-alpha-customers)) service\n(alpha).\n\nKnown clients of the REAPI include [bazel](https://github.com/bazelbuild/bazel)\nitself, [recc](https://gitlab.com/bloomberg/recc), and possibly\n[pants](https://github.com/pantsbuild/pants/pull/4910). \n\n## INSTRUCTIONS\n\n1. Clone this repository\n2. Edit the `WORKSPACE` file `k8s_defaults` rule to point to your kubernetes\n   cluster (should match `$ kubectl config current-context`)\n3. Build and deploy an implementation: for example: `$ (cd farm/ \u0026\u0026 make\n   install)`\n4. In a separate terminal, establish port-forwarding to the server\n   implementation `$ (cd farm/ \u0026\u0026 make port-forward)`\n5. Clone the abseil repository as a test case: `$ make abseil_clone`\n6. Compile abseil remotely: `$ make abseil`\n\n## NOTES\n\n* Bazel 0.17.1 or higher is required (primarily tested on 0.17.2 on an ubuntu\n  laptop).\n* Run all tests via `$ bazel test //...`.\n* Each implementation goes in its own namespace.  `$ kubectl get pods\n  --all-namespaces` to see all.\n* Consider adjusting `replicas` in the `deploy.yaml` files and/or `bazelrc`\n  file.\n\n## OBSERVATIONS\n\n### General \n\n* Logging in all 3 implementations is scant and makes debugging difficult.\n  Prometheus metrics are available in the barn impl (not examined thus far).\n\n### BuildFarm \n\n* BuildFarm worker does not detect if server goes down.  Must manually `kubectl\n  delete pod --selector=k8s-app=worker` when re-installing or updating server\n  deployment.\n\n* When a worker registers itself with the server (operation-queue), it provides\n  a dict of key:value pairs that must match the action execution requirements.\n  In particular, the `worker.config` `container-image` key MUST be exactly\n  matching the rbe_ubuntu image tag. \n\n### BuildBarn \n\n* After spinning up a new install, the service seems flaky at first.  Tend to\n  get several errors like: `/tmp/abseil-cpp/absl/utility/BUILD.bazel:22:1: C++\n  compilation of rule '//absl/utility:utility_test' failed (Exit 34). Note:\n  Remote connection/protocol failed with: execution failed catastrophically`. \n\n\u003e NOTE(@EdShoueten): There are three ways that can be used to alleviate this\n\u003e issue: \n\u003e - Spawn more workers on your cluster. \n\u003e - Pass in an explicit --jobs= to the build that is the same order of magnitude\n\u003e   as the number of workers. \n\u003e - Tune this flag on the scheduler process:\n\u003e   https://github.com/EdSchouten/bazel-buildbarn/blob/master/cmd/bbb_scheduler/main.go#L22\n\u003e\n\u003e [More details](https://groups.google.com/forum/#!topic/bazel-discuss/pPNIc9-liCE)\n\n### BuildGrid \n\n* Worker does not auto-reconnect to a new server (like buildfarm).\n* Instance name (`main`) must match across the `bazelrc` `--instance_name=main`,\n  server args `-scheduler main|ubuntu-scheduler:8981`, and worker args `bot\n  --remote=http://server:8980 --parent=main host-tools`\n* Overall robustness to changes (increases) in job size and worker size is low.\n  Seems to require resetting the server/workers in some cases.  Seems happiest\n  when job size matches worker replicas.\n\n[rules_docker]: https://github.com/bazelbuild/rules_docker \n[rules_k8s]: https://github.com/bazelbuild/rules_k8s\n[bazel-buildfarm]: https://github.com/bazelbuild/bazel-buildfarm/\n[bazel-buildbarn]: https://github.com/EdSchouten/bazel-buildbarn/\n[buildgrid]: https://gitlab.com/BuildGrid/buildgrid\n[remote-execution-api]: https://github.com/bazelbuild/remote-apis\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstackb%2Fbuildkube","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstackb%2Fbuildkube","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstackb%2Fbuildkube/lists"}