{"id":13489803,"url":"https://github.com/bluebrown/kobold","last_synced_at":"2025-08-12T16:32:51.893Z","repository":{"id":111481363,"uuid":"565251499","full_name":"bluebrown/kobold","owner":"bluebrown","description":"update container image references, based on webhook events","archived":false,"fork":false,"pushed_at":"2024-04-22T11:59:01.000Z","size":1336,"stargazers_count":12,"open_issues_count":13,"forks_count":6,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-04-22T13:07:00.586Z","etag":null,"topics":["argocd","containers","fluxcd","gitbot","gitops","image-update","kubernetes","webhook"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bluebrown.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-12T20:04:28.000Z","updated_at":"2024-05-18T18:21:26.348Z","dependencies_parsed_at":"2024-03-30T22:21:50.224Z","dependency_job_id":"65b5831a-7543-4c17-9bc0-69bc005314b8","html_url":"https://github.com/bluebrown/kobold","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/bluebrown/kobold","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluebrown%2Fkobold","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluebrown%2Fkobold/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluebrown%2Fkobold/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluebrown%2Fkobold/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bluebrown","download_url":"https://codeload.github.com/bluebrown/kobold/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bluebrown%2Fkobold/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270097291,"owners_count":24526641,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-12T02:00:09.011Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["argocd","containers","fluxcd","gitbot","gitops","image-update","kubernetes","webhook"],"created_at":"2024-07-31T19:00:35.578Z","updated_at":"2025-08-12T16:32:51.542Z","avatar_url":"https://github.com/bluebrown.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# Kobold\n\n\u003e [!NOTE]  \n\u003e If you are migrating from v.0.3 to v.0.4, you can use the `confix` command to\n\u003e migrate your config. See the [ConFix](#confix) section for more details.\n\nUpdate container image references in git repositories, based on recieved\nevents.\n\nKobold is meant to be used as a companion to other gitops tools, like ArgoCD or\nFlux. It keeps image references in your gitops repos up to date, while\ndelegating the actual deployment to your gitops tool of choice.\n\nTypically you would configure your container registry to send events to kobolds\nwebhook endpoint, whenever a new image is pushed.\n\n## Webhook\n\nIn servermode Kobold exposes a webhook endpoint at\n`$KOBOLD_ADDR_WEBHOOK/events/\u003cchannel\u003e`. The channel is used to identify the\npipelines to run. It accepts any content in the body since it is the [channels\ndecoder's](#decoders) responsibility to parse the body.\n\nFor example below is a request to a channel using the plain lines decoder.\n\n```text\nPOST /events/plain HTTP/1.1\nHost: kobold.myorg.io:8080\n\ndocker.io/library/busybox:v1.4@sha256:220611111e8c9bbe242e9dc1367c0fa89eef83f26203ee3f7c3764046e02b248\n```\n\n## Matching Rules\n\nKobold uses special inline comments in yaml files, to identify image references\nthat should be updated. It will match the reference from the incoming event,\nagainst the rules in the comment, and update the reference if a match is found.\n\n```yaml\napiVersion: v1\nkind: Pod\nmetadata:\n  name: my-pod\nspec:\n  containers:\n    - name: my-app\n      image: my.org/amazing/app # kobold: tag: ^1; type: semver\n```\n\nThe comments parsed by kobold have the following format:\n\n```console\n# kobold: tag: \u003ctag-name\u003e; type: \u003ctag-type\u003e\n```\n\nThe tag-type can be either `exact`, `semver`, or `regex`, and specifies how\nkobold should interpret the tag-name.\n\nFor example, if tag-type is semver, the tag-name can include common semantic\nversioning semantics, such as ^1 to denote that any tag between v1 and v2\nshould be matched (not including v2).\n\nNote that the regex type will add `^` and `$` to the tag-name, to ensure the\nregex matches the entire tag, so dont include them in the tag-name.\n\nIt is also possible to update only a part of a given image reference. For\nexample it is common for helm charts to split the reference across field, like\n`repo` and `tag`.\n\n```yaml\nimage:\n  tag: \"1.0\"  # kobold: tag: ^1; type: semver; part: tag; context: docker.io/library/busybox\n---\n```\n\nIn this case, the `part` field specifies the part of the image reference to be\nupdated, and the `context` field specifies the image reference to be updated.\nNext to `tag`, `digest` and `tag+digest`, are also supported.\n\n## Configuration\n\nKobold is configured by setting up named channels, and pipelines to run\nwhenever a message is recieved on a channel. The pipeline will clone the repo,\nand run a krm filter, before potentially pushing the changes back to the repo.\n\nIn the below example, the `example` pipeline is run, whenever a message is\nrecieved on the `a` or `b` channel.\n\n```toml\nversion = \"2\"\n\n[[channel]]\nname = \"a\"\n\n[[channel]]\nname = \"b\"\n\n[[pipeline]]\nname = \"example\"\nrepo_uri = \"git@github.com:bluebrown/example.git?ref=main\"\nchannels = [\"a\", \"b\"]\n```\n\n\u003cspan id=\"decoders\"\u003e\u003c/span\u003e\n\nMessages received on a channel have to be decoded, before they can be processed\nby the pipeline. By default, each line in a message is treated as a fully\nqualified image reference. You can change this behaviour by providing a decoder\nin the channel config. See the [builtin](#builtins) section for more details.\n\n```toml\n[[channel]]\nname = \"example\"\ndecoder = \"builtin.distribution@v1\"\n```\n\nOptionally, you can set a destination branch, to push the changes to. If you\ndont set a destination branch, the changes will be pushed to the source branch.\n\n```toml\n[[pipeline]]\nname = \"example\"\ndest_branch = \"release\"\n```\n\nIf you want to perform an action after the changes have been pushed to git, you\ncan attach a post hook to the pipeline. See the [builtin](#builtins) section\nfor more details.\n\n```toml\n[[pipeline]]\nname = \"example\"\npost_hook = \"builtin.github-pr@v1\"\n```\n\n### Scoping\n\nPipelines are scoped through the following URI format. Note the package is\noptional.\n\n```text\n\u003crepo\u003e?ref=\u003cref\u003e[\u0026pkg=\u003cpkg\u003e]\n```\n\nIf you want to scope a pipline beyond a sub directory (package), you can place\na .krmignore file at the package root, ignoring parts of the package.\n\n\u003e ignoreFilesMatcher handles `.krmignore` files, which allows for ignoring files\n\u003e or folders in a package. The format of this file is a subset of the gitignore\n\u003e format, with recursive patterns (like a/\\*\\*/c) not supported. If a file or\n\u003e folder matches any of the patterns in the .krmignore file for the package, it\n\u003e will be excluded.\n\nFor example, you have a helm like below, and want to ignore its template\ndirectory:\n\n```text\n. (package root)\n├── charts\n│   ├── my-chart\n│   └── .krmignore\n└── pod.yaml\n```\n\nThen you would place the following in the `charts/.krmignore` file:\n\n```text\n/*/charts/\n/*/templates/\n```\n\nThis works because the presense of a `.krmignore` makes the direcotry,\ncontaining the file, a sub package, and kobold will recurse into it.\n\n### Builtins\n\nThere are a few builtin decoders and post hooks, to support some common use\ncases. For example decoding events from a oci distribution registry, or opening\na pull request on github.\n\nThe starlark code for the builtins can be reviewed in the\n[builtin](./plugin/builtin) directory.\n\n#### Decoder\n\n##### `builtin.lines@v1`\n\nThe lines reads lines of image references:\n\n```text\ndocker.io/bluebrown/busybox:v1.3@sha256:220611111e8c9bbe242e9dc1367c0fa89eef83f26203ee3f7c3764046e02b248\ntest.azurecr.io/nginx:v1.1.1@sha256:220611111e8c9bbe242e9dc1367c0fa89eef83f26203ee3f7c3764046e02b248\n```\n\n##### `builtin.distribution@v1`\n\nThe distribution decoder parses a json message using the schema defined in\n\u003chttps://github.com/distribution/distribution\u003e.\n\nIt understands single events:\n\n```json\n{\n  \"request\": { \"host\": \"example.azurecr.io\" },\n  \"target\": {\n    \"digest\": \"sha256:xxxxd5c8786bb9e621a45ece0dbxxxx1cdc624ad20da9fe62e9d25490f33xxxx\",\n    \"repository\": \"bluebrown/busybox\",\n    \"tag\": \"v1\"\n  }\n}\n```\n\nOr wrapped in an envelope:\n\n```json\n{\"events\": [{\"...\"}]}\n```\n\n##### `builtin.dockerhub@v1`\n\nThe dockerhub decoder parses a json message using the schema defined in\n\u003chttps://docs.docker.com/docker-hub/webhooks/#push-event\u003e.\n\n```json\n{\n  \"push_data\": {\n    \"tag\": \"v1\"\n  },\n  \"repository\": {\n    \"repo_name\": \"bluebrown/busybox\"\n  }\n}\n```\n\n\u003e [!WARNING]  \n\u003e Since dockerhub does not set the image digest in the payload, kobold will\n\u003e update only the tag of matched image references. This will not work as\n\u003e expected when pusing the same tag repeatedly.\n\n#### Post Hooks\n\n##### `builtin.github-pr@v1`\n\nPerforms a github pull request. It requires the `GITHUB_TOKEN` environment\nvariable to be set.\n\n#### `builtin.ado-pr@v1`\n\nPerforms a azure devops pull request. It requires the `ADO_USR` and `ADO_PAT`\nenvironment variables to be set.\n\n##### `builtin.gitea-pr@v1`\n\nPerforms a gitea pull request. It requires the`GITEA_HOST` and\n`GITEA_AUTH_HEADER` environment variable to be set.\n\n### Extending Kobold\n\nKobold is designed to be extended. You can write your own decoders and post\nhooks.\n\nFor example, below is the builtin lines decoder. It simply splits the message\nrecieved on the channel by newlines, and returns the resulting list. Treating\neach line as a fully qalified image reference.\n\n```toml\n[[decoder]]\nname = \"builtin.lines@v1\"\nscript = \"\"\"\ndef main(input):\n    return input.split(\"\\n\")\n\"\"\"\n```\n\nThis post hook does nothing, but print the message to stdout.\n\n```toml\n[[post_hook]]\nname = \"builtin.print@v1\"\nscript = \"\"\"\ndef main(repo, src_branch, dest_branch, title, body, changes, warnings):\n    print(repo, src_branch, dest_branch, title, body, changes, warnings)\n\"\"\"\n```\n\nThis allows to integrate with any event producer and any git provider, since\nproducer and provider specific logic can be implemented in starlark.\n\n### Git\n\nKobold uses the git command line tool to interact with git. That means you can\nconfigure the underlying git client directly and according to gits\ndocumentation. For example you can mount a file to `/.gitconfig` in the kobold\ncontainer, or set git specific environment variables.\n\n## Cook Book\n\nThis section showcases some common use cases.\n\n### SSH\n\nTo use ssh, you can mount a directory to `/etc/kobold/.ssh` in the kobold container.\n\nThe directory should, at least contain, a known_hosts file and a default\nidentity file. The default identity file is usually named id_rsa. If more than\none key is required, you can place a config file in the .ssh directory, and\nconfigure the identity file for each host.\n\n```bash\n# this dir will be mounted\nmkdir -p .ssh\n# scan your git providers host key\nssh-keyscan github.com \u003e .ssh/known_hosts\n# generate a key (dont forget to add it to your git provider)\nssh-keygen -t ed25519 -f .ssh/id_ed25519 -N ''\n# set the permission to the kobold user\nchown -R 65532:65532 .ssh\n# run kobold\ndocker run -v \"$(pwd)/.ssh:/etc/kobold/.ssh\" ...\n```\n\n### Basic Auth\n\nYou can provide the credentials in the git uri, but this is not recommended,\nsince it will be stored in the kobold database.\n\nAlternatively, you can configure gits credentails helper, to use a file as\nbackend. For example via `/etc/kobold/.gitconfig`:\n\n```conf\n[credential]\n    helper = store --file ~/.git-credentials\n```\n\nWhen doing this, you can mount a credentials file to\n`/etc/kobold/.git-credentials` in the kobold container.\n\nBelow is an example entry in the credentials file:\n\n```text\nhttps://bob:s3cre7@mygithost\n```\n\n### Pull Requests\n\nFor a pull request setup, you want to set the destination branch to something\nother than the source branch. Then you can run a post hook, opening the pull\nrequest. There is already, amongst others, a [builtin](#post-hooks) post hook\nfor github, using the `GITHUB_TOKEN` environment variable, to authenticate.\n\n```toml\n[[pipeline]]\nname = \"my-github-pr\"\nchannels = [\"example\"]\nrepo_uri = \"git@github.com:bluebrown/foobar.git?ref=main\u0026pkg=manifests\"\ndest_branch = \"kobold\"\npost_hook = \"builtin.github-pr@v1\"\n```\n\n### Environment Promotion\n\nThe below example uses package scoping, to perform different actions based on\nthe environment. For stage, the updated image ref is directly applied to the\ncluster. For prod, a pull request is opened, and the image ref is updated after\nthe pull request is merged.\n\n```toml\n[[pipeline]]\nname = \"stage\"\nchannels = [\"distribution\"]\nrepo_uri = \"http://gitea/dev/test.git?ref=main\u0026pkg=stage\"\n\n[[pipeline]]\nname = \"prod\"\nchannels = [\"distribution\"]\nrepo_uri = \"http://gitea/dev/test.git?ref=main\u0026pkg=prod\"\ndest_branch = \"release\"\npost_hook = \"builtin.gitea-pr@v1\"\n```\n\n## Deployment\n\n### Kubernetes\n\n\u003e [!NOTE]\n\u003e You can use flags or environment variables to customize the kobold\n\u003e deployment. See the [server synopsis](#synopsis) for more info.\n\nThe kobold manifests come with a few secrets and config maps intended to be\noverwritten by the user. They are mostly empty by default, but mounted to the\nright location, so any values provided will be picked up.\n\n| Resources                  | Description                                           | Mount Location                 |\n| -------------------------- | ----------------------------------------------------- | ------------------------------ |\n| configmap/kobold-confd     | multiple .toml files containing kobold configs        | `/etc/kobold/conf.d`           |\n| configmap/kobold-gitconfig | `.gitconfig` key containing the git config            | `/etc/kobold/.gitconfig`       |\n| secret/kobold-gitcreds     | `.git-credentials` key containing the git credentials | `/etc/kobold/.git-credentials` |\n| secret/kobold-ssh          | arbitrary keys representing the contents of `.ssh`    | `/etc/kobold/.ssh`             |\n| secret/kobold-env          | arbitrary environment variables                       | _used as env vars_             |\n\nYou can use kustomize to merge your own configs into the kobold manifests. For\nexample:\n\n```yaml\napiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\nresources:\n  - https://github.com/bluebrown/kobold/releases/latest/download/kobold-manifests.yaml\nconfigMapGenerator:\n  - name: kobold-confd\n    behavior: merge\n    files:\n      - team-a.toml\n      - team-b.toml\nsecretGenerator:\n  - name: kobold-env\n    behavior: merge\n    literals:\n      - GITHUB_TOKEN=xxx\n```\n\n#### Ingress\n\n\u003e [!WARNING]\n\u003e Kobold has not built in security mechanism. If you are planning to expose\n\u003e kobold via ingress, you should secure these endpoints via ingress controller.\n\n```yaml\napiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n  name: kobold-ingress\n  annotations:\n    # https://kubernetes.github.io/ingress-nginx/examples/auth/basic/\n    nginx.ingress.kubernetes.io/auth-type: basic\n    nginx.ingress.kubernetes.io/auth-secret: kobold-basic-auth\n    nginx.ingress.kubernetes.io/auth-realm: Authentication Required\nspec:\n  ingressClassName: nginx\n  rules:\n  - host: kobold.myorg.dev\n    http:\n      paths:\n      - path: /events/\u003cmychannel\u003e\n        pathType: Exact\n        backend:\n          service:\n            name: kobold-webhook\n            port:\n              number: 80\n\n      # Caution! Exposing this endpoint\n      # should not be required in most cases.\n      # Do it only for a good reason.\n      - path: /\n        pathType: Prefix\n        backend:\n          service:\n            name: kobold-api\n            port:\n              number: 80\n```\n\n## Metrics\n\nKobold exposes prometheus metrics on the `$KOBOLD_ADDR_API/metrics` endpoint.\nThe metrics are prefixed with `kobold_`.\n\n```python\n# HELP kobold_git_fetch_total number of git fetches\n# TYPE kobold_git_fetch_total counter\nkobold_git_fetch_total{repo=\"git@github.com:bluebrown/foobar\"} 3\nkobold_git_fetch_total{repo=\"git@ssh.dev.azure.com:v3/myorg/myproject/kobold-test\"} 3\n# HELP kobold_git_push_total number of git pushes\n# TYPE kobold_git_push_total counter\nkobold_git_push_total{repo=\"git@github.com:bluebrown/foobar\"} 4\n# HELP kobold_image_seen_total number of images seen\n# TYPE kobold_image_seen_total counter\nkobold_image_seen_total{ref=\"library/busybox\"} 5\n# HELP kobold_msg_recv_total number of messages received\n# TYPE kobold_msg_recv_total counter\nkobold_msg_recv_total{channel=\"dockerhub\",rejected=\"false\"} 5\n# HELP kobold_run_active number of active runs\n# TYPE kobold_run_active gauge\nkobold_run_active 0\n# HELP kobold_run_status_total run status (task groups)\n# TYPE kobold_run_status_total counter\nkobold_run_status_total{repo=\"git@github.com:bluebrown/foobar\",status=\"success\"} 6\nkobold_run_status_total{repo=\"git@ssh.dev.azure.com:v3/myorg/myproject/kobold-test\",status=\"success\"} 3\n```\n\n## Web API\n\nIn server mode, kobold exposes a json over http api. The api docs are available\nat `$KOBOLD_ADDR_API/api/docs/`. Note the trailing slash.\n\n## SQL\n\nKobold uses sqlite3 as a database. You can interact with the database using the\nsqlite cli, built into the kobold container.\n\n```bash\n# inspect the db of a running kobold server\ndocker exec -ti my-kobold sqlite3 ~/.config/kobold.db\n# inspect local db files\ndocker run -ti -v \"$HOME/.config/kobold:/tmp\" bluebrown/kobold sqlite3 /tmp/kobold.db\n```\n\nIf you want to use your own sqlite binary, make sure that you have the `uuid`\nand `sha1` [extensions](https://sqlite.org/loadext.html) enabled.\n\n## Binaries\n\n```bash\nGOBIN=\"$(pwd)/bin\" go install ./cmd/...\n```\n\n### Server\n\nThe server is the main kobold binary. It runs a webhook server, decoding and\nemitting events received on an http endpoint. Decoded events are sheduled and\nprocessed in groups after a debounce period. This ensures the minimum amount of\ngit operations are performed.\n\n```bash\nbin/server\n```\n\n#### Synopsis\n\n```bash\nUsage of server:\n  -addr-api string\n        api listen address (env: KOBOLD_ADDR_API) (default \":9090\")\n  -addr-webhook string\n        webhook listen address (env: KOBOLD_ADDR_WEBHOOK) (default \":8080\")\n  -confd string\n        path to config dir (env: KOBOLD_CONFD)\n  -config string\n        path to config file (env: KOBOLD_CONFIG)\n  -db string\n        path to sqlite db file (env: KOBOLD_DB) (default \"~/.config/kobold/kobold.sqlite3\")\n  -debounce duration\n        debounce interval for webhook events (env: KOBOLD_DEBOUNCE) (default 1m0s)\n  -handler value\n        task handler, one of: print, kobold, error (env: KOBOLD_HANDLER)\n  -logfmt string\n        log format, one of: json, text (env: KOBOLD_LOGFMT) (default \"json\")\n  -loglvl int\n        log level (env: KOBOLD_LOGLVL)\n  -maxprocs int\n        max number of concurrent runs (env: KOBOLD_MAXPROCS) (default 10)\n  -prefix string\n        prefix for all routes, must NOT contain trailing slash (env: KOBOLD_PREFIX)\n```\n\n### Command Line Interface\n\nThe CLI, reads messages from stdin. One message per line. The messages are\ntreated according to the kobold.toml. The same way they would in the server\nbinary.\n\nThe difference is, that the cli will not wrap the pool in the scheduler,\nmeaning the task will run immedaitly after grouping them. The cli will exit\nafter all messages have been processed.\n\n```bash\nbin/cli -channel default -handler print \u003c testdata/events.txt\n```\n\n### Image Reference Updater\n\nThe `image-ref-updater` command is kobolds business logic, as a standalone krm\nfilter, that can be used by other tools like kpt or kustomize.\n\nIt will update image references in the provided resources list. Image\nreferences to match against, are read from the `functionConfig`, which has the\nfollowing format:\n\n```yaml\napiVersion: kobold/v1alpha3\nkind: List\nmetadata:\n  name: my-fn-config\nitems:\n  - docker.io/bluebrown/busybox:latest@sha256:220611111e8c9bbe242e9dc1367c0fa89eef83f26203ee3f7c3764046e02b248\n  - test.azurecr.io/nginx:v1@sha256:993518ca49ede3c4e751fe799837ede16e60bc410452e3922602ebceda9b4c73\n```\n\n### ConFix\n\nThe confix command can be used to help with migrating the configration to a\nnewer version. Incremental updates be required as the binrary only knows the\nlast versions config.\n\n```bash\nbin/confix -w -f kobold.toml\n```\n\nThe above command reads a file and writes the converted format to back to the\nfile, due to the `-w` flag. The `-f` flag is used to specify the file to read.\n\n## Development\n\nYou can start a local kind cluster for an end to end setp. The gitea page can\nbe viewed at \u003chttp://localhost:8080\u003e with the credentials `dev:dev123`.\n\n```bash\nmake e2e\ndocker tag $(docker pull busybox -q) localhost:8080/library/busybox:v1.2.3\ndocker push localhost:8080/library/busybox:v1.2.3\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbluebrown%2Fkobold","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbluebrown%2Fkobold","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbluebrown%2Fkobold/lists"}