{"id":19124323,"url":"https://github.com/buildkite/elastic-ci-stack-s3-secrets-hooks","last_synced_at":"2025-05-05T19:13:45.550Z","repository":{"id":45996809,"uuid":"89782577","full_name":"buildkite/elastic-ci-stack-s3-secrets-hooks","owner":"buildkite","description":"🕵️‍♀️ Expose secrets to your buildkite build steps via Amazon S3","archived":false,"fork":false,"pushed_at":"2022-11-24T05:40:01.000Z","size":184,"stargazers_count":24,"open_issues_count":4,"forks_count":17,"subscribers_count":19,"default_branch":"master","last_synced_at":"2024-06-20T03:37:57.550Z","etag":null,"topics":["aws","aws-kms","aws-s3","buildkite","s3","secrets"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/buildkite.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-04-29T11:58:38.000Z","updated_at":"2024-03-06T23:57:19.000Z","dependencies_parsed_at":"2023-01-22T01:16:34.003Z","dependency_job_id":null,"html_url":"https://github.com/buildkite/elastic-ci-stack-s3-secrets-hooks","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buildkite%2Felastic-ci-stack-s3-secrets-hooks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buildkite%2Felastic-ci-stack-s3-secrets-hooks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buildkite%2Felastic-ci-stack-s3-secrets-hooks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buildkite%2Felastic-ci-stack-s3-secrets-hooks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/buildkite","download_url":"https://codeload.github.com/buildkite/elastic-ci-stack-s3-secrets-hooks/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223789782,"owners_count":17203259,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-kms","aws-s3","buildkite","s3","secrets"],"created_at":"2024-11-09T05:28:45.097Z","updated_at":"2024-11-09T05:28:45.773Z","avatar_url":"https://github.com/buildkite.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AWS S3 Secrets Buildkite Plugin\n\nA set of agent hooks that expose secrets to build steps via Amazon S3 (encrypted-at-rest). Used in the [Elastic CI Stack for AWS](https://github.com/buildkite/elastic-ci-stack-for-aws).\n\nDifferent types of secrets are supported and exposed to your builds in appropriate ways:\n\n- `ssh-agent` for SSH Private Keys\n- Environment Variables for strings\n- `git-credential` via git's credential.helper\n\n## Installation\n\nThe hooks needs to be installed directly in the agent so that secrets can be downloaded before jobs attempt checking out your repository. We are going to assume that buildkite has been installed at `/buildkite`, but this will vary depending on your operating system. Change the instructions accordingly.\n\nThe core of the hook is an `s3secrets-helper` binary. This can be built using\n`go build` in the [`s3secrets-helper/`](s3secrets-helper) directory in this\nrepository, or downloaded from the assets attached to a [GitHub Release](https://github.com/buildkite/elastic-ci-stack-s3-secrets-hooks/releases).\nIt must be placed in `$PATH` to be found by the `hooks/environment` wrapper script.\n\n```bash\n# clone to a path your buildkite-agent can access\ngit clone https://github.com/buildkite-plugins/s3-secrets-buildkite-plugin.git /buildkite/s3_secrets\n(cd /buildkite/s3_secrets/s3secrets-helper \u0026\u0026 go build -o /usr/local/bin/s3secrets-helper)\n```\n\nModify your agent's hooks (see [Hook Locations](https://buildkite.com/docs/agent/v3/hooks#hook-locations)):\n\n### `${BUILDKITE_ROOT}/hooks/environment`\n\n```bash\nif [[ \"${SECRETS_PLUGIN_ENABLED:-1}\" == \"1\" ]] ; then\n  export BUILDKITE_PLUGIN_S3_SECRETS_BUCKET=\"my-s3-secrets-bucket\"\n\n  source /buildkite/s3_secrets/hooks/environment\nfi\n```\n\n### `${BUILDKITE_ROOT}/hooks/pre-exit`\n\n```bash\nif [[ \"${SECRETS_PLUGIN_ENABLED:-1}\" == \"1\" ]] ; then\n  export BUILDKITE_PLUGIN_S3_SECRETS_BUCKET=\"my-s3-secrets-bucket\"\n\n  source /buildkite/s3_secrets/hooks/pre-exit\nfi\n```\n\n## Usage\n\nWhen run via the agent environment and pre-exit hook, your builds will check in the s3 secrets bucket you created for secrets files in the following formats:\n\n- `s3://{bucket_name}/{pipeline}/private_ssh_key`\n- `s3://{bucket_name}/{pipeline}/environment` or `s3://{bucket_name}/{pipeline}/env`\n- `s3://{bucket_name}/{pipeline}/git-credentials`\n- `s3://{bucket_name}/private_ssh_key`\n- `s3://{bucket_name}/environment` or `s3://{bucket_name}/env`\n- `s3://{bucket_name}/git-credentials`\n\nThe private key is exposed to both the checkout and the command as an ssh-agent instance.\nThe secrets in the env file are exposed as environment variables.\nThe locations of git-credentials are passed via `GIT_CONFIG_PARAMETERS` environment to git.\n\n## Uploading Secrets\n\n### SSH Keys\n\nThis example uploads an ssh key and an environment file to the root of the bucket, which means it matches all pipelines that use it. You use per-pipeline overrides by adding a path prefix of `/my-pipeline/`.\n\n```bash\n# generate a deploy key for your project\nssh-keygen -t rsa -b 4096 -f id_rsa_buildkite\npbcopy \u003c id_rsa_buildkite.pub # paste this into your github deploy key\n\nexport secrets_bucket=my-buildkite-secrets\naws s3 cp --acl private --sse aws:kms id_rsa_buildkite \"s3://${secrets_bucket}/private_ssh_key\"\n```\n\nNote the `-sse aws:kms`, as without this your secrets will fail to download.\n\n### Git credentials\n\nFor git over https, you can use a `git-credentials` file with credential urls in the format of:\n\n```bash\nhttps://user:password@host/path/to/repo\n```\n\n```bash\naws s3 cp --acl private --sse aws:kms \u003c(echo \"https://user:password@host/path/to/repo\") \"s3://${secrets_bucket}/git-credentials\"\n```\n\nThese are then exposed via a [gitcredential helper](https://git-scm.com/docs/gitcredentials) which will download the\ncredentials as needed.\n\n### Environment variables\n\nKey values pairs can also be uploaded.\n\n```bash\naws s3 cp --acl private --sse aws:kms \u003c(echo \"MY_SECRET=blah\") \"s3://${secrets_bucket}/environment\"\n```\n\n## Options\n\n### `bucket`\n\nAn s3 bucket to look for secrets in.\n\n## License\n\nMIT (see [LICENSE](LICENSE))\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbuildkite%2Felastic-ci-stack-s3-secrets-hooks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbuildkite%2Felastic-ci-stack-s3-secrets-hooks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbuildkite%2Felastic-ci-stack-s3-secrets-hooks/lists"}