{"id":19142270,"url":"https://github.com/martinbaillie/ocistow","last_synced_at":"2025-08-18T08:04:27.469Z","repository":{"id":65146315,"uuid":"422176972","full_name":"martinbaillie/ocistow","owner":"martinbaillie","description":"Stream, Mutate and Sign Images with AWS Lambda and ECR","archived":false,"fork":false,"pushed_at":"2021-10-28T11:27:31.000Z","size":571,"stargazers_count":19,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-06T23:49:51.016Z","etag":null,"topics":["aws","aws-lambda","cosign","docker","ecr","oci","sigstore","slsa","supplychain"],"latest_commit_sha":null,"homepage":"https://martin.baillie.id/wrote/stream-mutate-and-sign-images-with-aws-lambda-and-ecr/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/martinbaillie.png","metadata":{"files":{"readme":"README.org","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-10-28T11:26:32.000Z","updated_at":"2024-07-10T10:33:44.000Z","dependencies_parsed_at":"2023-01-01T00:33:11.542Z","dependency_job_id":null,"html_url":"https://github.com/martinbaillie/ocistow","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/martinbaillie/ocistow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martinbaillie%2Focistow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martinbaillie%2Focistow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martinbaillie%2Focistow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martinbaillie%2Focistow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/martinbaillie","download_url":"https://codeload.github.com/martinbaillie/ocistow/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/martinbaillie%2Focistow/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270962391,"owners_count":24675965,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-18T02:00:08.743Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-lambda","cosign","docker","ecr","oci","sigstore","slsa","supplychain"],"created_at":"2024-11-09T07:26:34.942Z","updated_at":"2025-08-18T08:04:27.438Z","avatar_url":"https://github.com/martinbaillie.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"* ocistow :TOC_3_gh:\n- [[#about][About]]\n- [[#how-it-works][How it works]]\n- [[#try-it-yourself][Try it yourself]]\n  - [[#prerequisites][Prerequisites]]\n  - [[#cli-cmdocistow][CLI (cmd/ocistow)]]\n  - [[#lambda-cmdocistow-lambda][Lambda (cmd/ocistow-lambda)]]\n    - [[#deploy][Deploy]]\n    - [[#invoke][Invoke]]\n  - [[#verify-signatures-with-cosign][Verify signatures with =cosign=]]\n  - [[#insight][Insight]]\n\n* About\nThe =ocistow= codebase demonstrates a modern approach to a typical OCI image\npromotion workflow in AWS.\n\nIt houses a Lambda (and bonus CLI) that can efficiently stream and mutate\nupstream container image layers into an ECR destination and subsequently sign\nthem with KMS. It does so by utilising code from the excellent\n[[https://github.com/google/go-containerregistry][go-containerregistry]] and [[https://github.com/sigstore][sigstore]] projects.\n\n* How it works\n[[img/ocistow.png][img/ocistow.png]]\nGiven an invoke payload of:\n1. A source image reference (any public container registry / private ECR)\n2. A destination image reference (private ECR)\n3. Some annotations to add\n\nIt will:\n1. Stream only the missing image layers from source registry to destination ECR whilst handling ECR authentication\n2. Do so in memory, layer-by-layer[fn:1] (with Lambda's meagre 512mb filesystem remaining unused)\n3. Optionally mutate the image during this process to have user provided OCI\n   annotations and legacy Docker image labels (mimicking the sort of mandatory\n   tagging policy an organisation might have)\n4. And finally sign the image digests in AWS ECR using a KMS signing key for\n   later assertion of provenance at runtime (e.g. using a Kubernetes admission\n   controller like [[https://github.com/dlorenc/cosigned][cosigned]]).\n\n[fn:1]: Performance gains can be had by throwing more memory at the Lambda as\nthis results in more allocated CPU and critically, network (at AWS' discretion).\nEmpirically (though not very scientifically), I saw the following with the\nmassive *3+ gigabyte* TensorFlow images from [[https://gcr.io][gcr.io]].\n\n- Test 1 (vanilla Lambda settings 128mb memory): 8.06 minutes\n- Test 2 (maxed out Lambda settings 10240mb memory): *1.35 minutes*\n\nNo shared layers existed in my destination ECR between tests—all blobs were\nstreamed from source to destination.\n\n#+begin_quote\nNOTE: This would be interesting to give a run through the [[https://github.com/alexcasalboni/aws-lambda-power-tuning][AWS Lambda Power Tuner]].\n#+end_quote\n* Try it yourself\nI can’t imagine anyone using the Lambda (nor CLI) verbatim in their workflow\nunless it happened to solve an exact gap (let me know if you do!), but the\ncodebase may be a useful reference for informing your own build.\n\nFor example, a Lambda /like/ =ocistow= could be that final \"promotion\" step in\nan organisation's container image supply chain which first involves the image\nrunning a gauntlet of vulnerability/malware/compliance scans in a Step Function\nstate machine.\n\nHowever, outside of the Lambda space, CLIs like Google's [[https://https://github.com/google/go-containerregistry/blob/main/cmd/crane/doc/crane.md][crane]] and Sigstore's\n[[https://github.com/sigstore/cosign][cosign]] are much more polished and suitable to a range of container image\ncopy/mutate/sign workflows. You should check them out.\n** Prerequisites\nAn AWS account with a KMS signing key and ECR repository for playing with. The\nfollowing =aws= incantations will do the trick presuming you have an\nappropriately privileged session.\n\n#+begin_src shell\naws kms create-key \\\n    --key-usage SIGN_VERIFY \\\n    --customer-master-key-spec RSA_4096 \\\n    --tags TagKey=Name,TagValue=ocistow \\\n    --description \"ocistow demo\"\n\naws ecr create-repository --repository-name ocistow-demo\n#+end_src\n** CLI (cmd/ocistow)\nAs I was extracting the Lambda out from some larger research to post here, I\nrealised it would be easy to add a CLI. The =ocistow= CLI can do the same thing\nas the Lambda but from your local machine. Though, unless it fits your exact use\ncase, you may wish to just reference it for your own CLI implementation or otherwise\nreach for the much more practical Google [[https://github.com/google/go-containerregistry/blob/main/cmd/crane/doc/crane.md][crane]].\n\nIn any case, you can get a build of =ocistow= from the [[https://github.com/martinbaillie/ocistow/releases][releases]] section or build and run\nyourself (the repository is a [[./flake.nix][Nix flake]] if that's your thing, otherwise you'll\nwant a local Go 1.16+ toolchain):\n\n#+begin_src shell\ngo \u003cbuild|install|run\u003e github.com/martinbaillie/ocistow/cmd/ocistow -h\n#+end_src\n\n#+begin_example\nUsage of ocistow:\n  -annotations value\n        destination image annotations (key=value)\n  -aws-kms-key-arn string\n        AWS KMS key ARN to use for signing\n  -aws-region string\n        AWS region to use for operations\n  -aws-xray\n        whether to enable AWS Xray tracing\n  -copy\n        whether to copy the image (default true)\n  -debug\n        debug logging\n  -destination string\n        destination image\n  -sign\n        whether to sign the image (default true)\n  -source string\n        source image\n#+end_example\n\nKick the tyres by stowing DockerHub's =busybox:latest= into the demo ECR repository:\n#+begin_src shell\nocistow \\\n    -source=busybox \\\n    -destination=111111111111.dkr.ecr.ap-southeast-2.amazonaws.com/ocistow-demo \\\n    -aws-region=ap-southeast-2 \\\n    -aws-kms-key-arn=\"\u003cARN from Prerequisites\u003e\" \\\n    -annotations team=foo \\\n    -annotations owner=martin\n#+end_src\n\n#+begin_example\n21:09:00.000  annotations={\"owner\":\"martin\",\"team\":\"foo\"} component=service dst=111111111111.dkr.ecr.ap-southeast-2.amazonaws.com/ocistow-demo method=Copy src=busybox took=2.999482083s\n21:09:00.000  annotations={\"owner\":\"martin\",\"team\":\"foo\"} component=service dst=111111111111.dkr.ecr.ap-southeast-2.amazonaws.com/ocistow-demo method=Sign took=878.351667ms\n#+end_example\n\n** Lambda (cmd/ocistow-lambda)\n*** Deploy\nFor playing with the =ocistow-lambda= in your AWS account you can use the [[./env][CDK\ndeployment]] in this repository. There's a [[./Makefile][Makefile]] target that can kick this\nprocess off (though like the CLI instructions above, either use the Nix flake or\nget yourself Go 1.16+ and additionally NodeJS for the CDK).\n\n#+begin_src shell\nmake deploy AWS_KMS_KEY_ARN=\"\u003cARN from Prerequisites\u003e\"\n#+end_src\n\nThis will build and deploy an aarch64/Graviton version of the Lambda to your\naccount with necessary KMS/ECR permissions. Take note of the function ARN output\nfor later invocation.\n*** Invoke\nKick the tyres by stowing DockerHub's =busybox:latest= into the demo ECR repository:\n\n#+begin_quote\nNOTE: The Lambda expects a very simple [[https://github.com/martinbaillie/ocistow/blob/main/pkg/transport/lambda.go#L15-L19][JSON schema]] as its payload.\n#+end_quote\n\n#+begin_src shell\naws lambda invoke \\\n    --function-name \"arn:aws:lambda:ap-southeast-2:111111111111:function:ocistow-function\" --cli-binary-format raw-in-base64-out \\\n    --payload '{\n        \"SrcImageRef\":\"busybox\",\n        \"DstImageRef\": \"111111111111.dkr.ecr.ap-southeast-2.amazonaws.com/ocistow-demo\",\n        \"Annotations\":{\"team\":\"foo\", \"owner\":\"martin\"}\n        }' /dev/stderr\n#+end_src\n\n#+begin_example\n{\n    \"StatusCode\": 200,\n    \"ExecutedVersion\": \"$LATEST\"\n}\n#+end_example\n\n** Verify signatures with =cosign=\n#+begin_src shell\nAWS_REGION=ap-southeast-2 cosign verify \\\n    -key \"\u003cARN from Prerequisites\u003e\" \\\n    111111111111.dkr.ecr.ap-southeast-2.amazonaws.com/ocistow-demo\n#+end_src\n\n#+begin_example\nVerification for 111111111111.dkr.ecr.ap-southeast-2.amazonaws.com/ocistow-demo --\nThe following checks were performed on each of these signatures:\n  - The cosign claims were validated\n  - The signatures were verified against the specified public key\n  - Any certificates were verified against the Fulcio roots.\n\n[{\"critical\":{\"identity\":{\"docker-reference\":\"111111111111.dkr.ecr.ap-southeast-2.amazonaws.com/ocistow-demo\"},\"image\":{\"docker-manifest-digest\":\"sha256:ee16ac0396cdb32e870200cdcb30f9abcb6b95256e5b5cd57eb1fadf2d3b3c9d\"},\"type\":\"cosign container image signature\"},\"optional\":{\"team\":\"foo\",\"owner\":\"martin\"}}]\n#+end_example\n\n** Insight\nA [[https://stripe.com/blog/canonical-log-lines][canonical log line]] is output for each service method (Copy, Sign) which you'll find on the terminal output for CLI and in CloudWatch for Lambda.\n\nIf debug logging is enabled (flag: =-debug=, env: =DEBUG=) then much more\ndetailed output is made available from the backend libraries used.\n\nIf AWS Xray is enabled (flag: =-aws-xray=, env: =AWS_XRAY=) then detailed traces\nof the layer-by-layer =ocistow= operations are also propagated:\n\n[[img/segments.png][img/segments.png]]\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmartinbaillie%2Focistow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmartinbaillie%2Focistow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmartinbaillie%2Focistow/lists"}