{"id":15710359,"url":"https://github.com/aws/shim-loggers-for-containerd","last_synced_at":"2025-04-06T18:15:36.893Z","repository":{"id":37794792,"uuid":"232183960","full_name":"aws/shim-loggers-for-containerd","owner":"aws","description":"Shim logger repository for streaming container logs when using Containerd","archived":false,"fork":false,"pushed_at":"2025-03-28T01:57:14.000Z","size":931,"stargazers_count":50,"open_issues_count":15,"forks_count":32,"subscribers_count":65,"default_branch":"main","last_synced_at":"2025-03-30T17:08:48.355Z","etag":null,"topics":["containerd","containers","fargate","golang"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aws.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-06T20:56:16.000Z","updated_at":"2025-03-21T14:55:04.000Z","dependencies_parsed_at":"2023-09-21T21:30:54.836Z","dependency_job_id":"a5c8b405-f525-46ea-9a06-3eb55dcfc276","html_url":"https://github.com/aws/shim-loggers-for-containerd","commit_stats":{"total_commits":132,"total_committers":25,"mean_commits":5.28,"dds":0.6515151515151515,"last_synced_commit":"e2edbb248c64324e5d197f8e3de05a639237680e"},"previous_names":["aws/shim-loggers-for-containerd","aws/amazon-ecs-shim-loggers-for-containerd"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws%2Fshim-loggers-for-containerd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws%2Fshim-loggers-for-containerd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws%2Fshim-loggers-for-containerd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aws%2Fshim-loggers-for-containerd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aws","download_url":"https://codeload.github.com/aws/shim-loggers-for-containerd/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247526764,"owners_count":20953143,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["containerd","containers","fargate","golang"],"created_at":"2024-10-03T21:06:37.965Z","updated_at":"2025-04-06T18:15:36.855Z","avatar_url":"https://github.com/aws.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Shim loggers for containerd\n\n[![PkgGoDev](https://pkg.go.dev/badge/github.com/aws/shim-loggers-for-containerd)](https://pkg.go.dev/github.com/aws/shim-loggers-for-containerd)\n[![Go Report Card](https://goreportcard.com/badge/github.com/aws/shim-loggers-for-containerd)](https://goreportcard.com/report/github.com/aws/shim-loggers-for-containerd)\n[![Build](https://github.com/aws/shim-loggers-for-containerd/actions/workflows/ci.yaml/badge.svg)](https://github.com/aws/shim-loggers-for-containerd/actions/workflows/ci.yaml)\n[![Static Badge](https://img.shields.io/badge/Website-Benchmarks-blue)](https://aws.github.io/shim-loggers-for-containerd/dev/bench/)\n\nShim loggers for containerd is a collection of [containerd](https://github.com/containerd/containerd) compatible logger\nimplementations that send container logs to various destinations. The following destinations are currently supported:\n\n* [Amazon CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html)\n* [Splunk](https://www.splunk.com/en_us/central-log-management.html)\n* [Fluentd](http://www.fluentd.org/)\n\n## Build\n\nMake sure you have [golang](https://golang.org) installed. Then simply run `make build` to build the respective binaries. You might need to execute `make get-deps` to install some of the dependencies.\n\n## Usage\n\nContainerd supports shim plugins that redirect container output to a custom binary on Linux using STDIO URIs with\n[runc v2 runtime](https://github.com/containerd/containerd/tree/release/1.3/runtime/v2). These loggers can be used\neither programmatically or with the [ctr](https://github.com/projectatomic/containerd/blob/master/docs/cli.md) tool.\n\n### When using the `NewTask` API\n\nWhen using the [`NewTask`](https://github.com/containerd/containerd/blob/release/1.3/container.go#L208) API\nto start a container, simply provide the path to the built binary file `shim-loggers-for-containerd` and required\narguments. Note it's a good practice to clean up container resources with\n[`Delete`](https://github.com/containerd/containerd/blob/release/1.3/task.go#L287) API call after container exited\nas the container IO pipes are not closed if the shim process is still running.\n\nExample:\n\n```go\nNewTask(context, cio.BinaryIO(\"/usr/bin/shim-loggers-for-containerd\", args))\n```\n\n### When using the `ctr` tool\n\nWhen using [ctr](https://github.com/projectatomic/containerd/blob/master/docs/cli.md) tool to run\na container, provide the URI path to the binary file `shim-loggers-for-containerd` and required arguments as part of\nthe path.\n\nExample:\n\n```bash\nctr run \\\n    --runtime io.containerd.runc.v2 \\\n    --log-uri \"binary:///usr/bin/shim-loggers-for-containerd?--log-driver=awslogs\u0026--arg1=value1\u0026-args2=value2\" \\\n    docker.io/library/redis:alpine \\\n    redis\n```\n\n## Arguments\n\n### Common arguments\n\nThe following list of arguments apply to all of the shim logger binaries in this repo:\n\n|Name|Required|Description|\n|-|-|-|\n| log-driver | Yes | The name of the shim logger. Can be any of `awslogs`, `splunk` or `fluentd`. |\n| container-id | Yes | The container id |\n| container-name | Yes | The name of the container |\n| mode | No | Either `blocking` or `non-blocking`. In the `non-blocking` mode, log events are buffered and the application continues to execute even if these logs can't be drained or sent to the destination. Logs could also be lost when the buffer is full. |\n| max-buffer-size | No | Only supported in `non-blocking` mode. Set to `1m` (1MiB) by default. Example values: `200`, `4k`, `1m` etc. |\n| uid | No | Set a custom uid for the shim logger process. `0` is not supported. |\n| gid | No | Set a custom gid for the shim logger process. `0` is not supported. |\n| cleanup-time | No | Set a custom time for the shim logger process clean up itself. Set to `5s` (5 seconds) by default. Note the maximum supported value is 12 seconds, since containerd shim sets shim logger cleanup timeout value as 12 seconds. See [reference](https://github.com/containerd/containerd/commit/0dc7c8595627e38ca2b83d17a062b51f384c2025). |\n| container-image-id | No | The container image id. This is part of the docker config variables that can be logged by splunk log driver. |\n| container-image-name | No | The container image name. This is part of the docker config variables that can be logged by splunk log driver. |\n| container-env | No | The container environment variables map in json format. This is part of the docker config variables that can be logged by splunk log driver. |\n| container-labels | No | The container labels map in json format. This is part of the docker config variables that can be logged by splunk log driver. |\n\n### Windows specific arguments\n\nThe following list of arguments apply to Windows shim logger binaries in this repo:\n\n|Name|Required|Description|\n|-|-|-|\n| log-file-dir | No | Only supported in Windows. Will be the path where shim logger's log files are written. By default it is `\\ProgramData\\Amazon\\ECS\\log\\shim-logger`\n| proxy-variable | No | Only supported in Windows. The proxy variable will set the `HTTP_PROXY` and `HTTPS_PROXY` environment variables.\n\n### Additional log driver options\n\n#### Amazon CloudWatch Logs\n\nThe following additional arguments are supported for the `awslogs` shim logger binary, which can be used to send container logs to [Amazon CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html).\n\n| Name                         | Required | Description                                                                                                                                                                                                                                     |\n|------------------------------|----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| awslogs-group                | Yes      | The [log group](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatchLogsConcepts.html) in which the log stream for the container will be created.                                                                                |\n| awslogs-stream               | Yes      | The [log stream name](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatchLogsConcepts.html) to stream container logs to.                                                                                                        |\n| awslogs-region               | Yes      | The region name in which the log group and log stream needs to be created in.                                                                                                                                                                   |\n| awslogs-credentials-endpoint | Yes      | The endpoint from which credentials are retrieved from to connect to Amazon CloudWatch Logs.                                                                                                                                                    |\n| awslogs-create-group         | No       | Set to `false` by default. If the provided log group name does not exist and this value is set to `false`, the binary will directly exit with an error                                                                                          |\n| awslogs-create-stream        | No       | Set to `true` by default. The log stream will always be created unless this value specified to `false` explicitly. If the value is `false` and the log stream does not exist, logging will fail silently instead of failing the container task. |\n| awslogs-multiline-pattern    | No       | Matches the behavior of the [`awslogs` Docker log driver](https://docs.docker.com/config/containers/logging/awslogs/#amazon-cloudwatch-logs-options#awslogs-multiline-pattern).                                                                 |\n| awslogs-datetime-format      | No       | Matches the behavior of the [`awslogs` Docker log driver](https://docs.docker.com/config/containers/logging/awslogs/#amazon-cloudwatch-logs-options#awslogs-datetime-format)                                                                    |\n| awslogs-endpoint             | No       | Matches the behavior of the [`awslogs` Docker log driver](https://docs.docker.com/config/containers/logging/awslogs/#awslogs-endpoint)                                                                                                          |\n\n#### Splunk\n\nThe following additional arguments are supported for the `splunk` shim logger binary, which can be used to send container logs to [splunk](https://www.splunk.com/en_us/central-log-management.html).\nYou can find a description of what these parameters are used for [here](https://docs.docker.com/config/containers/logging/splunk/).\n\n|Name|Required|\n|-|-|\n| splunk-token | Yes |\n| splunk-url | Yes |\n| splunk-source | No |\n| splunk-sourcetype | No |\n| splunk-index | No |\n| splunk-capath | No |\n| splunk-caname | No |\n| splunk-insecureskipverify | No |\n| splunk-format | No |\n| splunk-verify-connection | No |\n| splunk-gzip | No |\n| splunk-gzip-level | No |\n| splunk-tag | No |\n| labels | No |\n| env | No |\n| env-regex | No |\n\n#### Fluentd\n\nThe following additional arguments are supported for the `fluentd` shim logger binary, which can be used to send container logs to  [Fluentd](https://www.fluentd.org). Note that all of these are optional arguments.\n\n| Name                         | Required | Description                                                                                                                                                   |\n|------------------------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| fluentd-address              | No       | The address of the Fluentd server to connect to. By default, the `localhost:24224` address is used.                                                           |\n| fluentd-async-connect        | No       | Specifies if the logger connects to Fluentd in background. Defaults to `false`.                                                                               |\n| fluentd-sub-second-precision | No       | Generates logs in nanoseconds. Defaults to `true`. Note that this is in contrast to the default behaviour of fluentd log driver where it defaults to `false`. |\n| fluentd-buffer-limit         | No       | Sets the number of events buffered in memory. The total memory limit is approximately this limit * the average log line length. Defaults to `1048576`.        |\n| fluentd-tag                  | No       | Specifies the tag used for log messages. Defaults to the first 12 characters of container ID.                                                                 |\n\n## License\n\nThis project is licensed under the Apache-2.0 License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faws%2Fshim-loggers-for-containerd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faws%2Fshim-loggers-for-containerd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faws%2Fshim-loggers-for-containerd/lists"}