{"id":19723506,"url":"https://github.com/remind101/logspout-kinesis","last_synced_at":"2025-06-23T02:07:14.729Z","repository":{"id":35692578,"uuid":"39969401","full_name":"remind101/logspout-kinesis","owner":"remind101","description":"A Logspout adapter for writing logs to Amazon Kinesis","archived":false,"fork":false,"pushed_at":"2016-05-11T22:56:30.000Z","size":590,"stargazers_count":16,"open_issues_count":4,"forks_count":2,"subscribers_count":76,"default_branch":"master","last_synced_at":"2025-04-29T22:36:31.885Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/remind101.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-07-30T19:49:29.000Z","updated_at":"2017-11-10T17:46:24.000Z","dependencies_parsed_at":"2022-09-03T23:51:51.891Z","dependency_job_id":null,"html_url":"https://github.com/remind101/logspout-kinesis","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/remind101/logspout-kinesis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/remind101%2Flogspout-kinesis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/remind101%2Flogspout-kinesis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/remind101%2Flogspout-kinesis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/remind101%2Flogspout-kinesis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/remind101","download_url":"https://codeload.github.com/remind101/logspout-kinesis/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/remind101%2Flogspout-kinesis/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261397374,"owners_count":23152489,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T23:22:25.929Z","updated_at":"2025-06-23T02:07:09.693Z","avatar_url":"https://github.com/remind101.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# logspout-kinesis [![Circle CI](https://circleci.com/gh/remind101/logspout-kinesis.svg?style=svg)](https://circleci.com/gh/remind101/logspout-kinesis)\n\nA [Logspout](https://github.com/gliderlabs/logspout) adapter for writing Docker container logs to [Amazon Kinesis](http://aws.amazon.com/kinesis/).\n\n## usage\nAfter you created your custom Logspout build (see below), you can just run it as:\n```console\n$ \tdocker run --rm \\\n\t\t-e LOGSPOUT=ignore \\\n\t\t--name=\"logspout\" \\\n\t\t--volume=/var/run/docker.sock:/var/run/docker.sock \\\n\t\tmycompany/logspout kinesis://\n```\n\nYou will need the Amazon credentials in the form of environment variables, or whatever suits you.\nYou will also need to set the environment variables `KINESIS_STREAM_TEMPLATE` and `KINESIS_PARTITION_KEY_TEMPLATE`, see below.\n\n### streams and partition key configuration\nYou can decide to route your logs to one or multiple Kinesis Streams (and their names) via the the `KINESIS_STREAM_TEMPLATE` environment variable.  Golang's [text/template](http://golang.org/pkg/text/template/) package is used for templating, where the model available for templating is the [Message struct](https://github.com/gliderlabs/logspout/blob/master/router/types.go).\n\nFor instance, you can decide to assign a Kinesis stream for each of you app (assuming you have a `app` label on your running containers:\n```console\n$ export KINESIS_STREAM_TEMPLATE={{ index .Container.Config.Labels \"app\" }}\n```\n\nOr you can use environment variables, with our provided lookUp template function:\n```console\n$ export KINESIS_STREAM_TEMPLATE={{ lookUp .Container.Config.Env \"APP_ID\" }}\n```\n\nThis will search through `.Container.Config.Env` for `APP_ID=*` and return the value after the `=`.\n\n**IMPORTANT**: if executing the stream name template results in an empty string, logspout-kinesis won't stream the log message. If debug logging is activated (see below), it will tell you so.\n\nYou can similarly decide the format of your partition key for each of your stream (here the app name and process type):\n```console\n$ export KINESIS_PARTITION_KEY_TEMPLATE={{ index .Container.Config.Labels \"app\" }}.{{ index .Container.Config.Labels \"process.type\" }}\n```\n\n**IMPORTANT**: if the partition key end up being an empty string, logspout-kinesis will default to set it as a uuid. If debug logging is activated (see below), it will tell you so.\n\n### stream creation\nBy default, logspout-kinesis **will** create a stream if it is missing from Kinesis.\n\n### stream tagging\nBy default, logspout-kinesis **will** tag a stream it just created.\n\nYou can set the `KINESIS_STREAM_TAG_KEY` environment variable to set the tag key, and the `KINESIS_STREAM_TAG_VALUE` variable to set the template you want to use for the tag value.\n\nA complete example would be:\n```console\nKINESIS_STREAM_TAG_KEY=\"app\"\nKINESIS_STREAM_TAG_VALUE={{ lookUp .Container.Config.Env \"EMPIRE_APPNAME\" }}\n```\n\n### logging\nTo activate logging, set the `KINESIS_DEBUG` environment variable to `true`.\n\n## build\n**logspout-kinesis** is a custom logspout module. To use it, create an empty Dockerfile based on `gliderlabs/logspout`, and import this `logspout-kinesis` package into a new `modules.go` file. The `gliderlabs/logspout` base image will `ONBUILD COPY` and replace the original `modules.go`.\n\nThe following example creates a minimal logspout image capable of writing Dockers logs to Kinesis:\n\nIn modules.go:\n```golang\npackage main\n\nimport (\n\t_ \"github.com/gliderlabs/logspout/httpstream\"\n\t_ \"github.com/gliderlabs/logspout/routesapi\"\n\t_ \"github.com/remind101/logspout-kinesis\"\n)\n```\n\nIn Dockerfile:\n```Dockerfile\nFROM gliderlabs/logspout:v3\n```\n\nFinal step, build the image:\n```console\n$ docker build -t mycompany/logspout .\n```\n\nMore information on custom modules is available at the Logspout repo: [Custom Logspout Modules](https://github.com/gliderlabs/logspout/tree/master/custom)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fremind101%2Flogspout-kinesis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fremind101%2Flogspout-kinesis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fremind101%2Flogspout-kinesis/lists"}