{"id":20852720,"url":"https://github.com/ninesstack/logtailer","last_synced_at":"2026-02-16T14:02:14.736Z","repository":{"id":233805765,"uuid":"747264022","full_name":"NinesStack/logtailer","owner":"NinesStack","description":"Discovers pod logs and follows them, sending output to UDP syslog","archived":false,"fork":false,"pushed_at":"2025-01-06T18:10:43.000Z","size":441,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-24T17:52:37.457Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NinesStack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":"support_test.go","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-23T15:39:01.000Z","updated_at":"2025-01-06T18:10:39.000Z","dependencies_parsed_at":"2024-04-17T11:32:07.746Z","dependency_job_id":"25afb957-ca4a-48ed-bd3c-4b22718907da","html_url":"https://github.com/NinesStack/logtailer","commit_stats":null,"previous_names":["ninesstack/logtailer"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NinesStack%2Flogtailer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NinesStack%2Flogtailer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NinesStack%2Flogtailer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NinesStack%2Flogtailer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NinesStack","download_url":"https://codeload.github.com/NinesStack/logtailer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248288340,"owners_count":21078900,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-18T03:18:43.305Z","updated_at":"2026-02-16T14:02:14.712Z","avatar_url":"https://github.com/NinesStack.png","language":"Go","readme":"logtailer\n=========\n\n**The problem this solves**\n\nYou have a Kubernetes cluster and you want to get all the logs for your\nservices. You don't necessarily want/need to ship all the logs, you only want\nthem for a selected list of pods. You want to prevent runaway logging by one\nservice from degrading logging for everyone else, and you don't want to blow\nyour logging budget every time someone mistakenly deploys with debug logging\nenabled.\n\nDownloading\n-----------\n\nThe simplest way to get `logtailer` is from the Docker images pushed to\nGitHub packages: https://github.com/NinesStack/logtailer/pkgs/container/logtailer\n\nAn example `manifest.yml` for deploying it is included in this source repo.\n\nMore Info\n---------\n\nThis is an **opinionated** service that is intended to run as a `DaemonSet` on\nKubernetes. It provides basic syslogging capabilities for services that don't\nsupport that themselves. It watches logs in `/var/log/pods` and then tails the\nlogs that appear, if the pod has annotations that tell it to do so. It needs to\nrun as a service user that has access to query all pods in all namespaces in\norder to read those annotations.\n\nThe following decisions were made and are implemented here:\n\n * All logs will be sent over UDP. The format is not real syslog, but many\n   log processing services handle this. For our use case, this works better\n   than actual syslog.\n\n * Logs will be rate limited\n\n * The wrapper for log output will be JSON. If you are already encoding logs\n   in JSON format, they will be wrapped in an outder JSON layer containing the\n   metadata.\n \n * You are using `containerd` as the runtime on your K8s cluster.\n\nHow It Works\n-------------\n\n 1. A `PodTracker` watches `/var/log/pods` with the `DirListDiscoverer` to find\n    any new pods\n \n 2. The `PodTracker` queries the `PodFilter` if it has a newly discovered pod,\n    in order to determine if it should indeed track this pod. The only\n    `PodFilter` currently implemented makes a call to the Kubernetes API to look\n    at the annotations on the pod. It expects a label called `ServiceName` to be\n    used for filtering. This may or may not match `app` labels or similar that\n    you already have.\n \n 3. In the event that it should track the pod, it starts up a `LogTailer` on\n    each log file discovered by the `DirListDiscoverer`\n \n 4. The `LogTailer` is instantiated with a `LogOutput` that specifies what to do\n    with log lines when they are retrieved. This is configured with a\n    `UDPSyslogger` output built around `logrus`. This is an interface so it's\n    entirely possible to implement different `LogOutput`s.\n \n 5. Wrapped around the `UDPSyslogger` is a `RateLimitingLogger` that prevents\n    logs from overruning the upstream. This is configurable with environment\n    variables, like the rest of the service.\n \n 6. When new log lines arrive, the `UDPSyslogger` strips off the Kubernetes-\n    specific preamble, and sends the remaining line wrapped in JSON with the\n    specified additional fields present. This is the format we always send to\n    Sumo Logic.\n\nLogging Format\n--------------\n\nThe UDP-relayed output is of the following form:\n\n```\n{\n   \"Container\" : \"thecontainer\",\n   \"Environment\" : \"dev\",\n   \"Hostname\" : \"ip-10-1-11-123.us-west-2.compute.internal\",\n   \"Level\" : \"info\",\n   \"Payload\" : \"[2024-04-17T10:02:25 (agent) #1][Info] service: Received HTTP request\",\n   \"PodName\" : \"default_service-74654768f-hqwwj_534d2d95-8eaa-4385-878e-c031bdc1c5c6\",\n   \"ServiceName\" : \"your-service\",\n   \"Timestamp\" : \"2024-04-17T10:02:25Z\"\n}\n```\n\n`Payload` contains the raw, original log line stripped of the\nKubernetes/containerd preamble.\n\nConfiguration\n-------------\n\nThis is configured with environment variables. They are all in `main.go` in\nthe `Config` struct at the top.\n\nServices can be configured to have their logs tailed by `logtailer` by\nadding an annotation to the template in the `Deployment` (or similar\nfor other workloads):\n\n```\ncommunity.com/TailLogs=true\n```\n\n`logtailer` will discover those annotations and enable log tailing and\nsyslogging.\n\nRate Limiter Reporting\n----------------------\n\nYou can configure `logtailer` to report to New Relic Insights for reporting on\nlog rate limiting. This is of course quite limiting for those who are not New\nRelic customers. A more open standard reporter will be forthcoming.\n\nEnhanced Log Level Extraction\n-----------------------------\n\nYou can configure `logtailer` to attempt to extract log levels with an enhanced regex\nfilter by setting the environment variable `ENABLE_REGEX_LOG_LEVEL_PARSING=true`.\nBy default this mode is disabled.\n\nRunning Locally for Testing\n---------------------------\n\nYou can invoke this from the current directory using the test fixtures\nto represent logfiles:\n\n```\nCACHE_FILE_PATH=./logtailer.json BASE_PATH=fixtures/pods DEBUG=true ./logtailer\n```\n\nIt will fail to call Kubernetes for filtering and will proceed with a\n`StubFilter` in place that always returns `true`.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fninesstack%2Flogtailer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fninesstack%2Flogtailer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fninesstack%2Flogtailer/lists"}