{"id":37200598,"url":"https://github.com/allianz-direct/tekton-s3-log-reader","last_synced_at":"2026-01-14T23:08:12.882Z","repository":{"id":44095777,"uuid":"510702479","full_name":"allianz-direct/tekton-s3-log-reader","owner":"allianz-direct","description":"Reader for long-term logs stored on S3 for Tekton Dashboard","archived":false,"fork":false,"pushed_at":"2023-09-18T08:08:41.000Z","size":193,"stargazers_count":8,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-06-19T23:15:31.600Z","etag":null,"topics":["aws","cicd","s3","tekton-dashboard","tektoncd"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/allianz-direct.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-05T11:14:08.000Z","updated_at":"2023-07-25T13:04:55.000Z","dependencies_parsed_at":"2024-06-19T22:53:37.039Z","dependency_job_id":"6e77fd0b-5d91-4373-bd3a-a2238670e38c","html_url":"https://github.com/allianz-direct/tekton-s3-log-reader","commit_stats":{"total_commits":5,"total_committers":2,"mean_commits":2.5,"dds":"0.19999999999999996","last_synced_commit":"8d7a3ab220e02d191f5851daf220063a4557c989"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/allianz-direct/tekton-s3-log-reader","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allianz-direct%2Ftekton-s3-log-reader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allianz-direct%2Ftekton-s3-log-reader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allianz-direct%2Ftekton-s3-log-reader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allianz-direct%2Ftekton-s3-log-reader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/allianz-direct","download_url":"https://codeload.github.com/allianz-direct/tekton-s3-log-reader/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allianz-direct%2Ftekton-s3-log-reader/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28437927,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T22:37:52.437Z","status":"ssl_error","status_checked_at":"2026-01-14T22:37:31.496Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cicd","s3","tekton-dashboard","tektoncd"],"created_at":"2026-01-14T23:08:10.332Z","updated_at":"2026-01-14T23:08:12.200Z","avatar_url":"https://github.com/allianz-direct.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tekton-s3-log-reader\n\nThe aim of this project is to read long term logs used by Tekton Dashboard so we will be able to release resources in your Kubernetes Cluster.\n\n## Architecture\n![architecture](img/tekton-logs-arch.png)\n\n## Deploy tekton-s3-log-reader\n\nThis application needs [AWS SDK Authentication](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials) to run. In our case, we use a ServiceAccount with `eks.amazonaws.com/role-arn` annotation to authenticate with AWS services from EKS cluster.\n\nIn addition, you can use `S3_BUCKET_NAME` where the application will find the logs or one of the following arguments and `AWS_REGION` to get the session:\n\n```bash\nUsage of ./tekton-s3-log-reader:\n  -b string\n    \tBucket name or S3_BUCKET_NAME env var\n  -cert string\n    \tTLS Certificate\n  -containerd\n    \tUse containerd log format\n  -key string\n    \tTLS Key\n  -p string\n    \tPort (default \":5001\")\n  -r string\n    \tAWS Region or AWS_REGION env var\n  -tls\n    \tUse TLS to expose endpoint\n```\n\n## Observability\ntekton-s3-log-reader has `/metrics` endpoint to monitor the behaviour using Prometheus.\n\n## Sample stack configuration\n### FluentBit Configuration\n#### Docker\n```yaml\ncustomParsers: |\n    [PARSER]\n        Name         docker-custom\n        Format       json\n        Time_Key     time\n        Time_Format  %Y-%m-%dT%H:%M:%S.%L\n        Time_Keep Off\n        json_date_key false\n        # Command      |  Decoder | Field | Optional Action\n        # =============|==================|=================\n        Decode_Field_As   escaped_utf8    log    do_next\n        Decode_Field_As   json       log\nfilters: |\n    [FILTER]\n        Name record_modifier\n        Match tekton.*\n        Remove_key stream\n        Remove_key stdout\n        Remove_key time\n        Remove_key date\ninput: |\n    [INPUT]\n        Name              tail\n        Alias             tekton-semantic\n        Path              /var/log/containers/*_build-release_*\n        Parser            docker-custom\n        Tag               tekton.\u003cnamespace_name\u003e.\u003cpod_name\u003e.\u003ccontainer_name\u003e\n        Tag_Regex         (?\u003cpod_name\u003e[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?\u003cnamespace_name\u003e[^_]+)_(?\u003ccontainer_name\u003e.+)-\n        Mem_Buf_Limit     100MB\n        Refresh_Interval  60\noutputs: |\n    [OUTPUT]\n        Name            s3\n        Alias           s3_tekton_logs\n        Match           tekton.*\n        bucket          YOUR_BUCKET\n        region          eu-central-1\n        total_file_size 250M\n        upload_timeout  1m\n        s3_key_format   /$TAG[1]/$TAG[2]/$TAG[3]/%Y%m%d%H%M%S.log\n        s3_key_format_tag_delimiters .\n```\n#### Containerd\n```yaml\ncustomParsers: |\n    [PARSER]\n        # http://rubular.com/r/tjUt3Awgg4\n        Name cri-custom\n        Format regex\n        Regex ^(?\u003ctime\u003e[^ ]+) (?\u003cstream\u003estdout|stderr) (?\u003clogtag\u003e[^ ]*) (?\u003clog\u003e.*)$\n        Time_Key    time\n        Time_Format %Y-%m-%dT%H:%M:%S.%L%z\nfilters: |\n    [FILTER]\n        Name record_modifier\n        Match tekton.*\n        Remove_key logtag\n        Remove_key stream\ninputs: |\n    [INPUT]\n        Name              tail\n        Alias             tekton-semantic\n        Path              /var/log/containers/*_build-release_*\n        Parser            cri-custom\n        Tag               tekton.\u003cnamespace_name\u003e.\u003cpod_name\u003e.\u003ccontainer_name\u003e\n        Tag_Regex         (?\u003cpod_name\u003e[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?\u003cnamespace_name\u003e[^_]+)_(?\u003ccontainer_name\u003e.+)-\n        Mem_Buf_Limit     100MB\n        Refresh_Interval  60\n        #DB                /fluentbit/db/tail.tekton.db\n        # the database is accessed only by Fluent Bit\n        #DB.locking        True\n        # Skip_Long_Lines On\noutputs: |\n    \n    [OUTPUT]\n        Name            s3\n        Alias           s3_tekton_logs\n        Match           tekton.*\n        bucket          YOUR_BUCKET\n        region          eu-central-1\n        total_file_size 250M\n        upload_timeout  1m\n        s3_key_format   /$TAG[1]/$TAG[2]/$TAG[3]/%Y%m%d%H%M%S.log\n        s3_key_format_tag_delimiters .\n```\n### Tekton Dashboard Configuration\n\nAdd the following flag pointing to the endpoint `tekton-s3-log-reader`: \n```yaml\n--external-logs=http://tekton-s3-log-reader.monitoring:5001/logs\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallianz-direct%2Ftekton-s3-log-reader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fallianz-direct%2Ftekton-s3-log-reader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallianz-direct%2Ftekton-s3-log-reader/lists"}