{"id":18834578,"url":"https://github.com/controlplane-com/accesslog-filter","last_synced_at":"2026-01-26T12:30:17.782Z","repository":{"id":139791707,"uuid":"518822383","full_name":"controlplane-com/accesslog-filter","owner":"controlplane-com","description":null,"archived":false,"fork":false,"pushed_at":"2022-07-28T11:43:58.000Z","size":10,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-12-30T07:42:28.762Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/controlplane-com.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-28T11:41:37.000Z","updated_at":"2022-07-28T11:42:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"842e5214-96be-41fb-8161-ae1cb03cb4d0","html_url":"https://github.com/controlplane-com/accesslog-filter","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/controlplane-com%2Faccesslog-filter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/controlplane-com%2Faccesslog-filter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/controlplane-com%2Faccesslog-filter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/controlplane-com%2Faccesslog-filter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/controlplane-com","download_url":"https://codeload.github.com/controlplane-com/accesslog-filter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239768970,"owners_count":19693772,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T02:13:19.660Z","updated_at":"2026-01-26T12:30:17.734Z","avatar_url":"https://github.com/controlplane-com.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Accesslog filter\n\nThis project contains a small AWS lambda function + the necessary Terraform boilerplate to deploy it.\n\nThe function listens to S3 events from a source bucket, filters out irrelevant log events and finally persists the rest in a destination bucket.\n\nIt is usually used with Control Plane's `External Logs` feature when the external log sink is an S3 bucket.\n\n# How to deploy\n\nYou need to pass terraform 3 variables. Consult the file `example.tfvars` for an example:\n\n```terraform\nregion = \"eu-central-1\"\n\nsource_uri = \"s3://julian-log-router-test/cpln-test/\"\n\n# AWS advises agains using the same bucket for input/output of a function\ndest_uri = \"s3://julian-log-router-filtered/access-logs-only\"\n```\n\n- `region`: the region where both the source and destination bucket exist. This region will also host the lambda function.\n- `source_uri`: an S3 path in the format `s3://bucket/path/to/what/ever`. Only files with this key prefix are considered. Used to filter by org.\n- `dest_uri`: an S3 path in the format `s3://bucket/path/to/what/ever`. Files will end up here after filtering.\n\n# Prerequisites\n\n- At least Terraform 0.14\n- Both buckets must exist. You probably want to setup retention policy anyway\n- The role under which you run TF must be able to create Roles and Policies (IAM)\n\n# What gets created under my account?\n\n- A lambda function\n- A role for the function (cloudwatch access, read access to source bucket, write access to dest bucket)\n- A trigger on the source bucket connected to the lambda\n\n# How are log files processed?\n\nThe processing just removes all logs where the container is not `_accesslog` but you are free to change it to suit your needs.\nSince the format of the log files is always `jsonl`, a simple string search is used to detect matches thus saving CPU on JSON parsing.\n\nAn empty file is just not written to the destination bucket. In CloudWatch, you can recognize it easily:\n\n```\n2022-07-28T14:15:13.018+03:00\tfile prefix/my-org/2022/07/28/11/05/XrowrWy2.jsonl does not contain access logs\n```\n\nThe lambda function tries its best to preserve the key structure of the processed files. For example if the input file is `{source_uri}/acme-org/2022/07/28/04/30/8Gdd2dwZ.jsonl`, then the resulting file will end up in `{dest_uri}/acme-org/2022/07/28/04/30/8Gdd2dwZ.jsonl`.\n\nFiles are gzipped (level 9) before uploading them to the destinatiob bucket.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontrolplane-com%2Faccesslog-filter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcontrolplane-com%2Faccesslog-filter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontrolplane-com%2Faccesslog-filter/lists"}