{"id":22816505,"url":"https://github.com/msysh/aws-sample-scheduled-cloudwatch-logs-export","last_synced_at":"2025-03-30T22:42:09.168Z","repository":{"id":217156841,"uuid":"743149176","full_name":"msysh/aws-sample-scheduled-cloudwatch-logs-export","owner":"msysh","description":"This solution exports CloudWatch Logs of last day for daily.","archived":false,"fork":false,"pushed_at":"2025-02-20T05:22:48.000Z","size":97,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-20T06:21:55.600Z","etag":null,"topics":["cloudwatch-logs","eventbridge","export","step-functions"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/msysh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-14T13:39:00.000Z","updated_at":"2025-02-20T05:04:04.000Z","dependencies_parsed_at":null,"dependency_job_id":"5a9398f5-e0ae-40f6-b893-d081035f9c0a","html_url":"https://github.com/msysh/aws-sample-scheduled-cloudwatch-logs-export","commit_stats":null,"previous_names":["msysh/aws-sample-scheduled-cloudwatch-logs-export"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msysh%2Faws-sample-scheduled-cloudwatch-logs-export","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msysh%2Faws-sample-scheduled-cloudwatch-logs-export/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msysh%2Faws-sample-scheduled-cloudwatch-logs-export/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msysh%2Faws-sample-scheduled-cloudwatch-logs-export/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/msysh","download_url":"https://codeload.github.com/msysh/aws-sample-scheduled-cloudwatch-logs-export/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246390858,"owners_count":20769476,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloudwatch-logs","eventbridge","export","step-functions"],"created_at":"2024-12-12T14:07:45.435Z","updated_at":"2025-03-30T22:42:09.163Z","avatar_url":"https://github.com/msysh.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Scheduled export for Amazon CloudWatch Logs\n\nThis is a sample solution with AWS CDK. It exports CloudWatch Logs of last day for daily.\n\nThe logs exported by CloudWatch Logs are stored in folders separated by task ID on S3, which can make searching the logs later from Athena more complex. This sample solution moves the exported logs into `yyyy/MM/dd` folders instead. Each log file is renamed to `\u003cLogStream Name\u003e-\u003cSequential Number\u003e.gz`. See the \"[Log files layout](#log-files-layout)\" for more details.\n\n## Architecture\n\n![Step Functions state machine flow](./doc/images/architecture.svg)\n\n## Important\n\n\u003e [!IMPORTANT]\n\u003e If you want to export CloudWatch Logs to S3 and search them with Athena, using a Subscription Filter and Firehose may provide better search performance in some cases. If the export task outputs many small files, searching with Athena can become inefficient and perform poorly. To mitigate this, if you will use AWS Glue to compact the many small files into one large file, it may be better to use Firehose from the start.\n\n\u003e [!NOTE]\n\u003e CloudWatch Logs charges are based on two methods: Ingestion and Storage. In fact, the Ingestion cost is dominant. While it's true that storing logs in S3 is cheaper, keeping them in CloudWatch Logs does not make a significant difference. If your requirement is simply searching with CloudWatch Logs Insights, [setting the log class to the Infrequent Access](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatch_Logs_Log_Classes.html) may be the best option.\n\n## Pre-requirement\n\nThis project are provisioned by the AWS Cloud Development Kit (CDK). If you have not installed the CDK, first install it by referring to the [documents](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html).\n\n## How to deploy\n\n### 1. Specify LogGroup\n\n[Specify an exporting LogGroup in CDK code](./lib/stack.ts#L17)\n\n```typescript\nconst targetLogGroupName = '\u003cPlease specify an exporting LogGroup\u003e';\n```\n\n### 2. Specify execution timing\n\n[Specify an execution timing in CDK code](./lib/stack.ts#L293-#L300)\n\n```typescript\nschedule: events.Schedule.cron({\n  minute: '0',\n  hour: '1',\n  day: '*',\n  month: '*',\n  // weekDay: '?',\n  year: '*',\n}),\n```\n\nIf you want specify by `rate`, following:\n\n```typescript\nschedule: events.Schedule.rate(\n  cdk.Duration.hours(24)\n),\n```\n\n#### (Optional)\n\nSince the parameters of `CreateExportTask`, `From` and `To`, are specified at the [Prepare state with JSONata](./lib/stack.ts#L89-L100), if necessary, please customize the it.\n\n### 3. Deploy AWS resources\n\n```sh\ncdk deploy\n```\n\nIf you have never run `cdk` command, firstly you may need to run `cdk bootstrap`.\n\n## Log files layout\n\nExported files by `CreateExportTask` are stored in following:\n\n```\nbucket-name\n  + (prefix: if specified)\n    + \u003cExport Task ID\u003e\n      + \u003cLog Stream Name\u003e\n        + 000000.gz (Sequential number)\n            :\n            :\n```\n\nThis sample solution moves log files into following:\n\n```\nbucket-name\n  + (Destination Prefix: if specified, default is \"exported-logs\")\n    + yyyy\n      + MM\n        + dd\n          + \u003cLog Stream Name\u003e-000000.gz\n                     :\n                     :\n```\n\n\u003e [!Tip]\n\u003e `yyyy/MM/dd` is identified at the [Prepare state with JSONata](./lib/stack.ts#L89-L100) as destination date prefix. If you want to change it, you can customize it.\n\n\u003e [!Tip]\n\u003e Task results for moving files are stored at `\u003cDestination Bucket\u003e/result-write-logs-for-moving-files/\u003cTaskID\u003e/*`. If you want to change the prefix, you can modify at [here](./lib/stack.ts#L20).\n\n## Clean up\n\nIf you want to remove this solution, please execute following command:\n\n```sh\ncdk destroy\n```\n\nThe S3 Bucket for log files destination is remaining. If you want to delete the bucket, manual deletion using management console.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmsysh%2Faws-sample-scheduled-cloudwatch-logs-export","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmsysh%2Faws-sample-scheduled-cloudwatch-logs-export","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmsysh%2Faws-sample-scheduled-cloudwatch-logs-export/lists"}