{"id":18293233,"url":"https://github.com/maki-daisuke/cflogparser","last_synced_at":"2025-04-09T08:15:15.843Z","repository":{"id":136977048,"uuid":"213424832","full_name":"Maki-Daisuke/cflogparser","owner":"Maki-Daisuke","description":"Parser for Amazon Web Services CloudFront Logs for Go","archived":false,"fork":false,"pushed_at":"2019-10-08T18:52:19.000Z","size":17,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-02T00:52:09.773Z","etag":null,"topics":["amazon-web-services","cloudfront","golang","log","parser"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Maki-Daisuke.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-07T15:51:01.000Z","updated_at":"2024-08-02T23:03:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"66f76fd1-bdc2-4789-b58f-ca06e9876d82","html_url":"https://github.com/Maki-Daisuke/cflogparser","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Maki-Daisuke%2Fcflogparser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Maki-Daisuke%2Fcflogparser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Maki-Daisuke%2Fcflogparser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Maki-Daisuke%2Fcflogparser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Maki-Daisuke","download_url":"https://codeload.github.com/Maki-Daisuke/cflogparser/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247999864,"owners_count":21031046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-web-services","cloudfront","golang","log","parser"],"created_at":"2024-11-05T14:23:21.177Z","updated_at":"2025-04-09T08:15:15.816Z","avatar_url":"https://github.com/Maki-Daisuke.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"cflogparser\n===========\n\nDescription\n-----------\n\nA parser for Amazon Web Services (AWS) CloudFront log implemented in Go\n\n\nExample\n-------\n\n```golang\npackage main\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\t\"strings\"\n\n\t\"github.com/Maki-Daisuke/cflogparser\"\n\t\"github.com/mattn/go-forlines\"\n)\n\nfunc main() {\n\tcnt := map[string]int{} // Count URI\n\n\t// Read from Stdin, parse each line and count accesses to each URI\n\tforlines.Do(os.Stdin, func(line string) error {\n\t\tif strings.HasPrefix(line, \"#\") {\n\t\t\t// Ignore leading comment lines for meta-information\n\t\t\treturn nil\n\t\t}\n\t\tlog, err := cflogparser.ParseLineWeb(line)  // ParseLineWeb returns *WebLog\n\t\tif err != nil {\n\t\t\tfmt.Fprintln(os.Stderr, err)\n\t\t\treturn nil\n\t\t}\n\t\tcnt[log.URI]++\n\t\treturn nil\n\t})\n\n\tfor k, v := range cnt {\n\t\tfmt.Printf(\"%d\\t%s\\n\", v, k)\n\t}\n}\n```\n\n\nSupported Formtats\n------------------\n\n- CloudFront Access log\n  - Web Distribution Log File Format: Version 1.0\n  - RTMP Distribution Log File Format: Version 1.0\n\n\nAPI\n---\n\nSee [Godoc](https://godoc.org/github.com/Maki-Daisuke/cflogparser).\n\n\nRationale\n--------\n\nCloudFront log format is very simple and looks easy to parse. It is essentially \nURL-encoded values delimitted by `'\\t'` (tab) character. But, there are putfalls.\n\nFor example, it escape some characters twice, such as escaping `' '` (space) to `\"%2520\"`. \n(See [this AWS Discussion Forums post](https://forums.aws.amazon.com/thread.jspa?threadID=134017) \nfor the historical background.)\n\nAnother example is that it records `\"-\"` when there is no value to record even though that \nfield expect an integer value.\n\nWe needed something that strictly complies with \n[the specification](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html).\n\n\nLicense\n-------\n\nMIT License\n\n\nAuthor\n------\n\nDaisuke (yet another) Maki\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaki-daisuke%2Fcflogparser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaki-daisuke%2Fcflogparser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaki-daisuke%2Fcflogparser/lists"}