{"id":22526771,"url":"https://github.com/yinheli/ucloud-cdn-log-parser","last_synced_at":"2025-08-03T22:31:14.996Z","repository":{"id":228048360,"uuid":"773023623","full_name":"yinheli/ucloud-cdn-log-parser","owner":"yinheli","description":"ucloud cdn log parser","archived":false,"fork":false,"pushed_at":"2024-11-01T15:29:58.000Z","size":40,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-10T09:06:45.922Z","etag":null,"topics":["log-parser","ucloud"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yinheli.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-16T14:32:10.000Z","updated_at":"2024-09-08T02:43:43.000Z","dependencies_parsed_at":"2024-09-08T03:51:00.006Z","dependency_job_id":null,"html_url":"https://github.com/yinheli/ucloud-cdn-log-parser","commit_stats":null,"previous_names":["yinheli/ucloud-cdn-log-parser"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yinheli%2Fucloud-cdn-log-parser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yinheli%2Fucloud-cdn-log-parser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yinheli%2Fucloud-cdn-log-parser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yinheli%2Fucloud-cdn-log-parser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yinheli","download_url":"https://codeload.github.com/yinheli/ucloud-cdn-log-parser/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228567101,"owners_count":17938012,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["log-parser","ucloud"],"created_at":"2024-12-07T06:18:30.051Z","updated_at":"2024-12-07T06:18:30.782Z","avatar_url":"https://github.com/yinheli.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ucloud-cdn-log-parser\n\nParse ucloud cdn log to csv/tsv format with/without header, then you can use duckdb / clickhouse local / etc to analyze the log.\n\n## Install\n\nDownload from release page, which is built by github action.\n\nor install via cargo or build from source.\n\n```bash\ncargo install ucloud-cdn-log-parser --locked\n```\n\n## Usage\n\n```bash\n# download logs\n\n# parse \u0026 convert to csv / parquet\n\nzcat *.gz | \\\n  ucloud-cdn-log-parser \u003e log.csv\n\n## use duckdb\nzcat *.gz | \\\n  ucloud-cdn-log-parser | \\\n  pv | \\\n  duckdb -c \"\n    copy \n      (select * from read_csv('/dev/stdin')) \n      to 'log.parquet.zst' \n      (format parquet, compression 'zstd');\"\n\n# now you can use duckdb / clickhouse local / etc to query the log\n```\n\n\nAnalyze log with duckdb for example:\n\n```sql\n-- get top 100 client_ip by sent_bytes_incl_header in last 6 hours\nselect\n    client_ip,\n    format_bytes(sum(sent_bytes_incl_header)::bigint) sent_bytes,\n    count(*) as n\nfrom 'log.parquet.zst'\nwhere date_time \u003e (now() - interval 6 hours)::timestamp\ngroup by client_ip\norder by sum(sent_bytes_incl_header) desc\nlimit 100;\n\n-- get top 100 request_method_url_protocol by sent_bytes_incl_header in last 6 hours\nselect\n    request_method_url_protocol,\n    format_bytes(sum(sent_bytes_incl_header)::bigint) sent_bytes,\n    count(*) as n\nfrom 'log.parquet.zst'\nwhere date_time \u003e (now() - interval 6 hours)::timestamp\ngroup by request_method_url_protocol\norder by sum(sent_bytes_incl_header) desc, n desc\nlimit 100;\n```\n\n## Reference\n\n- ucloud log format https://docs.ucloud.cn/ucdn/guide/LOG\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyinheli%2Fucloud-cdn-log-parser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyinheli%2Fucloud-cdn-log-parser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyinheli%2Fucloud-cdn-log-parser/lists"}