{"id":27118408,"url":"https://github.com/suconghou/log2sqlite","last_synced_at":"2026-02-17T02:33:21.741Z","repository":{"id":70470946,"uuid":"601880353","full_name":"suconghou/log2sqlite","owner":"suconghou","description":"nginx access log to sqlite","archived":false,"fork":false,"pushed_at":"2025-03-17T09:18:29.000Z","size":118,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-24T22:42:04.058Z","etag":null,"topics":["log-analysis","logparser","nginx-log-analysis","nginx-logs","sqlite3"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/suconghou.png","metadata":{"files":{"readme":"README.MD","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-02-15T02:41:33.000Z","updated_at":"2025-03-23T10:18:48.000Z","dependencies_parsed_at":"2023-03-02T00:31:22.216Z","dependency_job_id":"1edffb2d-5d96-4a3d-9e24-26ed97096d2b","html_url":"https://github.com/suconghou/log2sqlite","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/suconghou/log2sqlite","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suconghou%2Flog2sqlite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suconghou%2Flog2sqlite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suconghou%2Flog2sqlite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suconghou%2Flog2sqlite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/suconghou","download_url":"https://codeload.github.com/suconghou/log2sqlite/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suconghou%2Flog2sqlite/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29531020,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-17T00:57:22.232Z","status":"online","status_checked_at":"2026-02-17T02:00:08.105Z","response_time":100,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["log-analysis","logparser","nginx-log-analysis","nginx-logs","sqlite3"],"created_at":"2025-04-07T07:57:45.187Z","updated_at":"2026-02-17T02:33:21.724Z","avatar_url":"https://github.com/suconghou.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\nparse nginx access log into sqlite database\n\n解析的日志格式\n```\n    log_format vhosts '$remote_addr - $remote_user [$time_local] \"$request\" '\n                      '$status $body_bytes_sent \"$http_referer\" '\n                      '\"$http_user_agent\" \"$http_x_forwarded_for\" '\n                      '$host $request_length $bytes_sent $upstream_addr '\n                      '$upstream_status $request_time $upstream_response_time '\n                      '$upstream_connect_time $upstream_header_time';\n\n\n```\n\n依赖 sqlite3 , alpine 内静态编译可 `apk add sqlite-dev sqlite-static`\n\n以下三个版本均在`nimlang/nim:latest-alpine-slim`中编译\n\n## nim 版本\n\n```\nnim --mm:arc -d:release --dynlibOverride:libsqlite3 --passC:-flto --passL:-flto --passL:-s --passL:-static --passL:-lsqlite3 --opt:speed c main\n```\n\n\n## c++ 版本\n\n\n```\ng++ -Wall -std=c++20 -O3 -lsqlite3 main.cpp\n\ng++ -Wall -std=c++20 -flto=auto -static-libstdc++ -static-libgcc --static -Wl,-Bstatic,--gc-sections -O3 -ffunction-sections -fdata-sections main.cpp -lsqlite3 -o log2sqlite\n```\n\n## c 版本\n\n\nmac\n```\ngcc -Wall -std=c17 -O3 -lsqlite3 main.c\n```\n\nlinux 静态编译，因strptime函数的申明问题(https://stackoverflow.com/questions/43460876/trouble-including-function-declaration-for-strptime)\n\n我们需要使用`gnu17`标准\n```\n\ngcc -Wall -std=gnu17 -flto=auto -static-libstdc++ -static-libgcc --static -Wl,-Bstatic,--gc-sections -O3 -ffunction-sections -fdata-sections main.c -lsqlite3 -o log2sqlite\n\n```\n\n注意： `-lsqlite3` 必须在`main.c`之后\n\n\n每秒约处理20万条日志，c++版本比c版本慢约10%,nim版本和c++版本速度相当\n\n\n测试数据700MB+, 约200万行\n\n| 版本  | 时间  |\n| ------ | ------    |\n| c     | 7.395s     |\n| c++   | 10.791s    |\n| nim   | 10.024s    |\n\n可使用sqlite3命令对数据进行分析\n\n```\nsqlite3 nginx_log.db 'select count(1) as n,request from nginx_log group by request order by n desc limit 5;'\n\n\n或者sql查询存储到sql文件\n\nsqlite3 nginx_log.db \u003c /tmp/1.sql \n\n\nsqlite3 nginx_log.db 'select count(1) as n,request,http_user_agent from nginx_log group by http_user_agent,request order by n desc limit 5;'\n\nsqlite3 nginx_log.db 'select count(1) as n,request,remote_addr from nginx_log group by remote_addr,request order by n desc limit 5;'\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuconghou%2Flog2sqlite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsuconghou%2Flog2sqlite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuconghou%2Flog2sqlite/lists"}