{"id":26233574,"url":"https://github.com/thevxn/xilt","last_synced_at":"2026-02-23T02:08:07.518Z","repository":{"id":282056417,"uuid":"946908111","full_name":"thevxn/xilt","owner":"thevxn","description":"A utility for parsing Common and Combined Log Format (CLF) log files and storing them in SQLite for further analysis. Built with concurrency in mind.","archived":false,"fork":false,"pushed_at":"2025-03-29T11:23:47.000Z","size":113,"stargazers_count":22,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-29T12:25:12.638Z","etag":null,"topics":["go","golang","log","log-parser","log-parsers","log-parsing","parser","sqlite"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thevxn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-11T21:33:56.000Z","updated_at":"2025-03-29T11:23:51.000Z","dependencies_parsed_at":"2025-03-12T15:47:35.958Z","dependency_job_id":null,"html_url":"https://github.com/thevxn/xilt","commit_stats":null,"previous_names":["thevxn/xilt"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thevxn%2Fxilt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thevxn%2Fxilt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thevxn%2Fxilt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thevxn%2Fxilt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thevxn","download_url":"https://codeload.github.com/thevxn/xilt/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250237832,"owners_count":21397401,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","log","log-parser","log-parsers","log-parsing","parser","sqlite"],"created_at":"2025-03-13T01:16:31.086Z","updated_at":"2026-02-23T02:08:02.471Z","avatar_url":"https://github.com/thevxn.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# xilt\n\n[![Go Report Card](https://goreportcard.com/badge/go.vxn.dev/xilt)](https://goreportcard.com/report/go.vxn.dev/xilt) [![Go Reference](https://pkg.go.dev/badge/go.vxn.dev/xilt#readme-usage.svg)](https://pkg.go.dev/go.vxn.dev/xilt#readme-usage) [![Go Coverage](https://github.com/thevxn/xilt/wiki/coverage.svg)](https://raw.githack.com/wiki/thevxn/xilt/coverage.html)\n[![libs.tech recommends](https://libs.tech/project/946908111/badge.svg)](https://libs.tech/project/946908111/xilt)\n\nA utility for parsing Common and Combined Log Format (CLF) log files and storing them in SQLite for further analysis. Built with concurrency in mind.\n\n## Usage\n\n### Install\n\n```sh\ngo install go.vxn.dev/xilt/cmd/xilt@latest\n```\n\n### Run\n\n```sh\nxilt [logFilePath] [dbFilePath]\n```\n\n- Default `logFilePath` = `./access.log`\n- Default `dbFilePath` = `./logs.db`\n\n### Flags\n\n```text\n$ xilt -h\nUsage of xilt:\n  -avgLogSize float\n        Defines the average size of one log in MB. Used for calculating the number of goroutines to spin up. (default 0.001)\n  -batchSize int\n        Defines the batch size. Used for calculating the number of goroutines to spin up. (default 5000)\n  -i    Defines whether indexes should be created in the parsed logs' table.\n  -maxMemUsage int\n        Defines the maximum allowed memory usage in Megabytes. Used for calculating the number of goroutines to spin up. (default 100)\n  -v    Defines whether verbose mode should be used.\n```\n\n### Indexes\n\nCurrently, if the `-i` flag is used, the following indexes are created:\n\n- IP\n- Method\n- Referer\n- Route\n- TimestampUTC\n- TimestampUTC + IP\n\nThis may be subject to change in the future depending on further fine-tuning and configurable parameters extension.\n\n## Example\n\n`logs.txt`:\n\n```text\n192.168.1.101 user-identifier alice [12/Nov/2023:14:23:45 -0400] \"POST /index.html?query=search HTTP/1.1\" 404 1234 \"https://example.com\" \"Mozilla/5.0\"\n10.0.0.15 user-identifier bob [15/Dec/2023:09:12:33 -0500] \"GET /about.html HTTP/1.0\" 200 5678 \"https://google.com\" \"Chrome/120.0\"\n172.16.254.1 user-identifier charlie [20/Jan/2024:18:45:22 -0700] \"PUT /upload/file.txt HTTP/1.1\" 201 9876 \"https://dropbox.com\" \"PostmanRuntime/7.26\"\n203.0.113.45 user-identifier dave [05/Feb/2024:22:11:09 +0000] \"DELETE /resource/123 HTTP/1.1\" 204 0 \"https://api.example.com\" \"curl/7.68\"\n198.51.100.22 user-identifier eve [10/Mar/2024:07:34:56 -0800] \"PATCH /update/profile HTTP/1.1\" 400 2345 \"https://app.example.com\" \"Python-requests/2.25\"\n192.0.2.33 user-identifier frank [25/Apr/2024:12:55:12 +0200] \"HEAD /status HTTP/1.0\" 302 0 \"https://redirect.com\" \"Safari/15.0\"\n203.0.113.78 user-identifier grace [30/May/2024:16:44:33 -0600] \"OPTIONS /options HTTP/1.1\" 200 3456 \"https://options.example.com\" \"Edge/120.0\"\n198.51.100.99 user-identifier heidi [15/Jun/2024:03:22:11 -0300] \"GET /download/file.zip HTTP/1.1\" 500 6789 \"https://download.com\" \"Wget/1.21\"\n192.168.0.55 user-identifier ivan [20/Jul/2024:10:11:12 -0400] \"POST /submit/form HTTP/1.1\" 201 4321 \"https://form.example.com\" \"Firefox/120.0\"\n10.0.1.100 user-identifier jack [25/Aug/2024:19:33:44 -0700] \"GET /images/logo.png HTTP/1.0\" 304 0 \"https://cache.example.com\" \"Opera/90.0\"\n```\n\nRun `xilt -i logs.txt logs.db`:\n\n```text\n$ xilt -i logs.txt logs.db\n2025/03/11 22:38:44 beginning reading from log file and the parsing process...\n2025/03/11 22:38:44 creating table indexes...\n2025/03/11 22:38:44 table indexes created...\n2025/03/11 22:38:44 log parsing finished\n2025/03/11 22:38:44 elapsed time: 624.7111ms\n```\n\nThe resulting `logs.db` file:\n![log.db](docs/log.db.png)\n\n## Benchmark\n\nCurrently, a basic benchmark is available using the following hardware:\n\n- AMD Ryzen™ 7 5800X3D\n- 32 GB DDR4 RAM\n\nA large log file containing 17 379 453 logs (~1.63 GB in size) takes 1 minute, 55 seconds on average to be parsed and stored in SQLite if not using indexes. Index creation on a table of such size prolongs the duration to ~4 minutes.\n\nAll logs in the file are the same: `127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326`.\n\nWithout indexes:\n\n```text\n$ xilt logs benchmark.db\n2025/03/11 22:47:18 beginning reading from log file and the parsing process...\n2025/03/11 22:49:12 log parsing finished\n2025/03/11 22:49:12 elapsed time: 1m54.7359703s\n```\n\nWith indexes:\n\n```text\n$ xilt -i logs benchmark.db\n2025/03/11 22:54:47 beginning reading from log file and the parsing process...\n2025/03/11 22:56:38 creating table indexes...\n2025/03/11 22:58:46 table indexes created...\n2025/03/11 22:58:46 log parsing finished\n2025/03/11 22:58:46 elapsed time: 3m59.774765s\n```\n\n### Memory Usage\n\nUsing the default presets (`batchSize=5000`, `avgLogSize=0.001`, `maxMemUsage=100`) to parse the aforementioned benchmark file leads to an average RAM usage of ~85 MB.\n\n## Build \u0026 Run Locally\n\n```sh\ngit clone https://github.com/thevxn/xilt\ngo build ./cmd/xilt\n./xilt\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthevxn%2Fxilt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthevxn%2Fxilt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthevxn%2Fxilt/lists"}