{"id":13503157,"url":"https://github.com/nickbabcock/rrinlog","last_synced_at":"2025-07-20T15:33:51.127Z","repository":{"id":48359289,"uuid":"111625374","full_name":"nickbabcock/rrinlog","owner":"nickbabcock","description":"Replacing Elasticsearch with Rust and SQLite","archived":false,"fork":false,"pushed_at":"2021-07-30T05:15:24.000Z","size":886,"stargazers_count":186,"open_issues_count":11,"forks_count":14,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-06T17:37:28.492Z","etag":null,"topics":["actix-web","diesel","elasticsearch","grafana","nginx","rocket","sqlite"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nickbabcock.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-11-22T02:17:31.000Z","updated_at":"2025-02-17T03:25:28.000Z","dependencies_parsed_at":"2022-08-28T16:50:16.699Z","dependency_job_id":null,"html_url":"https://github.com/nickbabcock/rrinlog","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/nickbabcock/rrinlog","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickbabcock%2Frrinlog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickbabcock%2Frrinlog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickbabcock%2Frrinlog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickbabcock%2Frrinlog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nickbabcock","download_url":"https://codeload.github.com/nickbabcock/rrinlog/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickbabcock%2Frrinlog/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266151526,"owners_count":23884436,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["actix-web","diesel","elasticsearch","grafana","nginx","rocket","sqlite"],"created_at":"2024-07-31T22:02:39.438Z","updated_at":"2025-07-20T15:33:51.109Z","avatar_url":"https://github.com/nickbabcock.png","language":"Rust","funding_links":[],"categories":["Rust","sqlite"],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/nickbabcock/rrinlog.svg?branch=master)](https://travis-ci.org/nickbabcock/rrinlog)\n\n# rrinlog\n\nrrinlog is my attempt at [Replacing Elasticsearch with Rust and SQLite](https://nbsoftsolutions.com/blog/replacing-elasticsearch-with-rust-and-sqlite) for my nginx access logs, as Elasticsearch is a resource hungry application even at idle. rrinlog's success has been outstanding, with a 100x reduction in memory, 1000x reduction in CPU usage, and 100x reduction in disk usage.\n\nThis project contains two binaries:\n\n- `rrinlog` is for consuming nginx acces logs and storing them in a SQLite database. This binary may be built on Rust stable.\n- `rrinlog-server` exposes this SQLite database according to Grafana's [JSON API datasource](https://github.com/grafana/simple-json-datasource). This binary requires Rust nightly. Current Rust web framework is actix web, but there is a [Rocket branch](https://github.com/nickbabcock/rrinlog/tree/rocket) as well\n\nThis project currently isn't meant at replacing Elasticsearch for the general populous for the following reasons:\n\n### Custon Nginx Access Log\n\n`rrinlog` ingests a custom nginx access log format:\n\n```\nlog_format vhost    '$remote_addr - $remote_user [$time_local] '\n                    '\"$request\" $status $body_bytes_sent '\n                    '\"$http_referer\" \"$http_user_agent\" \"$host\"';\n```\n\nAny other format would likely result in parsing errors.\n\n### Hardcoded SQL Queries\n\n`rrinlog-server` let's me know what my top blog articles with the following SQL query:\n\n```sql\nSELECT referer,\n       Count(*) AS views\nFROM   logs\nWHERE  host = 'comments.nbsoftsolutions.com'\n       AND method = 'GET'\n       AND path \u003c\u003e '/js/embed.min.js'\n       AND epoch \u003e= ?\n       AND epoch \u003c ?\n       AND referer \u003c\u003e '-'\n       AND remote_addr \u003c\u003e ?\nGROUP  BY referer\nORDER  BY views DESC\n```\n\nThis SQL query is tailored to me and how my blog is setup, so make no mistake that the intended audience with this query is solely me :smile:\n\n### Limited Endpoints\n\nThese hardcoded SQL queries are needed as Grafana doesn't support SQLite as a native datasource. One day it may be supported like Mysql and Postgres, but until that day, `rrinlog-server` contains only a limited set of visualizations:\n\n- What are my top blog articles\n- How much outbound web data is leaving the server to other external IPs\n- How many requests are being serviced by other virtual hosts\n\n### No GeoIP Capabilities\n\nElasticsearch has the ability to take an IP address and turn it into a\nlocation. This is called\n[GeoIP](https://www.elastic.co/blog/geoip-in-the-elastic-stack). I had a\nGrafana panel showing the top visiting cities, which is novel but not critical\nto monitor. Migrating from Elasticsearch meant I had to remove the\nvisualization.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnickbabcock%2Frrinlog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnickbabcock%2Frrinlog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnickbabcock%2Frrinlog/lists"}