{"id":19078658,"url":"https://github.com/nidorx/sqlog","last_synced_at":"2025-09-07T18:32:10.802Z","repository":{"id":257806734,"uuid":"864736260","full_name":"nidorx/sqlog","owner":"nidorx","description":"SQLog - Connecting the dots","archived":false,"fork":false,"pushed_at":"2024-11-25T23:09:24.000Z","size":901,"stargazers_count":29,"open_issues_count":5,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-08T07:35:59.632Z","etag":null,"topics":["bloom-filter","golang","hyperloglog","logging","lsm-tree","observability","sqlite"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nidorx.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-29T02:54:37.000Z","updated_at":"2025-02-01T19:02:45.000Z","dependencies_parsed_at":"2024-10-02T07:42:07.715Z","dependency_job_id":"69bbf731-4afd-411a-ab00-ad2b48926305","html_url":"https://github.com/nidorx/sqlog","commit_stats":null,"previous_names":["nidorx/sqlog"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/nidorx/sqlog","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nidorx%2Fsqlog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nidorx%2Fsqlog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nidorx%2Fsqlog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nidorx%2Fsqlog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nidorx","download_url":"https://codeload.github.com/nidorx/sqlog/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nidorx%2Fsqlog/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274078589,"owners_count":25218705,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-07T02:00:09.463Z","response_time":67,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bloom-filter","golang","hyperloglog","logging","lsm-tree","observability","sqlite"],"created_at":"2024-11-09T02:11:22.618Z","updated_at":"2025-09-07T18:32:10.385Z","avatar_url":"https://github.com/nidorx.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cbr\u003e\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"./docs/logo.png\" /\u003e\n    \u003cp align=\"center\"\u003e\n        \u003cstrong\u003eSQLog\u003c/strong\u003e - Connecting the dots\n    \u003c/p\u003e\n\u003c/div\u003e\n\n\u003c!-- PROJECT SHIELDS --\u003e\n[![Go][actions-shield]][actions-url]\n[![Go Report Card][goreport-shield]][goreport-url]\n[![MIT License][license-shield]][license-url]\n\n[actions-shield]: https://github.com/nidorx/sqlog/actions/workflows/go.yml/badge.svg\n[actions-url]: https://github.com/nidorx/sqlog/actions/workflows/go.yml\n[goreport-shield]: https://goreportcard.com/badge/github.com/nidorx/sqlog\n[goreport-url]: https://goreportcard.com/report/github.com/nidorx/sqlog\n[license-shield]: https://img.shields.io/github/license/nidorx/sqlog.svg?style=flat\n[license-url]: https://github.com/nidorx/sqlog/blob/main/LICENSE.txt\n\n\n**SQLog** is a **Golang** library that simplifies log management using **slog**. **SQLog** offers a lightweight and reliable solution for logging, making it perfect for developers seeking a cost-effective, high-performance way to monitor their applications.\n\n## Usage\n\nBelow is an example of using **SQLog** with the SQLite storage, with the interface exposed on port `8080`. When you access `http://localhost:8080?msg=test`, any query parameter will be sent to the log.\n\nYou can view the generated logs at `http://localhost:8080/logs/`.\n\n```go\nimport (\n\t\"log/slog\"\n\t\"net/http\"\n\t\"os\"\n\t\"strings\"\n\n\t\"github.com/nidorx/sqlog\"\n\n    // sqlite storage\n\t\"github.com/nidorx/sqlog/sqlite\"\n\n    // sqlite driver\n    _ \"github.com/mattn/go-sqlite3\"\n)\n\nfunc main() {\n\tstorage, _ := sqlite.New(nil)\n\tlogger, _ := sqlog.New(\u0026sqlog.Config{\n\t\tStorage: storage,\n\t})\n\n\t// magic\n\tslog.SetDefault(slog.New(logger.Handler()))\n\n\tif true {\n\t\t// In the local environment, you can send the log to standard output.\n\t\tlogger.Fanout(slog.NewTextHandler(os.Stdout, nil))\n\t}\n\n\t// SQLog ui/api handler\n\tlogHttpHandler := logger.HttpHandler()\n\n\t// http handler (... nidorx/chain, gin-gonic/gin, gorilla/mux, julienschmidt/httprouter)\n\thttpHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\t\tif idx := strings.LastIndex(r.URL.Path, \"/logs\"); idx \u003e= 0 {\n\t\t\tlogHttpHandler.ServeHTTP(w, r)\n\t\t} else {\n\t\t\targs := []any{}\n\t\t\tmsg := \"I ❤️ SQLog\"\n\t\t\tfor k, v := range r.URL.Query() {\n\t\t\t\tif k == \"msg\" {\n\t\t\t\t\tmsg = strings.Join(v, \",\")\n\t\t\t\t} else {\n\t\t\t\t\targs = append(args, slog.Any(k, strings.Join(v, \",\")))\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tslog.Log(r.Context(), slog.LevelInfo, msg, args...)\n\t\t\tw.Write([]byte(\"🫶\"))\n\t\t}\n\t})\n\n\thttp.ListenAndServe(\":8080\", httpHandler)\n}\n```\n\n## Demo\n\nYou can view the current version of the SQLog demo at the links below:\n\nThe demo project captures all mouse movements and sends them to the server, which logs the received parameters.\n\n- **Log generator**: https://sqlog-demo.onrender.com/\n- **SQLog UI** : https://sqlog-demo.onrender.com/logs/\n\n    \u003e **IMPORTANT**! I am using the [free version of Render](https://docs.render.com/free), so there is no guarantee of service stability or availability.\n\n[SQLog-demo.webm](https://github.com/user-attachments/assets/046b65c9-dd36-4779-8b15-915be5f7e3f3)\n\nThe source code is in the [demo directory](./demo/).\n\n\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"./docs/ui.png\" /\u003e\n\u003c/div\u003e\n\n\n## How SQLog works?\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"./docs/diagram.png\" /\u003e\n\u003c/div\u003e\n\n**SQLog** is an efficient library for capturing and managing logs, designed with a focus on performance and low latency. The architecture of the solution consists of four main components: **Handler**, **Ingester**, **Chunk**, and **Storage**. Each of these components plays a crucial role in the ingestion, storage, and retrieval of logs.\n\n1. **Handler**\n\n    The **Handler** is an implementation of `slog.Handler`, allowing **SQLog** to be easily integrated as the standard logger in Go. It transforms logs coming from `slog` into JSON format using the default encoder `slog.NewJSONHandler` or a custom encoder, providing flexibility in log formatting.\n\n2. **Ingester**\n\n    The **Ingester** manages the ingestion of logs from the **Handler** in a non-blocking manner. It maintains an in-memory buffer that stores log entries in blocks called **Chunks**. When a **Chunk** reaches its maximum capacity, the Ingester starts using the next **Chunk**, marking the current one for writing to **Storage**. At regular intervals, the Ingester invokes the `Flush` method of the **Storage** to persist filled **Chunks**.\n\n    \u003e **Performance**: One notable feature of the Ingester is its non-blocking implementation, which avoids using mutexes to ensure concurrency. Instead, it utilizes atomic operations from Go (`sync/atomic`), allowing multiple goroutines to write logs simultaneously without waiting on each other, resulting in superior performance and reduced latency.\n\n3. **Chunk**\n\n    The **Chunk** is a non-blocking structure that allows continuous writing of log entries. Each **Chunk** is a node in a linked list that has a reference to the next **Chunk**. The Ingester maintains references to two nodes: the current **Chunk**, which is still accepting new entries, and the flush **Chunk**, which contains completed entries that have not yet been persisted.\n\n    \u003e **Data Structure**: This linked list approach not only simplifies the management of **Chunks** but also facilitates fast, non-blocking writing. By updating the internal references during the flush process, the Ingester ensures that logs are written efficiently with minimal performance impact.\n\n4. **Storage**\n\n    The **Storage** is responsible for the persistence of logs. **SQLog**'s modular architecture allows for the implementation of different types of storage, such as disk files, databases, and external systems. Currently, there is a [native implementation for SQLite (see more details)](./sqlite) and another in development for in-memory persistence.\n\n    **Responsibilities**:\n    - **Persistence**: The Storage receives and stores **Chunks** from the Ingester.\n    - **Search**: It also allows for the retrieval of records, which is used by the **SQLog** API for log visualization.\n\nThe combination of these layers makes **SQLog** a robust and efficient solution for log management, optimizing performance through a non-blocking architecture and the use of atomic operations. This results in fast, real-time log capture capable of handling high workloads without compromising efficiency.\n\n## Requirements\n\nTo use the builtin SQLite Storage implementation, you need to register the driver of your choice.\n\nExamples:\n\n- `github.com/mattn/go-sqlite3`\n- `modernc.org/sqlite`\n\n\n## @TODO/IDEAS/ROADMAP\n\nIf you would like to contribute to this project, here are some tasks and ideas listed below. Feel free to suggest and implement new features in **SQLog**.\n\nIf you decide to work on a task, please leave a comment on the Issue so that others can collaborate.\n\n- **[InMemory Storage](https://github.com/nidorx/sqlog/issues/4)**\n- **[Alerts](https://github.com/nidorx/sqlog/issues/2)** -  Enable the creation of alerts within SQLog. The solution should leverage the syntax of the language to evaluate logs at regular intervals and trigger alerts when specific conditions are met.\n- [Metrics (Count, AVG, Dashboards)](https://github.com/nidorx/sqlog/issues/3)\n- NOT, REGEX (https://www.sqlite.org/lang_expr.html)\n\n\nAll kinds of contributions are welcome!\n\n🐛 **Found a bug?**  \nLet me know by [creating an issue][new-issue].\n\n## References\n\n- This project was inspired by [Blacklite](https://github.com/tersesystems/blacklite)\n- https://github.com/IGLOU-EU/go-wildcard\n- The interface is inspired by [Datadog](https://www.datadoghq.com/)\n\n\n[bugs]: https://github.com/nidorx/sqlog/issues?q=is%3Aissue+is%3Aopen+label%3Abug\n[features]: https://github.com/nidorx/sqlog/issues?q=is%3Aissue+is%3Aopen+label%3Afeature\n[new-issue]: https://github.com/nidorx/sqlog/issues/new/choose\n[discussions]: https://github.com/go-path/di/discussions\n\n## License\n\nThis code is distributed under the terms and conditions of the [MIT license](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnidorx%2Fsqlog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnidorx%2Fsqlog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnidorx%2Fsqlog/lists"}