{"id":13393927,"url":"https://github.com/oklog/oklog","last_synced_at":"2025-03-13T19:31:52.737Z","repository":{"id":57479296,"uuid":"78433294","full_name":"oklog/oklog","owner":"oklog","description":"A distributed and coördination-free log management system","archived":true,"fork":false,"pushed_at":"2018-09-07T21:45:27.000Z","size":842,"stargazers_count":2979,"open_issues_count":43,"forks_count":167,"subscribers_count":108,"default_branch":"master","last_synced_at":"2025-03-10T05:39:13.162Z","etag":null,"topics":["distributed","distributed-systems","log","logging","metallica"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oklog.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-01-09T13:54:38.000Z","updated_at":"2025-03-04T10:36:16.000Z","dependencies_parsed_at":"2022-09-13T03:10:56.330Z","dependency_job_id":null,"html_url":"https://github.com/oklog/oklog","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oklog%2Foklog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oklog%2Foklog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oklog%2Foklog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oklog%2Foklog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oklog","download_url":"https://codeload.github.com/oklog/oklog/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243469224,"owners_count":20295715,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed","distributed-systems","log","logging","metallica"],"created_at":"2024-07-30T17:01:02.753Z","updated_at":"2025-03-13T19:31:52.174Z","avatar_url":"https://github.com/oklog.png","language":"Go","readme":"# OK Log is archived\n\nI hoped to find the opportunity to continue developing OK Log after the spike of its creation.\nUnfortunately, despite effort, no such opportunity presented itself.\nPlease look at OK Log for inspiration, and consider using the (maintained!) projects that came from it,\n[ulid](https://github.com/oklog/ulid) and [run](https://github.com/oklog/run).\n\n---\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"oklog.png\" width=500 height=142 alt=\"OK Log\"/\u003e\u003cbr/\u003e\n\u003ca href=\"https://github.com/oklog/oklog/releases/latest\"\u003e\u003cimg src=\"https://img.shields.io/github/release/oklog/oklog.svg?style=flat-square\"/\u003e\u003c/a\u003e\n\u003ca href=\"https://travis-ci.org/oklog/oklog\"\u003e\u003cimg src=\"https://img.shields.io/travis/oklog/oklog.svg?style=flat-square\"/\u003e\u003c/a\u003e\n\u003ca href=\"https://goreportcard.com/report/github.com/oklog/oklog\"\u003e\u003cimg src=\"https://goreportcard.com/badge/github.com/oklog/oklog?style=flat-square\u0026x=1\"/\u003e\u003c/a\u003e\n\u003ca href=\"LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/license-Apache%202.0-blue.svg?style=flat-square\"/\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nOK Log is a distributed and coördination-free log management system for big ol' clusters.\nIt's an on-prem solution that's designed to be a sort of building block: easy to understand, easy to operate, and easy to extend.\n\n- [Article motivating the system](https://peter.bourgon.org/ok-log)\n- [Detailed system design](DESIGN.md)\n- [Architectural Decision Records](doc/arch)\n\n## Is OK Log for me?\n\nYou may consider OK Log if...\n\n- You're tailing your logs manually, find it annoying, and want to aggregate them without a lot of fuss\n- You're using a hosted solution like Loggly, and want to move logs on-prem\n- You're using Elasticsearch, but find it unreliable, difficult to operate, or don't use many of its features\n- You're using a custom log pipeline with e.g. Fluentd or Logstash, and having performance problems\n- You just wanna, like, grep your logs — why is this all so complicated?\n\n## Getting OK Log\n\nOK Log is distributed as a single, statically-linked binary for a variety of target architectures.\nDownload the latest release from [the releases page](https://github.com/oklog/oklog/releases).\n\n## Quickstart\n\n```sh\n$ oklog ingeststore -store.segment-replication-factor 1\n$ ./myservice | oklog forward localhost\n$ oklog query -from 5m -q Hello\n2017-01-01 12:34:56 Hello world!\n```\n\n## Deploying\n\n### Small installations\n\nIf you have relatively small log volume, you can deploy a cluster of identical ingeststore nodes.\nBy default, the replication factor is 2, so you need at least 2 nodes.\nUse the -cluster flag to specify a routable IP address or hostname for each node to advertise itself on.\nAnd let each node know about at least one other node with the -peer flag.\n\n```sh\nfoo$ oklog ingeststore -cluster foo -peer foo -peer bar -peer baz\nbar$ oklog ingeststore -cluster bar -peer foo -peer bar -peer baz\nbaz$ oklog ingeststore -cluster baz -peer foo -peer bar -peer baz\n```\n\nTo grow the cluster, just add a new node, and tell it about at least one other node via the -peer flag.\nOptionally, you can run the rebalance tool (TODO) to redistribute the data over the new topology.\nTo shrink the cluster, just kill nodes fewer than the replication factor,\n and run the repair tool (TODO) to re-replicate lost records.\n\nAll configuration is done via commandline flags.\nYou can change things like the log retention period (default 7d),\n the target segment file size (default 128MB),\n and maximum time (age) of various stages of the logging pipeline.\nMost defaults should be sane, but you should always audit for your environment.\n\n### Large installations\n\nIf you have relatively large log volume, you can split the ingest and store (query) responsibilities.\nIngest nodes make lots of sequential writes, and benefit from fast disks and moderate CPU.\nStore nodes make lots of random reads and writes, and benefit from large disks and lots of memory.\nBoth ingest and store nodes join the same cluster, so provide them with the same set of peers.\n\n```sh\ningest1$ oklog ingest -cluster 10.1.0.1 -peer ...\ningest2$ oklog ingest -cluster 10.1.0.2 -peer ...\n\nstore1$ oklog store -cluster 10.1.9.1 -peer ...\nstore2$ oklog store -cluster 10.1.9.2 -peer ...\nstore3$ oklog store -cluster 10.1.9.3 -peer ...\n```\n\nTo add more raw ingest capacity, add more ingest nodes to the cluster.\nTo add more storage or query capacity, add more store nodes.\nAlso, make sure you have enough store nodes to consume from the ingest nodes without backing up.\n\n## Forwarding\n\nThe forwarder is basically just netcat with some reconnect logic.\nPipe the stdout/stderr of your service to the forwarder, configured to talk to your ingesters.\n\n```sh\n$ ./myservice | oklog forward ingest1 ingest2\n```\n\nOK Log integrates in a straightforward way with runtimes like Docker and Kubernetes.\nSee [the Integrations page](https://github.com/oklog/oklog/wiki/Integrations) for more details.\n\n## Querying\n\nQuerying is an HTTP GET to /query on any of the store nodes.\nOK Log comes with a query tool to make it easier to play with.\nOne good thing is to first use the -stats flag to refine your query.\nWhen you're satisfied it's sufficiently constrained, drop -stats to get results.\n\n```sh\n$ oklog query -from 2h -to 1h -q \"myservice.*(WARN|ERROR)\" -regex\n2016-01-01 10:34:58 [myservice] request_id 187634 -- [WARN] Get /check: HTTP 419 (0B received)\n2016-01-01 10:35:02 [myservice] request_id 288211 -- [ERROR] Post /ok: HTTP 500 (0B received)\n2016-01-01 10:35:09 [myservice] request_id 291014 -- [WARN] Get /next: HTTP 401 (0B received)\n ...\n```\n\nTo query structured logs, combine a basic grep filter expression with a tool like [jq](https://stedolan.github.io/jq/).\n\n```sh\n$ oklog query -from 1h -q /api/v1/login\n{\"remote_addr\":\"10.34.115.3:50032\",\"path\":\"/api/v1/login\",\"method\":\"POST\",\"status_code\":200}\n{\"remote_addr\":\"10.9.101.113:51442\",\"path\":\"/api/v1/login\",\"method\":\"POST\",\"status_code\":500}\n{\"remote_addr\":\"10.9.55.2:55210\",\"path\":\"/api/v1/login\",\"method\":\"POST\",\"status_code\":200}\n{\"remote_addr\":\"10.34.115.1:51610\",\"path\":\"/api/v1/login\",\"method\":\"POST\",\"status_code\":200}\n...\n\n$ oklog query -from 1h -q /api/v1/login | jq '. | select(.status_code == 500)'\n{\n\t\"remote_addr\": \"10.9.55.2:55210\",\n\t\"path\": \"/api/v1/login\",\n\t\"method\": \"POST\",\n\t\"status_code\": 500\n}\n...\n```\n\n## UI\n\nOK Log ships with a basic UI for making queries.\nYou can access it on any store or ingeststore node, on the public API port (default 7650), path `/ui`.\nSo, e.g. http://localhost:7650/ui.\n\n## Further reading\n\n### Integrations\n\n- [Docker](https://github.com/oklog/oklog/wiki/Integrations#docker)\n- [Kubernetes](https://github.com/oklog/oklog/wiki/Integrations#kubernetes)\n\n### Unofficial Docker images\n\n- [OK Log](https://hub.docker.com/r/oklog/oklog/)\n\n### Translation\n\n- [中文介绍](https://github.com/1046102779/OKLog) \n\n---\n\n[OK icon](https://thenounproject.com/term/like/594444/) by\n [Karthik Srinivas](https://thenounproject.com/aathis/) from the\n [Noun Project](https://thenounproject.com).\nDevelopment supported by [DigitalOcean](https://digitalocean.com).\n","funding_links":[],"categories":["Databases","Go","distributed-systems"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foklog%2Foklog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foklog%2Foklog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foklog%2Foklog/lists"}