{"id":17526490,"url":"https://github.com/customerio/esdb","last_synced_at":"2025-04-09T10:07:10.332Z","repository":{"id":18067451,"uuid":"21127930","full_name":"customerio/esdb","owner":"customerio","description":"Event-stream flat file database - Immutable storage for timestamped event streams","archived":false,"fork":false,"pushed_at":"2024-11-14T13:50:00.000Z","size":13415,"stargazers_count":70,"open_issues_count":3,"forks_count":7,"subscribers_count":41,"default_branch":"main","last_synced_at":"2025-04-02T20:50:32.301Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/customerio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-06-23T13:50:22.000Z","updated_at":"2025-01-14T06:17:27.000Z","dependencies_parsed_at":"2024-06-20T15:43:29.115Z","dependency_job_id":"6a3022f9-b1f4-408c-a311-726aefa3713d","html_url":"https://github.com/customerio/esdb","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/customerio%2Fesdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/customerio%2Fesdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/customerio%2Fesdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/customerio%2Fesdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/customerio","download_url":"https://codeload.github.com/customerio/esdb/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248018060,"owners_count":21034048,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-20T15:01:49.675Z","updated_at":"2025-04-09T10:07:10.309Z","avatar_url":"https://github.com/customerio.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# Event Stream Database\n\nImmutable storage for timestamped event streams. Inspired by\n[CDB](http://cr.yp.to/cdb.html) and\n[LevelDB](http://en.wikipedia.org/wiki/LevelDB)'s SSTable file format.\n\nAt [Customer.io](http://customer.io), we process billions of events every\nmonth, and have a need for maintaining all historic events in full resolution\nin a simple manner (not feasible to simply distill into timeseries data).\n\nAfter investigating strategies for maintaining these events, we enjoyed the\nsimple approach of archiving old events into structured flat files which\nprovides dead simple backups, restores, and needs no running process to work.\nAfter investigating current strategies and getting down and dirty with CDB and\nLevelDB, we choose to create ESDB as a stategy dedicated to querying event\nstream data.\n\n*WARNING: version 0.0.00000001. I wrote this in one day as a proof of concept.\nWe're planning on using it in production but there's a lot of testing,\nfine-tuning, and potential large changes to do before then.*\n\n### Example\n\nLet's assume you're tracking website activity.\n\nYou would like to store all pageviews, clicks, and purchases for every user of\nyour site.  You'd like to quickly scan through the entire history for\nprocessing. However, you'd also like to occasionally just retrieve purchase\nevents without the overhead of scanning through all the pageview and click\ndata.\n\n```\npackage main\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"os\"\n\n  \"github.com/customerio/esdb\"\n)\n\ntype event struct {\n\tcustomerId string\n\ttimestamp  int\n\teventType  string\n\tdata       map[string]string\n}\n\nfunc main() {\n\tevents := []event{\n\t\tevent{\"1\", 1403534919, \"page\", map[string]string{\"url\": \"http://mysite.com/\"}},\n\t\tevent{\"1\", 1403534920, \"click\", map[string]string{\"button_text\": \"Checkout\"}},\n\t\tevent{\"1\", 1403534921, \"page\", map[string]string{\"url\": \"http://mysite.com/checkout\"}},\n\t\tevent{\"1\", 1403534923, \"purchase\", map[string]string{\"total\": \"42.99\"}},\n\t\tevent{\"1\", 1403534923, \"page\", map[string]string{\"url\": \"http://mysite.com/thankyou\"}},\n\t\tevent{\"2\", 1403534919, \"page\", map[string]string{\"url\": \"http://mysite.com/\"}},\n\t\tevent{\"2\", 1403534920, \"click\", map[string]string{\"button_text\": \"About\"}},\n\t\tevent{\"2\", 1403534921, \"page\", map[string]string{\"url\": \"http://mysite.com/about\"}},\n\t\tevent{\"3\", 1403534919, \"page\", map[string]string{\"url\": \"http://mysite.com/\"}},\n\t\tevent{\"3\", 1403534920, \"click\", map[string]string{\"button_text\": \"About\"}},\n\t\tevent{\"3\", 1403534921, \"page\", map[string]string{\"url\": \"http://mysite.com/about\"}},\n\t\tevent{\"3\", 1403534922, \"click\", map[string]string{\"button_text\": \"Checkout\"}},\n\t\tevent{\"3\", 1403534923, \"purchase\", map[string]string{\"total\": \"126.99\"}},\n\t\tevent{\"3\", 1403534923, \"page\", map[string]string{\"url\": \"http://mysite.com/thankyou\"}},\n\t}\n\n\tos.MkdirAll(\"tmp\", 0755)\n\n\t// In case we've already created the file.\n\tos.Remove(\"tmp/activity.esdb\")\n\n\twriter, err := esdb.New(\"tmp/activity.esdb\")\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\tfor _, e := range events {\n\t\tvalue, _ := json.Marshal(e.data)\n\n\t\twriter.Add(\n\t\t\t[]byte(e.customerId), // space the event will be stored under.\n\t\t\tvalue,                // value can be any binary data.\n\t\t\te.timestamp,          // all events will be stored sorted by this value.\n\t\t\t\"\",                   // grouping. \"\" here means no grouping, store sequentially by timestamp.\n\t\t\tmap[string]string{\n\t\t\t\t\"type\": e.eventType, // We'll define one secondary index on event type.\n\t\t\t},\n\t\t)\n\t}\n\n\terr = writer.Write()\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\tdb, err := esdb.Open(\"tmp/activity.esdb\")\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\t// Stream through all customer 1's activity\n\tfmt.Println(\"activity for 1:\")\n\tdb.Find([]byte(\"1\")).Scan(\"\", func(event *Event) bool {\n\t\tfmt.Println(string(event.Data))\n\t\treturn true // continue\n\t})\n\n\t// Stream through all customer 2's activity\n\tfmt.Println(\"\\nactivity for 2:\")\n\tdb.Find([]byte(\"2\")).Scan(\"\", func(event *Event) bool {\n\t\tfmt.Println(string(event.Data))\n\t\treturn true // continue\n\t})\n\n\t// Just retrieve customer 1's purchases\n\tfmt.Println(\"\\npurchases for 1:\")\n\tdb.Find([]byte(\"1\")).ScanIndex(\"type\", \"purchase\", func(event *Event) bool {\n\t\tfmt.Println(string(event.Data))\n\t\treturn true // continue\n\t})\n\n\t// Just retrieve customer 3's clicks ordered descending\n\tfmt.Println(\"\\nclicks for 3:\")\n\tdb.Find([]byte(\"3\")).RevScanIndex(\"type\", \"click\", func(event *Event) bool {\n\t\tfmt.Println(string(event.Data))\n\t\treturn true // continue\n\t})\n\n\t// Output:\n\t// activity for 1:\n\t// {\"total\":\"42.99\"}\n\t// {\"url\":\"http://mysite.com/thankyou\"}\n\t// {\"url\":\"http://mysite.com/checkout\"}\n\t// {\"button_text\":\"Checkout\"}\n\t// {\"url\":\"http://mysite.com/\"}\n\t//\n\t// activity for 2:\n\t// {\"url\":\"http://mysite.com/about\"}\n\t// {\"button_text\":\"About\"}\n\t// {\"url\":\"http://mysite.com/\"}\n\t//\n\t// purchases for 1:\n\t// {\"total\":\"42.99\"}\n\t//\n\t// clicks for 3:\n\t// {\"button_text\":\"Checkout\"}\n\t// {\"button_text\":\"About\"}\n}\n```\n\n### Goals/Benefits\n\n1. Fast streaming of events ordered by timestamp.\n\n   Events are stored on disk sequentially ordered by timestamp (or, an optional\ngrouping and timestamp). This means scanning through a series of ordered events\nis extremely fast!\n\n2. Secondary indexes for quickly scanning through just a subset of events.\n\n   Scanning though a subset of events is preferable to scanning through the\nentire event stream, especially when the event stream contains many events\nwhich you aren't currently interested. ESDB allows definition of secondary\nindexes which allow quick retrieval of just events in the index.\n   \n   Secondary indexes are retrieved in timestamp order, and can be scanned\nforward and backwards.\n   \n3. Low overhead.\n\n   Event overhead: as little 3 bytes + 17 bytes for each secondary index for\neach event stored. The 17 byte secondary index overhead is only applied for\nindexes which the particular event is apart.\n   \n   File overhead: offset information based on the number of spaces and\ngroupings which are created is maintained at the end of the file. Generally, in\na reasonably sized file, this should be negligible as event data and per event\noverhead should be the main driver of file size.\n\n### Format \n\n`TODO :(`\n\n### Benchmarks\n\n`TODO :(`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcustomerio%2Fesdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcustomerio%2Fesdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcustomerio%2Fesdb/lists"}