{"id":20704153,"url":"https://github.com/solidsnack/logl","last_synced_at":"2025-03-11T04:40:53.250Z","repository":{"id":1400455,"uuid":"1428079","full_name":"solidsnack/logl","owner":"solidsnack","description":null,"archived":false,"fork":false,"pushed_at":"2011-10-12T23:10:51.000Z","size":310,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-05T17:08:16.746Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/solidsnack.png","metadata":{"files":{"readme":"README","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2011-03-01T21:52:25.000Z","updated_at":"2020-03-10T03:47:17.000Z","dependencies_parsed_at":"2022-07-29T12:49:04.813Z","dependency_job_id":null,"html_url":"https://github.com/solidsnack/logl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/solidsnack%2Flogl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/solidsnack%2Flogl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/solidsnack%2Flogl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/solidsnack%2Flogl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/solidsnack","download_url":"https://codeload.github.com/solidsnack/logl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242973962,"owners_count":20215245,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T01:11:13.203Z","updated_at":"2025-03-11T04:40:53.220Z","avatar_url":"https://github.com/solidsnack.png","language":"Haskell","funding_links":[],"categories":[],"sub_categories":[],"readme":"This work is a result of a conversation with Mark McGranaghan, of Heroku, who\nadvocated logging with compaction as a core interface for reliable storage at\na time when I was quite convinced that random access I/O was the only way to\ngo. The system provides fault tolerant storage and retrieval of append-only,\nbranchy logs. Four log operations are supported:\n\n  * Allocation of a new, empty log.\n\n  * Append of new log entries below a log pointer.\n\n  * Retrieval of the subtree below a log pointer.\n\n  * Deallocation of a log and all its entries.\n\nCompaction is implemented by allocating a new log, copying the good parts into\nit and de-allocating the old log. Query of log entries within a log, on\ntimestamps and content, is planned but not supported at present.\n\n\n -- How A Request Is Handled --------------------------------------------------\n\nA client, wishing to store or retrieve log data, communicates with a single\nfront-end (perhaps assigned at random, it makes no difference which one). The\nfront-end assigns a unique ID for those requests that amount to allocation of\nspace -- creation of a log or log entries -- and, based on the log ID,\ndistributes the query over the backends.\n\n                                   front-end 0            backend 0\n\n    client ---- query/action ----\u003e front-end 1 --+------\u003e backend 1\n                                                 |\n                                   front-end 2   +------\u003e backend 2\n                                                 |\n                                   ...           +------\u003e backend 3\n\n                                                          backend 4\n\n                                                          ...\n\nThe result of a request is the majority result of the backends, if a majority\nexists; if no majority is exists, it is taken to be a rejection of a write or\nan indicator that the data is unavailable for reading.\n\nOnce a log is created, its attributes -- tag and time -- can not be changed;\nand this is similarly the case for log entries. The API does not, in fact,\nprovide any instructions that allow one to write to an item with a given ID;\nIDs for logs and log entries are assigned by a front-end and then forwarded to\nthe backends and returned to the clients.\n\n\n -- Caching, Deletion \u0026 Failure To Retrieve -----------------------------------\n\nThe way a log entry is formed, it must explicitly mention the log entry before\nit. One consequence of this is that concurrent writers branch the log instead\nof interleaving; another is that the ordering in the logs is fundamentally\ncausal, not time-based, and time synchronization issues across the system can\nnot in any way influence the ordering of log entries. A chain of log entries,\nfrom a child to a parent, may be cached indefinitely, as both the content and\nthe structure of the chain can not, in anyway, be affected by subsequent\noperations.\n\nWhen a log is deleted, the effect is as though the log had not been created in\nthe first place; there is no way to distinguish between data that was removed\nand data that never existed in the first place or data for which there has\nbeen an outage that prevents its retrieval.\n\nIt is worth asking, how does deletion interact with the indefinite caching\nproperty? The system does not guarantee to always return a result for a given\nquery; but when it does return a result, the mapping between entry IDs and\nentry contents is always the same and the path from a child to its parent is\nalways the same. This is the meaning of the non-distinguishability of deletion\nand a simple failure to retrieve data.\n\n\n -- Implementation Notes ------------------------------------------------------\n\nThe front-end is implemented in Haskell, a futuristic, purely functional\nprogramming language. The Glorious Glasgow Haskell Compiler's threading\nsupport simplified some aspects of this project.\n\nThe backend is SQL stored procedures for Postgres, chosen for its well\nthought-out approach to data durability. The Haskell front-end is given a few\ndatabase connection strings, connects to each database and loads up the\ndefinition of tables and stored procedures if they are not already present.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsolidsnack%2Flogl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsolidsnack%2Flogl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsolidsnack%2Flogl/lists"}