{"id":21620747,"url":"https://github.com/miku/microblob","last_synced_at":"2025-04-10T05:07:21.787Z","repository":{"id":57552292,"uuid":"82800367","full_name":"miku/microblob","owner":"miku","description":"Serve millions of JSON documents via HTTP.","archived":false,"fork":false,"pushed_at":"2024-11-07T08:47:50.000Z","size":6521,"stargazers_count":68,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-10T05:07:01.729Z","etag":null,"topics":["http","json","key-value","store"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/miku.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-02-22T12:11:58.000Z","updated_at":"2025-03-11T12:32:29.000Z","dependencies_parsed_at":"2024-06-20T16:34:41.422Z","dependency_job_id":"025aab09-fb00-4c89-9096-b71724090821","html_url":"https://github.com/miku/microblob","commit_stats":{"total_commits":227,"total_committers":2,"mean_commits":113.5,"dds":0.004405286343612369,"last_synced_commit":"3050d596479484088431801b9a3a1deaeb0e03dd"},"previous_names":[],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miku%2Fmicroblob","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miku%2Fmicroblob/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miku%2Fmicroblob/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miku%2Fmicroblob/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/miku","download_url":"https://codeload.github.com/miku/microblob/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248161273,"owners_count":21057555,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["http","json","key-value","store"],"created_at":"2024-11-24T23:12:39.835Z","updated_at":"2025-04-10T05:07:21.751Z","avatar_url":"https://github.com/miku.png","language":"Go","readme":"# microblob\n\n----\n\nNOTICE: Main deployment has been decommissioned. Do not expect this repo to\nreceive updates in the future.\n\n----\n\nmicroblob is a simplistic key-value store, that serves JSON documents from a\nfile over HTTP. It is implemented in a few hundred lines of code and does not\ncontain many features.\n\nWarning: This server **SHOULD NEVER BE EXPOSED PUBLICLY** as it contains no\nsecurity, rate-limiting or other safety measures whatsoever.\n\nmicroblob was written in 2017 as an ad-hoc solution to replace a previous setup\nusing [memcachedb](https://en.wikipedia.org/wiki/MemcacheDB) (which was getting\nslow). The main goal has been to serve about 200M JSON documents from a\n\"persistent key-value store\" over HTTP and to support frequent, fast rebuilds;\nwith limited disk space and potentially limited memory. Code lacks tests and I\nwould write it differently today. However, it ran without issues and happily\nserved up to 400 requests/s with limited resources and with average response\ntimes of around 1ms.\n\n[![DOI](https://zenodo.org/badge/82800367.svg)](https://zenodo.org/badge/latestdoi/82800367) [![Project Status: Inactive – The project has reached a stable, usable state but is no longer being actively developed; support/maintenance will be provided as time allows.](https://www.repostatus.org/badges/latest/inactive.svg)](https://www.repostatus.org/#inactive)\n\nThis project has been developed for [Project finc](https://finc.info) at [Leipzig University Library](https://ub.uni-leipzig.de).\n\n```shell\n$ cat file.ldj\n{\"id\": \"some-id-1\", \"name\": \"alice\"}\n{\"id\": \"some-id-2\", \"name\": \"bob\"}\n\n$ microblob -key id file.ldj\nINFO[0000] creating db fixtures/file.ldj.832a9151.db ...\nINFO[0000] listening at http://127.0.0.1:8820 (fixtures/file.ldj.832a9151.db)\n```\n\nIt supports fast rebuilds from scratch, as the preferred way to deploy this is\nfor a *build-once* *update-never* use case. It scales up and down with memory\nand can serve hundred million documents and more.\n\nInspiration: [So what's wrong with 1975\nprogramming?](http://varnish-cache.org/docs/trunk/phk/notes.html#so-what-s-wrong-with-1975-programming)\nIdea: Instead of implementing complicated caching mechanisms, we hand over\ncaching completely to the operating system and try to stay out of its way.\n\nInserts are fast, since no data is actually moved. 150 million (1kB) documents\ncan be serveable within an hour.\n\n* ㊗️ 2017-06-30 first 100 million requests served in production\n\nFurther documentation: [docs/microblob.md](docs/microblob.md)\n\n# Update via curl\n\nTo send compressed data with curl:\n\n```shell\n$ curl -v --data-binary @- localhost:8820/update?key=id \u003c \u003c(gunzip -c fixtures/fake.ldj.gz)\n...\n```\n\n# Usage\n\n```shell\nUsage of microblob:\n  -addr string\n        address to serve (default \"127.0.0.1:8820\")\n  -backend string\n        backend to use: leveldb, debug (default \"leveldb\")\n  -batch int\n        number of lines in a batch (default 50000)\n  -c string\n        load options from a config (ini) file\n  -create-db-only\n        build the database only, then exit\n  -db string\n        the root directory, by default: 1000.ldj -\u003e 1000.ldj.05028f38.db (based on flags)\n  -ignore-missing-keys\n        ignore record, that do not have a the specified key\n  -key string\n        key to extract, json, top-level only\n  -log string\n        access log file, don't log if empty\n  -r string\n        regular expression to use as key extractor\n  -s string\n        the config file section to use (default \"main\")\n  -t    top level key extractor\n  -version\n        show version and exit\n```\n\n# What it doesn't do\n\n* no deletions (microblob is currently append-only and does not care about\n  garbage, so if you add more and more things, you will run out of space)\n* no compression (yet)\n* no security (anyone can query or update via HTTP)\n\n# Installation\n\nDebian and RPM packages: see [releases](https://github.com/miku/microblob/releases).\n\nOr:\n\n```shell\n$ go install github.com/miku/microblob/cmd/microblob@latest\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiku%2Fmicroblob","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmiku%2Fmicroblob","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiku%2Fmicroblob/lists"}