{"id":15489769,"url":"https://github.com/theandrew168/pymkv","last_synced_at":"2025-08-16T15:10:17.767Z","repository":{"id":134126274,"uuid":"327468094","full_name":"theandrew168/pymkv","owner":"theandrew168","description":"Exploring George Hotz's minikeyvalue design in Python","archived":false,"fork":false,"pushed_at":"2021-01-10T18:22:15.000Z","size":39,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-08-16T12:45:14.382Z","etag":null,"topics":["go","key-value","nginx","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/theandrew168.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-07T01:09:36.000Z","updated_at":"2025-04-27T16:31:50.000Z","dependencies_parsed_at":null,"dependency_job_id":"1b15a839-5aea-4c47-aa5b-20e24a80bed7","html_url":"https://github.com/theandrew168/pymkv","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/theandrew168/pymkv","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theandrew168%2Fpymkv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theandrew168%2Fpymkv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theandrew168%2Fpymkv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theandrew168%2Fpymkv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/theandrew168","download_url":"https://codeload.github.com/theandrew168/pymkv/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theandrew168%2Fpymkv/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270728168,"owners_count":24635153,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-16T02:00:11.002Z","response_time":91,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","key-value","nginx","python"],"created_at":"2024-10-02T07:07:44.107Z","updated_at":"2025-08-16T15:10:17.733Z","avatar_url":"https://github.com/theandrew168.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pymkv\nExploring George Hotz's [minikeyvalue](https://github.com/geohot/minikeyvalue) design in Python.\nOverall, I think George's implementation (and language choice) is better.\nSure, my Python version can match his when it comes to performance and throughput but it takes an extra reverse proxy layer (more NGINX) to get there.\nGo really is a powerhouse when it comes to high-performance network-based applications and it shows.\n\n## Design\nVolume \"servers\" store all of the actual data.\nThe data is evenly distributed into a predictable nested directory structure based on the base64'd hash of its contents.\nCould be text, a file, doesn't really matter.\nIt's all bytes.\nThis part of the program is just an NGINX subprocess.\nThe config is generated as needed, thrown into a tempfile, and started as a child process.\nNGINX then starts up its own worker child processes.\n\nThe index server manages a mapping of where stuff lives on the volume servers.\nA [DBM database](https://docs.python.org/3/library/dbm.html) is used to store this mapping in a fast, persistent way.\nThe index server also exposes the primary API.\nGET gets data, PUT stores data, DELETE deletes data.\nPretty simple!\nThe index server talks with the volume servers in a similar manner.\n\n## System Dependencies\nBesides [Python](https://www.python.org), this project also depends on having [NGINX](http://nginx.org/) installed.\n```\n# linux, debian-based\nsudo apt install nginx\n\n# macos\nbrew install nginx\n```\n\n## Python Dependencies\nIf you are unfamiliar with [virtual environments](https://docs.python.org/3/library/venv.html), I suggest taking a brief moment to learn about them and set one up.\nThe Python docs provide a great [tutorial](https://docs.python.org/3/tutorial/venv.html) for getting started with virtual environments and packages.\n\nInstall the project's Python dependencies via:\n```\npip install -r requirements.txt\n```\n\n## Running\nFirst start one or more volume servers (these could run on different systems):\n```\n# ignore the nginx \"could not open error log\" warnings\npython3 volume.py 3001 /tmp/volume1/ \u0026\npython3 volume.py 3002 /tmp/volume2/ \u0026\npython3 volume.py 3003 /tmp/volume3/ \u0026\n```\n\nThen start the index server using the addresses of the volume servers:\n```\npython3 index.py localhost:3001 localhost:3002 localhost:3003\n```\n\n## Usage\nWith the index server running on port 3000, the following commands demonstrate core functionality:\n```\n# put \"bigswag\" in key \"wehave\"\ncurl -v -L -X PUT -d bigswag http://localhost:3000/wehave\n\n# get key \"wehave\" (should be \"bigswag\")\ncurl -v -L -X GET http://localhost:3000/wehave\n\n# delete key \"wehave\"\ncurl -v -L -X DELETE http://localhost:3000/wehave\n\n# put file in key \"file.txt\"\ncurl -v -L -X PUT -T /path/to/local/file.txt http://localhost:3000/file.txt\n\n# get file in key \"file.txt\"\ncurl -v -L -X GET -o /path/to/local/file.txt http://localhost:3000/file.txt\n```\n\n## Performance\nAll of these benchmarks were executed against a small cluster (1 index, 3 volumes) of Digital Ocean droplets (1vCPU, 512MB, $5/month).\n\n| **Benchmark** | **Command** |\n| --- | --- |\n| **fetch missing** | `hey -c 100 -z 10s http://\u003cindex_server_ip\u003e:3000/missing` |\n| **fetch present** | `hey -c 100 -z 10s http://\u003cindex_server_ip\u003e:3000/present` |\n| **thrasher.go** | `go run extras/thrasher.go -addr \u003cindex_server_ip\u003e:3000` |\n\nEach of these results are the average of three trials and are measured in requests per second.\n\n| **Benchmark** | **[pymkv](https://github.com/theandrew168/pymkv)** | **[minikeyvalue](https://github.com/geohot/minikeyvalue)** |\n| --- | --- | --- |\n| **fetch missing** | 1397 | 2152 |\n| **fetch present** | 1012 | 941 |\n| **thrasher.go** | 99 | 102 |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftheandrew168%2Fpymkv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftheandrew168%2Fpymkv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftheandrew168%2Fpymkv/lists"}