{"id":21206183,"url":"https://github.com/rtmigo/pickledir_py","last_synced_at":"2026-04-17T06:35:19.402Z","repository":{"id":46321166,"uuid":"366860234","full_name":"rtmigo/pickledir_py","owner":"rtmigo","description":"File-based key-value storage. Serializes keys and values with pickle","archived":false,"fork":false,"pushed_at":"2022-01-02T17:43:43.000Z","size":91,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-09-29T01:18:57.518Z","etag":null,"topics":["cache","caching","data","directory","file","linux","macos","package","pickle","python","windows"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/pickledir/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rtmigo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-12T21:38:18.000Z","updated_at":"2022-07-18T19:50:05.000Z","dependencies_parsed_at":"2022-09-02T06:53:02.730Z","dependency_job_id":null,"html_url":"https://github.com/rtmigo/pickledir_py","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"purl":"pkg:github/rtmigo/pickledir_py","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rtmigo%2Fpickledir_py","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rtmigo%2Fpickledir_py/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rtmigo%2Fpickledir_py/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rtmigo%2Fpickledir_py/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rtmigo","download_url":"https://codeload.github.com/rtmigo/pickledir_py/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rtmigo%2Fpickledir_py/sbom","scorecard":{"id":788176,"data":{"date":"2025-08-11","repo":{"name":"github.com/rtmigo/pickledir_py","commit":"f5e71e4923df59483dfe2a999735c5d6f7a8f9fb"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.8,"checks":[{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/ci.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:27: update your workflow using https://app.stepsecurity.io/secureworkflow/rtmigo/pickledir_py/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:29: update your workflow using https://app.stepsecurity.io/secureworkflow/rtmigo/pickledir_py/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:57: update your workflow using https://app.stepsecurity.io/secureworkflow/rtmigo/pickledir_py/ci.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:59: update your workflow using https://app.stepsecurity.io/secureworkflow/rtmigo/pickledir_py/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:73: update your workflow using https://app.stepsecurity.io/secureworkflow/rtmigo/pickledir_py/ci.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:77: update your workflow using https://app.stepsecurity.io/secureworkflow/rtmigo/pickledir_py/ci.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:87: update your workflow using https://app.stepsecurity.io/secureworkflow/rtmigo/pickledir_py/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:95: update your workflow using https://app.stepsecurity.io/secureworkflow/rtmigo/pickledir_py/ci.yml/master?enable=pin","Warn: pipCommand not pinned by hash: pyrel.sh:148","Warn: pipCommand not pinned by hash: pyrel.sh:149","Warn: pipCommand not pinned by hash: pyrel.sh:177","Warn: pipCommand not pinned by hash: pyrel.sh:178","Warn: pipCommand not pinned by hash: pyrel.sh:214","Warn: pipCommand not pinned by hash: .github/workflows/ci.yml:34","Warn: pipCommand not pinned by hash: .github/workflows/ci.yml:35","Warn: pipCommand not pinned by hash: .github/workflows/ci.yml:36","Warn: pipCommand not pinned by hash: .github/workflows/ci.yml:46","Warn: pipCommand not pinned by hash: .github/workflows/ci.yml:100","Warn: pipCommand not pinned by hash: .github/workflows/ci.yml:101","Info:   0 out of   5 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   3 third-party GitHubAction dependencies pinned","Info:   0 out of  11 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-23T06:40:48.660Z","repository_id":46321166,"created_at":"2025-08-23T06:40:48.660Z","updated_at":"2025-08-23T06:40:48.660Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31918825,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-16T18:22:33.417Z","status":"online","status_checked_at":"2026-04-17T02:00:06.879Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache","caching","data","directory","file","linux","macos","package","pickle","python","windows"],"created_at":"2024-11-20T20:54:43.460Z","updated_at":"2026-04-17T06:35:19.383Z","avatar_url":"https://github.com/rtmigo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [pickledir](https://github.com/rtmigo/pickledir_py#readme)\n\nFile-based key-value storage.\n\nKeys and values are serialized\nwith [pickle](https://docs.python.org/3/library/pickle.html). Data is kept in\nfiles in the specified directory.\n\nCI-tested with Python 3.8-3.9 on macOS, Ubuntu and Windows.\n\n---\n\n\nThe storage has zero initialization time, fast random access, fast reads and\nwrites.\n\nUnlike [shelve](https://docs.python.org/3/library/shelve.html), the data saved\nby PickleDir is cross-platform: you can write it on Linux and read the same\nfiles on Windows. Unlike most database-based caching solutions (including the\nshelve), the PickleDir does not require the \"open\" and \"close\" storage. It's\nalways open since it's just a directory in the file system.\n\nPickleDir is better for casual data storage. Database-based solutions are\npreferred when your storage has many elements (3 thousand or more). They will\nalso be faster when working with a predictably high load in terms of reading and\nwriting.\n\n# Install\n\n``` bash\n$ pip3 install pickledir\n```\n\n# Use\n\n## Create\n\n``` python3\nfrom pickledir import PickleDir\n\ncache = PickleDir('path/to/my_cache_dir')\n```\n\n## Write\n\nKeys do not need to be hashable. They only need to be serializable with `pickle`\n.\n\nWhen you assign a value, the data is literally written to a file.\n\n``` python3\ncache['key'] = 'hello, user!'\ncache[5] = 23\ncache[{'a', 'b', 'c'}] = 'abc'\n```\n\n## Read\n\n``` python3\nprint(cache['key'])\nprint(cache[5])\nprint(cache[{'a', 'b', 'c'}])\n```\n\n## Read all values\n\n``` python3 \nfor key, value in cache.items():\n    print(key, value)\n```    \n\n## Delete item\n\n``` python3\ndel cache['key']\n```\n\n## Type hints\n\n``` python3\n# declaring PickleDir with string keys and integer values:\n\ncache: PickleDir[str, int] = PickleDir('path/to/my_cache_dir')\n```\n\n## Set expiration time on writing\n\nThe expired items will be removed from the storage.\n\n``` python3    \ncache.set('a', 1000, max_age = datetime.timedelta(seconds=1))\nprint(cache.get('a'))  # 1000\ntime.sleep(2)     \nprint(cache.get('a'))  # None (and removed from storage)\n```\n\n## Set expiration time on reading\n\nThe expired items will not be returned, but kept in the storage.\n\n``` python3\ncache['b'] = 1000\ntime.sleep(2)\ncache.get('b' max_age = datetime.timedelta(seconds=1)) # None\ncache.get('b' max_age = datetime.timedelta(seconds=9)) # 1000\n```\n\n## Set data version\n\nSetting the data version makes it easy to mark old data as obsolete.\n\nFor example, you cached the result of a function, and then changed the\nimplementation of that function. In this case, there is no need to delete old\nfiles from the cache. Just change the version number.\n\n``` python3 \ncache = PickleDir('path/to/dir', version=1)\ncache['a'] = 'some_data'\n```\n\nYou can read all stored data while the `version` value is `1`.\n\n``` python3 \ncache = PickleDir('path/to/dir', version=1)\nprint(cache.get('a'))  # 'some_data'\n```\n\nIf you decide that all the data in the cache is out of date, just pass the\nconstructor a version number that you haven't used before.\n\n``` python3 \ncache = PickleDir('path/to/dir', version=2)\nprint(cache.get('a'))  # None\n```\n\nNow all that is saved with version `2` is actual data. Any other version is\nconsidered obsolete and will be gradually removed.\n\nDo not create the `PickleDir` with an old version number. It will make the data\nunpredictable.\n\n``` python3\ncacheV1 = PickleDir('path/to/dir', version=1)  # ok\ncacheV1['a'] = 'old A'\ncacheV1['b'] = 'old B'\n\ncacheV2 = PickleDir('path/to/dir', version=2)  # ok\ncacheV2['a'] = 'new A'\n\ncacheV1 = PickleDir('path/to/dir', version=1)  # don't do this\nprint(cacheV1.get('b'))  # Schrödinger's data ('old B' or None)\n```\n\n# Benchmarks\n\nCasually saving 10 items and reading them again:\n\n``` python3\nfor i in range(10):\n    cache[str(i)] = {\"data\": i, \"other\": None}\nfor i in range(10):\n    _ = cache[str(i)]\n```\n\nStorage | Time\n--------|-----\n`PickleDir` | 0.42\n[`shelve`](https://docs.python.org/3/library/shelve.html) | 6.68\n[`diskcache.Cache`](https://pypi.org/project/diskcache/) | 1.09\n\nMeasured on macOS, Python 3.8, SATA HDD (not SSD), Journaled HFS+.\n\nSee sources\nin [benchmark](https://github.com/rtmigo/pickledir_py/tree/dev/benchmark) dir.\n\nThe main advantage of\n`pickledir` is the lack of time required to create a database or initialize\ntables. If we did not save 10 items, but 1000 in a row,\n`shelve` and `diskcache` would be faster than `pickledir`.\n\n# Under the hood\n\nSerialized data is stored inside files in the same directory. Each file contains\none or more items. The maximum number of files is limited to 4096. The values\nare uniformly distributed between the files.\n\nReading is slower when a file contains more than one item. Therefore, the\nPickleDir is better suited for cases with the number of items within a few\nthousand.\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frtmigo%2Fpickledir_py","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frtmigo%2Fpickledir_py","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frtmigo%2Fpickledir_py/lists"}