{"id":14956094,"url":"https://github.com/ipfs/go-ds-flatfs","last_synced_at":"2025-04-08T08:17:43.453Z","repository":{"id":10675070,"uuid":"66589944","full_name":"ipfs/go-ds-flatfs","owner":"ipfs","description":"A datastore implementation using sharded directories and flat files to store data","archived":false,"fork":false,"pushed_at":"2025-03-29T08:32:10.000Z","size":291,"stargazers_count":48,"open_issues_count":8,"forks_count":22,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-04-01T07:44:58.796Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ipfs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-08-25T20:14:03.000Z","updated_at":"2025-03-28T13:08:58.000Z","dependencies_parsed_at":"2022-08-07T06:00:17.979Z","dependency_job_id":"29b9543d-9e4f-4c15-b40b-8bb593dbdd74","html_url":"https://github.com/ipfs/go-ds-flatfs","commit_stats":{"total_commits":251,"total_committers":34,"mean_commits":7.382352941176471,"dds":0.7768924302788844,"last_synced_commit":"88dceb19765360fd9659e9dc377e59494bb0d9bd"},"previous_names":[],"tags_count":39,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ipfs%2Fgo-ds-flatfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ipfs%2Fgo-ds-flatfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ipfs%2Fgo-ds-flatfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ipfs%2Fgo-ds-flatfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ipfs","download_url":"https://codeload.github.com/ipfs/go-ds-flatfs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247801175,"owners_count":20998339,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T13:12:18.229Z","updated_at":"2025-04-08T08:17:43.407Z","avatar_url":"https://github.com/ipfs.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# go-ds-flatfs\n\n[![](https://img.shields.io/badge/made%20by-Protocol%20Labs-blue.svg?style=flat-square)](http://ipn.io)\n[![](https://img.shields.io/badge/project-IPFS-blue.svg?style=flat-square)](http://ipfs.io/)\n[![](https://img.shields.io/badge/freenode-%23ipfs-blue.svg?style=flat-square)](http://webchat.freenode.net/?channels=%23ipfs)\n[![standard-readme compliant](https://img.shields.io/badge/standard--readme-OK-green.svg?style=flat-square)](https://github.com/RichardLitt/standard-readme)\n[![GoDoc](https://godoc.org/github.com/ipfs/go-ds-flatfs?status.svg)](https://godoc.org/github.com/ipfs/go-ds-flatfs)\n[![Build Status](https://travis-ci.org/ipfs/go-ds-flatfs.svg?branch=master)](https://travis-ci.org/ipfs/go-ds-flatfs)\n[![Coverage Status](https://img.shields.io/codecov/c/github/ipfs/go-ds-flatfs.svg)](https://codecov.io/gh/ipfs/go-ds-flatfs)\n\n\n\u003e A datastore implementation using sharded directories and flat files to store data\n\n`go-ds-flatfs` is used by `go-ipfs` to store raw block contents on disk. It supports several sharding functions (prefix, suffix, next-to-last/*).\n\nIt is _not_ a general-purpose datastore and has several important restrictions.\nSee the restrictions section for details.\n\n## Lead Maintainer\n\n[Jakub Sztandera](https://github.com/kubuxu)\n\n## Table of Contents\n\n- [Install](#install)\n- [Usage](#usage)\n- [Contribute](#contribute)\n- [License](#license)\n\n## Install\n\n`go-ds-flatfs` can be used like any Go module:\n\n\n```\nimport \"github.com/ipfs/go-ds-flatfs\"\n```\n\n## Usage\n\nCheck the [GoDoc module documentation](https://godoc.org/github.com/ipfs/go-ds-flatfs) for an overview of this module's\nfunctionality.\n\n### Restrictions\n\nFlatFS keys are severely restricted. Only keys that match `/[0-9A-Z+-_=]\\+` are\nallowed. That is, keys may only contain upper-case alpha-numeric characters,\n'-', '+', '_', and '='. This is because values are written directly to the\nfilesystem without encoding.\n\nImportantly, this means namespaced keys (e.g., /FOO/BAR), are _not_ allowed.\nAttempts to write to such keys will result in an error.\n\n### DiskUsage and Accuracy\n\nThis datastore implements the [`PersistentDatastore`](https://godoc.org/github.com/ipfs/go-datastore#PersistentDatastore) interface. It offers a `DiskUsage()` method which strives to find a balance between accuracy and performance. This implies:\n\n* The total disk usage of a datastore is calculated when opening the datastore\n* The current disk usage is cached frequently in a file in the datastore root (`diskUsage.cache` by default). This file is also\nwritten when the datastore is closed.\n* If this file is not present when the datastore is opened:\n  * The disk usage will be calculated by walking the datastore's directory tree and estimating the size of each folder.\n  * This may be a very slow operation for huge datastores or datastores with slow disks\n  * The operation is time-limited (5 minutes by default).\n  * Upon timeout, the remaining folders will be assumed to have the average of the previously processed ones.\n* After opening, the disk usage is updated in every write/delete operation.\n\nThis means that for certain datastores (huge ones, those with very slow disks or special content), the values reported by\n`DiskUsage()` might be reduced accuracy and the first startup (without a `diskUsage.cache` file present), might be slow.\n\nIf you need increased accuracy or a fast start from the first time, you can manually create or update the\n`diskUsage.cache` file.\n\nThe file `diskUsage.cache` is a JSON file with two fields `diskUsage` and `accuracy`.  For example the JSON file for a\nsmall repo might be:\n\n```\n{\"diskUsage\":6357,\"accuracy\":\"initial-exact\"}\n```\n\n`diskUsage` is the calculated disk usage and `accuracy` is a note on the accuracy of the initial calculation.  If the\ninitial calculation was accurate the file will contain the value `initial-exact`.  If some of the directories have too\nmany entries and the disk usage for that directory was estimated based on the first 2000 entries, the file will contain\n`initial-approximate`.  If the calculation took too long and timed out as indicated above, the file will contain\n`initial-timed-out`.\n\nIf the initial calculation timed out the JSON file might be:\n```\n{\"diskUsage\":7589482442898,\"accuracy\":\"initial-timed-out\"}\n\n```\n\nTo fix this with a more accurate value you could do (in the datastore root):\n\n    $ du -sb .\n    7536515831332    .\n    $ echo -n '{\"diskUsage\":7536515831332,\"accuracy\":\"initial-exact\"}' \u003e diskUsage.cache\n\n## Contribute\n\nPRs accepted.\n\nSmall note: If editing the README, please conform to the [standard-readme](https://github.com/RichardLitt/standard-readme) specification.\n\n## License\n\nMIT © Protocol Labs, Inc.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fipfs%2Fgo-ds-flatfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fipfs%2Fgo-ds-flatfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fipfs%2Fgo-ds-flatfs/lists"}