{"id":19767793,"url":"https://github.com/lighthouse-web3/go-ds-s3","last_synced_at":"2025-02-28T04:19:15.551Z","repository":{"id":64298504,"uuid":"532219279","full_name":"lighthouse-web3/go-ds-s3","owner":"lighthouse-web3","description":null,"archived":false,"fork":false,"pushed_at":"2022-10-31T20:42:01.000Z","size":100,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-11T00:28:23.808Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lighthouse-web3.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-09-03T09:53:06.000Z","updated_at":"2022-10-31T20:12:34.000Z","dependencies_parsed_at":"2023-01-15T09:01:07.547Z","dependency_job_id":null,"html_url":"https://github.com/lighthouse-web3/go-ds-s3","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lighthouse-web3%2Fgo-ds-s3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lighthouse-web3%2Fgo-ds-s3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lighthouse-web3%2Fgo-ds-s3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lighthouse-web3%2Fgo-ds-s3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lighthouse-web3","download_url":"https://codeload.github.com/lighthouse-web3/go-ds-s3/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241100236,"owners_count":19909671,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T04:32:24.556Z","updated_at":"2025-02-28T04:19:15.533Z","avatar_url":"https://github.com/lighthouse-web3.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# S3 Datastore Implementation\n\nThis is an implementation of the datastore interface backed by amazon s3.\n\n**NOTE:** Plugins only work on Linux and MacOS at the moment. You can track the progress of this issue here: https://github.com/golang/go/issues/19282\n\n## Building and Installing\n\nYou must build the plugin with the _exact_ version of go used to build the go-ipfs binary you will use it with. You can find the go version for go-ipfs builds from dist.ipfs.io in the build-info file, e.g. https://dist.ipfs.io/go-ipfs/v0.4.22/build-info or by running `ipfs version --all`.\n\nIn addition to needing the exact version of go, you need to build the correct version of this plugin.\n\n- To build against a released version of go-ipfs, checkout the `release/v$VERSION` branch and build.\n- To build against a custom (local) build of go-ipfs, run `make IPFS_VERSION=/path/to/go-ipfs/source`.\n\nYou can then install it into your local IPFS repo by running `make install`.\n\n## Bundling\n\nAs go plugins can be finicky to correctly compile and install, you may want to consider bundling this plugin and re-building go-ipfs. If you do it this way, you won't need to install the `.so` file in your local repo, i.e following the above Building and Installing section, and you won't need to worry about getting all the versions to match up.\n\n```bash\n# We use go modules for everything.\n\u003e export GO111MODULE=on\n\n# Clone go-ipfs.\n\u003e git clone https://github.com/ipfs/go-ipfs\n\u003e cd go-ipfs\n\n# Pull in the datastore plugin (you can specify a version other than latest if you'd like).\n\u003e go get github.com/lighthouse-web3/go-ds-s3/plugin@latest\n\n# Add the plugin to the preload list.\n\u003e echo -en \"\\ns3ds github.com/lighthouse-web3/go-ds-s3/plugin 0\" \u003e\u003e plugin/loader/preload_list\n\n# ( this first pass will fail ) Try to build go-ipfs with the plugin\n\u003e make build\n\n# Update the deptree\n\u003e go mod tidy\n\n# Now rebuild go-ipfs with the plugin\n\u003e make build\n\n# (Optionally) install go-ipfs\n\u003e make install\n```\n\n## Detailed Installation\n\nFor a brand new ipfs instance (no data stored yet):\n\n1. Copy s3plugin.so $IPFS_DIR/plugins/go-ds-s3.so (or run `make install` if you are installing locally).\n2. Run `ipfs init`.\n3. Edit $IPFS_DIR/config to include s3 details (see Configuration below).\n4. Overwrite `$IPFS_DIR/datastore_spec` as specified below (_Don't do this on an instance with existing data - it will be lost_).\n\n### Configuration\n\nThe config file should include the following:\n\n```json\n{\n  \"Datastore\": {\n  ...\n\n    \"Spec\": {\n      \"mounts\": [\n        {\n          \"child\": {\n            \"type\": \"s3ds\",\n            \"region\": \"us-east-1\",\n            \"bucket\": \"$bucketname\",\n            \"rootDirectory\": \"$bucketsubdirectory\",\n            \"accessKey\": \"\",\n            \"secretKey\": \"\"\n          },\n          \"mountpoint\": \"/blocks\",\n          \"prefix\": \"s3.datastore\",\n          \"type\": \"measure\"\n        },\n```\n\nIf the access and secret key are blank they will be loaded from the usual ~/.aws/.\nIf you are on another S3 compatible provider, e.g. Linode, then your config should be:\n\n```json\n{\n  \"Datastore\": {\n  ...\n\n    \"Spec\": {\n      \"mounts\": [\n        {\n          \"child\": {\n            \"type\": \"s3ds\",\n            \"region\": \"us-east-1\",\n            \"bucket\": \"$bucketname\",\n            \"rootDirectory\": \"$bucketsubdirectory\",\n            \"regionEndpoint\": \"us-east-1.linodeobjects.com\",\n            \"accessKey\": \"\",\n            \"secretKey\": \"\"\n          },\n          \"mountpoint\": \"/blocks\",\n          \"prefix\": \"s3.datastore\",\n          \"type\": \"measure\"\n        },\n```\n\nIf you are configuring a brand new ipfs instance without any data, you can overwrite the datastore_spec file with:\n\n```\n{\"mounts\":[{\"bucket\":\"$bucketname\",\"mountpoint\":\"/blocks\",\"region\":\"us-east-1\",\"rootDirectory\":\"$bucketsubdirectory\"},{\"mountpoint\":\"/\",\"path\":\"datastore\",\"type\":\"levelds\"}],\"type\":\"mount\"}\n```\n\nOtherwise, you need to do a datastore migration.\n\n## Contribute\n\nFeel free to join in. All welcome. Open an [issue](https://github.com/lighthouse-web3/go-ds-s3/issues)!\n\n### Want to hack on IPFS?\n\n[![](https://cdn.rawgit.com/jbenet/contribute-ipfs-gif/master/img/contribute.gif)](https://github.com/ipfs/community/blob/master/CONTRIBUTING.md)\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flighthouse-web3%2Fgo-ds-s3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flighthouse-web3%2Fgo-ds-s3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flighthouse-web3%2Fgo-ds-s3/lists"}