{"id":15061723,"url":"https://github.com/allisterb/go-ds-motr","last_synced_at":"2026-01-02T12:49:21.258Z","repository":{"id":37265969,"uuid":"502610599","full_name":"allisterb/go-ds-motr","owner":"allisterb","description":"IPFS data store plugin for the CORTX Motr key-value store","archived":false,"fork":false,"pushed_at":"2022-07-04T18:26:35.000Z","size":256,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-22T02:14:59.565Z","etag":null,"topics":["cortx","ipfs"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/allisterb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-12T12:31:46.000Z","updated_at":"2022-06-27T15:02:16.000Z","dependencies_parsed_at":"2022-07-12T05:00:58.877Z","dependency_job_id":null,"html_url":"https://github.com/allisterb/go-ds-motr","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allisterb%2Fgo-ds-motr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allisterb%2Fgo-ds-motr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allisterb%2Fgo-ds-motr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allisterb%2Fgo-ds-motr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/allisterb","download_url":"https://codeload.github.com/allisterb/go-ds-motr/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243723178,"owners_count":20337327,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cortx","ipfs"],"created_at":"2024-09-24T23:24:13.608Z","updated_at":"2026-01-02T12:49:21.218Z","avatar_url":"https://github.com/allisterb.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# About\n\ngo-ds-motr is a IPFS [data store plugin](https://github.com/ipfs/go-datastore) implementation that uses the Go bindings to the CORTX [Motr C API](https://github.com/Seagate/cortx-motr/blob/main/doc/motr-developer-guide.md) to store IPFS data directly in indexes in the Motr key-value store. This allows IPFS servers to use the full capabilities and scalability of CORTX, instead of relying on a generic S3 REST API and HTTP calls. go-ds-motr stores and retrieves IPFS blocks from Motr using the native Motr client API when requested by the other IPFS subsystems using Motr key ids derived from the IPFS CIDs. go-ds-motr can consume CORTX-specific configuration and parameters specified via the IPFS configuration file and can access the full range of native functionality exposed by the Motr client API.\n\nIn simple benchmarks go-ds-motr is vastly more performant than the S3 data store plugin, achieving a 13x speedup for add operations:\n#### Adding 93Mb file to IPFS using S3 data store from cold start:\n```cmd\n[root@cortx-ova-rgw go-ds-motr]# time ../go-ipfs/cmd/ipfs/ipfs add \"01 Track01.flac\"                                                                                                           \nadded QmXUdQD5gHs483TCYFTEgFsve4J1sgfM4FGs9XLZzE3obv 01 Track01.flac                                                                                                                           \n 93.83 MiB / 93.83 MiB [=======================================================================================================================================================================\n======================] 100.00%                                                                                                                                                                \nreal    1m20.728s                                                                                                                                                                              \nuser    0m0.437s                                                                                                                                                                               \nsys     0m0.334s                                                                                                            \n```\n#### Adding 93Mb file to IPFS using go-ds-motr from cold start:\n```cmd\n[root@venus go-ds-motr]# time ../go-ipfs/cmd/ipfs/ipfs add \"01 Track01.flac\"                                                                                                                   \nadded QmXUdQD5gHs483TCYFTEgFsve4J1sgfM4FGs9XLZzE3obv 01 Track01.flac                                                                                                                           \n 93.83 MiB / 93.83 MiB [==============================================================================================================================================================] 100.00%\nreal    0m6.308s                                                                                                \nuser    0m0.615s                                                                                             \nsys     0m0.117s\n```\n\n\n# Requirements\n1. Access to a Motr cluster and the `motr-devel` package installed. See the CORTX [repo](https://github.com/Seagate/cortx/releases).\n2. Go 1.17. See the official Go [site](https://go.dev/doc/install) for download and install instructions.\n\n# Installation\n1. Clone the go-ipfs repository: `git clone https://github.com/ipfs/go-ipfs.git`\n2. Checkout v0.13.0 of the code: `cd go-ipfs \u0026\u0026  git checkout tags/v0.13.0`\n3. Add the following line to the `plugin/loader/preload.list` file: `motords github.com/allisterb/go-ds-motr/plugin *`. This tells the build system to build the go-ds-motr plugin together with the other plugins specified when building go-ipfs.\n4. Get the Go go-ds-motr module code from GitHub: `go get github.com/allisterb/go-ds-motr/plugin@HEAD`.\n5. Build go-ipfs: `make build`. This will create the ipfs executable at `cmd/ipfs/ipfs` in your `go-ipfs` directory.\n6. Initialize your IPFS repository: `cmd/ipfs/ipfs init`. This will create an `.ipfs` directory in your `$HOME` directory with your IPFS server configuration.\n7. Edit the `config` JSON configuration file In your `$HOME/.ipfs` directory. Change the `Spec` key to the following structure:\n    ```json\n    \"Spec\": {\n              \"child\": {\n                \"haxAddr\": \"inet:tcp:192.168.1.161@22001\",\n                \"index\": \"0x7800000000000123:0x123456780\",\n                \"leveldbPath\": \"/home/allisterb/.leveldb/ipfs\",\n                \"localAddr\": \"inet:tcp:192.168.1.161@22501\",\n                \"processFid\": \"0x7200000000000001:0x3\",\n                \"profileFid\": \"0x7000000000000001:0x0\",\n                \"type\": \"motrds\"\n              },\n              \"prefix\": \"motr.datastore\",\n              \"type\": \"measure\"\n            },\n    ```\n    Set `haxAddr`, `localAddr`, `profileFid`, `processFid`, `index`, to your Motr HA endpoint address, local endpoint address, cluster profile FID, local process FID, and Motr key-value index name respectively. Set `levelDBPath` to the path where the LevelDB key index will be stored on-disk e.g. `$HOME/.leveldb/ipfs`. An example `config` file using the go-ds-motr plugin is [here](https://github.com/allisterb/go-ds-motr)\n8. Edit the `datastore_spec` file in your `$HOME/.ipfs` folder and replace with the following text:\n```json \n{\"haxAddr\":\"inet:tcp:192.168.1.161@22001\",\"index\":\"0x7800000000000123:0x123456780\",\"leveldbPath\":\"/home/allisterb/.leveldb/ipfs\",\"localAddr\":\"inet:tcp:192.168.1.161@22501\",\"processFid\":\"0x7200000000000001:0x3\",\"profileFid\":\"0x7000000000000001:0x0\"}\n```\nEnter the configuration values just as you entered for the `.config` file. A sample IPFS `datastore_spec` file is [here](https://github.com/allisterb/go-ds-motr/blob/master/config_example/datastore_spec).\n\n9. To test these values before starting the server, clone `https://github.com/allisterb/go-ds-motr` and run the CLI commands to test access to the store e.g:\n` ./run.sh store -L inet:tcp:192.168.1.161@22501 -H inet:tcp:192.168.1.161@22001 -P 0x7200000000000001:0x3 -C 0x7000000000000001:0x0 0x7800000000000123:0x123456780 foo bar`\nwill store the value `bar` for the key `foo` in the Motr  index `0x7800000000000123:0x123456780`\n```cmd\n[allisterb@mars go-ds-motr]$ ./run.sh store -L inet:tcp:192.168.1.161@22501 -H inet:tcp:192.168.1.161@22001 -P 0x7200000000000001:0x3 -C 0x7000000000000001:0x0 0x7800000000000123:0x123456780 foo bar\n   ____                   ____                  __  __           _\n  / ___|   ___           |  _ \\   ___          |  \\/  |   ___   | |_   _ __\n | |  _   / _ \\   _____  | | | | / __|  _____  | |\\/| |  / _ \\  | __| | '__|\n | |_| | | (_) | |_____| | |_| | \\__ \\ |_____| | |  | | | (_) | | |_  | |\n  \\____|  \\___/          |____/  |___/         |_|  |_|  \\___/   \\__| |_|\n\n2022-06-24T15:19:03.979-0400    INFO    CLI     go-ds-motr/main.go:93   Initialized Motr client.\n2022-06-24T15:19:03.979-0400    INFO    CLI     go-ds-motr/main.go:98   initialized Motr key-value index 0x7800000000000123:0x123456780.\n2022-06-24T15:19:03.985-0400    INFO    CLI     go-ds-motr/main.go:178  Put object at key foo in index 0x7800000000000123:0x123456780\n[allisterb@mars go-ds-motr]$ ./run.sh store -L inet:tcp:192.168.1.161@22501 -H inet:tcp:192.168.1.161@22001 -P 0x7200000000000001:0x3 -C 0x7000000000000001:0x0 0x7800000000000123:0x123456780 foo -s\n    ____                   ____                  __  __           _\n  / ___|   ___           |  _ \\   ___          |  \\/  |   ___   | |_   _ __\n | |  _   / _ \\   _____  | | | | / __|  _____  | |\\/| |  / _ \\  | __| | '__|\n | |_| | | (_) | |_____| | |_| | \\__ \\ |_____| | |  | | | (_) | | |_  | |\n  \\____|  \\___/          |____/  |___/         |_|  |_|  \\___/   \\__| |_|\n\n2022-06-24T16:40:15.926-0400    INFO    CLI     go-ds-motr/main.go:93   Initialized Motr client.\n2022-06-24T16:40:15.927-0400    INFO    CLI     go-ds-motr/main.go:98   initialized Motr key-value index 0x7800000000000123:0x123456780.\n2022-06-24T16:40:15.961-0400    INFO    CLI     go-ds-motr/main.go:214  The size of object at key foo in index 0x7800000000000123:0x123456780 is 3.\n```\n10. If you need to create the Motr index you can do that from the CLI too:\n```cmd\n[allisterb@mars go-ds-motr]$ ./run.sh index -L inet:tcp:192.168.1.161@22501 -H inet:tcp:192.168.1.161@22001 -P 0x7200000000000001:0x3 -C 0x7000000000000001:0x0 0x7800000000000123:0x123456780 --create\n   ____                   ____                  __  __           _\n  / ___|   ___           |  _ \\   ___          |  \\/  |   ___   | |_   _ __\n | |  _   / _ \\   _____  | | | | / __|  _____  | |\\/| |  / _ \\  | __| | '__|\n | |_| | | (_) | |_____| | |_| | \\__ \\ |_____| | |  | | | (_) | | |_  | |\n  \\____|  \\___/          |____/  |___/         |_|  |_|  \\___/   \\__| |_|\n\n2022-06-25T14:28:51.766-0400    INFO    CLI     go-ds-motr/main.go:121  Initialized Motr client.\n2022-06-25T14:28:51.766-0400    INFO    motrds  mio/mkv.go:95   Creating index 0x7800000000000123:0x123456780...\n2022-06-25T14:28:51.768-0400    INFO    CLI     go-ds-motr/main.go:231  Created or opened existing Motr key-value index 0x7800000000000123:0x123456780.\n```                                                                                                                                                                                          \n11. Set the [UDP receive buffer size](https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size) to 2500000 to avoid [this warning message](https://discuss.ipfs.io/t/docker-failed-to-sufficiently-increase-receive-buffer-size/12498) when starting IPFS: `sudo sysctl -w net.core.rmem_max=2500000`\n \n12. When everything is ready, start the IPFS server: \n`GOLOG_LOG_LEVEL=error,motrds=debug cmd/ipfs/ipfs daemon`.\nThis will start the go-ipfs server with the default logging level set to only print errors except for the motrds plugin which will be logging in debug mode.\n![goipfsstartup](https://dm2301files.storage.live.com/y4mHDFP81DM0sRwtw_q4V3l5ksiUxmbCwrzalWucqAokzwJhAj4OAnEMldPP96pDUc8NXdmeFH2Pb_DRjeSqqb4QRPpLoCTP0PfQHcOLVdea81e4mxBKkVuwitPkdrXOUAsvn4ZgoLpYN6afZY9E9Y0lZ6m58ulscymR-MVYdGJfzyRm1DsO1I8vNxQY6EnP-t8?width=1920\u0026height=884\u0026cropmode=none)\nYou should see diagnostic messages from the motrds plugin indicating it initialized successfully and is handling queries and requests for data from IPFS.\n\n# Benchmarking\nYou can run `benchmark.sh` from the go-ds-motr repo to get a idea of how performant the data store is\n\n![benchmark](https://dm2301files.storage.live.com/y4mrOtvFoyt1br2fA5zhouJX_SZZDsNW_ma8mxas_BI0l3mgIo7ummUC_b1MwR3HPboEREdq3J7ecpd3opaaudminonrenX_yGLEdyZIZKn9iZiE5gTzljQ3NL2qymLC0jweRrqEN6WzQ-mpFHFmQxJHEnEMUO7boWXcCd-BfN7fR-9jRcxxN_RtpyIiO2m_yip?width=1918\u0026height=1017\u0026cropmode=none)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallisterb%2Fgo-ds-motr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fallisterb%2Fgo-ds-motr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallisterb%2Fgo-ds-motr/lists"}