https://github.com/storj/ipfs-go-ds-storj

Storj datastore for IPFS
https://github.com/storj/ipfs-go-ds-storj

Last synced: about 1 month ago
JSON representation

Storj datastore for IPFS

• Host: GitHub
• URL: https://github.com/storj/ipfs-go-ds-storj
• Owner: storj
• License: mit
• Created: 2021-12-02T11:57:38.000Z (about 3 years ago)
• Default Branch: main
• Last Pushed: 2023-06-20T14:59:39.000Z (over 1 year ago)
• Last Synced: 2024-10-08T17:04:14.560Z (2 months ago)
• Language: Go
• Size: 407 KB
• Stars: 15
• Watchers: 2
• Forks: 6
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-storj - Using Storj as IPFS backend

README

        
# Storj Datastore for IPFS

This is an implementation of the IPFS datastore interface backed by Storj.

**NOTE:** Plugins only work on Linux and MacOS at the moment. You can track the progress of this issue here: https://github.com/golang/go/issues/19282

## Bundling

The datastore plugin must be compiled and bundled together with go-ipfs. The plugin will be part of the go-ipfs binary after executing these steps:

```bash

# We use go modules for everything.

> export GO111MODULE=on

# Clone go-ipfs.

> git clone https://github.com/ipfs/kubo

> cd go-ipfs

# Checkout the desired release tag of go-ipfs.

> git checkout v0.19.2

# Pull in the datastore plugin (you can specify a version other than latest if you'd like).

> go get storj.io/ipfs-go-ds-storj/plugin@latest

# Add the plugin to the preload list.

> echo -e "\nstorjds storj.io/ipfs-go-ds-storj/plugin 0" >> plugin/loader/preload_list

# Update the deptree

> go mod tidy

# Now rebuild go-ipfs with the plugin. The go-ipfs binary will be available at cmd/ipfs/ipfs.

> make build

# (Optionally) install go-ipfs. The go-ipfs binary will be copied to ~/go/bin/ipfs.

> make install

```

## Installation

For a brand new ipfs instance (no data stored yet):

1. Bundle the datastore plugin in go-ipfs as described above.

2. Delete `$IPFS_DIR` if it exists.

3. Run `ipfs init --empty-repo` (The `--empty-repo` flag prevents adding doc files to the local IPFS node, which may cause issues before configuring the datastore plugin). 

4. Edit `$IPFS_DIR/config` to include Storj details (see Configuration below).

5. Overwrite `$IPFS_DIR/datastore_spec` as specified below (*Don't do this on an instance with existing data - it will be lost*).

### Configuration

The `config` file should include the following:

```json

{

  "Datastore": {

  ...

    "Spec": {

      "mounts": [

        {

          "child": {

            "type": "storjds",

            "dbURI": "$databaseURI",

            "bucket": "$bucketname",

            "accessGrant": "$accessGrant",

            "packInterval": "$packInterval",

            "debugAddr": "$debugAddr",

            "updateBloomFilter": "$updateBloomFilter",

            "nodeConnectionPoolCapacity": "100",

            "nodeConnectionPoolKeyCapacity": "5",

            "nodeConnectionPoolIdleExpiration": "2m",

            "satelliteConnectionPoolCapacity": "10",

            "satelliteConnectionPoolKeyCapacity": "0",

            "satelliteConnectionPoolIdleExpiration": "2m"

          },

          "mountpoint": "/",

          "prefix": "storj.datastore",

          "type": "measure"

        },

        {

          "child": {

            "compression": "none",

            "path": "providers",

            "type": "levelds"

          },

          "mountpoint": "/providers",

          "prefix": "leveldb.datastore",

          "type": "measure"

        }

      ],

      "type": "mount"

    }

    ...

  }

  ...

}

```

`$databaseURI` is the URI to a Postgres or CockroachDB database installation. This database is used for local caching of blocks before they are packed and uploaded to the Storj bucket. The database must exists. 

`$bucketname` is a bucket on Storj DCS. It must be created.

`$accessGrant` is an access grant for Storj DCS with full permission on the entire `$bucketname` bucket.

`$packInterval` is an optional time duration that sets the packing interval. The default packing interval is 1 minute. If set to a negative duration, e.g. `-1m`, the packing job is disabled.

`$debugAddr` is an optional `[host]:port` address to listen on for the debug endpoints. If not set, the debug endpoints are disabled.

`$updateBloomFilter` is an optional boolean that enables the bloom filter updater. If not set, the updater is disabled. 

`nodeConnectionPoolCapacity` is an optional total number of connections to keep open in the Storj node connection pool. Default: 500.

`nodeConnectionPoolKeyCapacity` is an optonal maximum number of connections to keep open per Storj node in the connection pool. Zero means no limit. Default: 5.

`nodeConnectionPoolIdleExpiration` is an optional duration for how long a connection in the Storj node connection pool is allowed to be kept idle. Zero means no expiraton. Default: 2m.

`satelliteConnectionPoolCapacity` is an optional total number of connections to keep open in the Storj satellite connection pool. Default: 10.

`satelliteConnectionPoolKeyCapacity` is an optonal maximum number of connections to keep open per Storj satellite in the connection pool. Zero means no limit. Default: 0.

`satelliteConnectionPoolIdleExpiration` is an optional duration for how long a connection in the Storj satellite connection pool is allowed to be kept idle. Zero means no expiraton. Default: 2m.

If you are configuring a brand new ipfs instance without any data, you can overwrite the `datastore_spec` file with:

```json

{"mounts":[{"mountpoint":"/providers","path":"providers","type":"levelds"},{"bucket":"$bucketname","mountpoint":"/"}],"type":"mount"}

```

Otherwise, you need to do a datastore migration.

## Run With Docker

```

docker run --rm -d \

    --network host \

    -e STORJ_DATABASE_URL= \

    -e STORJ_BUCKET= \

    -e STORJ_ACCESS= \

    -e IPFS_IDENTITY_PEER_ID= \

    -e IPFS_IDENTITY_PRIVATE_KEY= \

    -e IPFS_GATEWAY_NO_FETCH=true \

    -e IPFS_GATEWAY_DOMAIN= \

    -e IPFS_GATEWAY_USE_SUBDOMAINS=false \

    -e IPFS_GATEWAY_PORT=8080 \

    -e IPFS_API_PORT=5001 \

    -e IPFS_INDEX_PROVIDER_URL=http://index-provider:50617 \

    -e IPFS_BLOOM_FILTER_SIZE=1048576 \

    -e STORJ_UPDATE_BLOOM_FILTER=true \

    -e STORJ_PACK_INTERVAL=5m \

    -e STORJ_DEBUG_ADDR=<[host]:port> \

    -e GOLOG_FILE=/app/log/output.log \

    -e GOLOG_LOG_LEVEL="storjds=info" \

    -e STORJ_NODE_CONNECTION_POOL_CAPACITY=100 \

    -e STORJ_NODE_CONNECTION_POOL_KEY_CAPACITY=5 \

    -e STORJ_NODE_CONNECTION_POOL_IDLE_EXPIRATION=2m \

    -e STORJ_SATELLITE_CONNECTION_POOL_CAPACITY=10 \

    -e STORJ_SATELLITE_CONNECTION_POOL_KEY_CAPACITY=0 \

    -e STORJ_SATELLITE_CONNECTION_POOL_IDLE_EXPIRATION=2m \

    --mount type=bind,source=,destination=/app/log \

    storjlabs/ipfs-go-ds-storj:

```

Docker images are published to https://hub.docker.com/r/storjlabs/ipfs-go-ds-storj.

`STORJ_DATABASE_URL` can be set to a Postgres or CockroachDB database URL.

`STORJ_BUCKET` must be set to an existing bucket.

`STORJ_ACCESS` must be set to an access grant with full permission to `STORJ_BUCKET`.

`IPFS_IDENTITY_PEER_ID` can be set optionally to preserve the node identity between runs. The current peer ID can be found under `Identity.PeerID` in the config file.

`IPFS_IDENTITY_PRIVATE_KEY` must be set if `IPFS_IDENTITY_PEER_ID` is set, and the provided private key must match the peer ID. The current private key can be found under `Identity.PrivKey` in the config file.

`IPFS_GATEWAY_NO_FETCH` determines if the IPFS gateway is open (if set to false) or restricted (if set to true). Restricted gateways serve files only from the local IPFS node. Open gateways search the IPFS network if the file is not present on the local IPFS node.

`IPFS_GATEWAY_DOMAIN` can be set to the domain name assigned to the IPFS gateway. If set, `IPFS_GATEWAY_USE_SUBDOMAINS` determines if to use subdomains resolution style. See https://docs.ipfs.io/concepts/ipfs-gateway/#subdomain for details.

`IPFS_GATEWAY_PORT` can be set to change the IPFS Gateway port from the default 8080.

`IPFS_API_PORT` can be set to change the IPFS HTTP API port from the default 5001.

`IPFS_INDEX_PROVIDER_URL` can be set to the delegated routing URL of an [IPNI Index Provider](https://github.com/storj/ipfs-index-provider).

`IPFS_ADDRESSES_SWARM` can be set to change the IPFS [Addresses.Swarm](https://github.com/ipfs/kubo/blob/master/docs/config.md#addressesswarm) config. It will overwrite the default values.

`IPFS_PEERING_PEERS` can be set to add extra public peers in the IPFS [Peering.Peers](https://github.com/ipfs/kubo/blob/master/docs/config.md#peeringpeers) config.

`IPFS_BLOOM_FILTER_SIZE` sets the size in bytes of the datastore bloom filter. It is recommended to set this on production installations for reducing the number of calls to the database due to incoming requests from the IPFS network. Default value is 0, which means that the bloom filter is disabled. Details in https://github.com/ipfs/kubo/blob/master/docs/config.md#datastorebloomfiltersize.

`STORJ_UPDATE_BLOOM_FILTER` enables the bloom filter updater, which listens to changes in the local database and updates the datastore bloom filter. The default value is false. It should be enabled when running multiple nodes attached to the same datastore. Only CockroachDB is supported. Requires admin privileges for the CockroachDB user.

`STORJ_PACK_INTERVAL` can be set to change the packing interval. The default packing interval is 1 minute. If set to a negative duration, e.g. `-1m`, the packing job is disabled.

`STORJ_DEBUG_ADDR` can be set to a specific `[host]:port` address to listen on for the debug endpoints. If not set, the debug endpoints are disabled.

`GOLOG_FILE` sets the log file location. If not set, the logs are printed to the standard error.

`GOLOG_LOG_LEVEL` sets the log level. The default level is ERROR. Use `storjds=` to set the log level of only the Storj datastore plugin. See https://github.com/ipfs/go-log#golog_log_level for details.

`STORJ_NODE_CONNECTION_POOL_CAPACITY` sets the total number of connections to keep open in the Storj node connection pool. Default: 500.

`STORJ_NODE_CONNECTION_POOL_KEY_CAPACITY` sets the maximum number of connections to keep open per Storj node in the connection pool. Zero means no limit. Default: 5.

`STORJ_NODE_CONNECTION_POOL_IDLE_EXPIRATION` sets how long a connection in the Storj node connection pool is allowed to be kept idle. Zero means no expiraton. Default: 2m.

`STORJ_SATELLITE_CONNECTION_POOL_CAPACITY` sets the total number of connections to keep open in the Storj satellite connection pool. Default: 10.

`STORJ_SATELLITE_CONNECTION_POOL_KEY_CAPACITY` sets the maximum number of connections to keep open per Storj satellite in the connection pool. Zero means no limit. Default: 0.

`STORJ_SATELLITE_CONNECTION_POOL_IDLE_EXPIRATION` sets how long a connection in the Storj satellite connection pool is allowed to be kept idle. Zero means no expiraton. Default: 2m.

## License

MIT