{"id":13731714,"url":"https://github.com/textileio/textile","last_synced_at":"2025-04-12T22:37:34.960Z","repository":{"id":37764408,"uuid":"226931674","full_name":"textileio/textile","owner":"textileio","description":"Textile hub services and buckets lib","archived":false,"fork":false,"pushed_at":"2023-05-05T02:32:42.000Z","size":10355,"stargazers_count":226,"open_issues_count":104,"forks_count":46,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-04T02:09:29.086Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/textileio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-12-09T17:42:59.000Z","updated_at":"2024-11-04T23:59:45.000Z","dependencies_parsed_at":"2024-01-07T21:02:41.568Z","dependency_job_id":"445743f5-38db-47a6-abdd-18344db8d4b5","html_url":"https://github.com/textileio/textile","commit_stats":{"total_commits":276,"total_committers":13,"mean_commits":21.23076923076923,"dds":"0.46376811594202894","last_synced_commit":"3d0748d28671c20f9651fdb4e25bab79f487438c"},"previous_names":[],"tags_count":67,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/textileio%2Ftextile","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/textileio%2Ftextile/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/textileio%2Ftextile/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/textileio%2Ftextile/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/textileio","download_url":"https://codeload.github.com/textileio/textile/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248643009,"owners_count":21138353,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T02:01:36.770Z","updated_at":"2025-04-12T22:37:34.938Z","avatar_url":"https://github.com/textileio.png","language":"Go","funding_links":[],"categories":["Go","others"],"sub_categories":[],"readme":"## DEPRECATION NOTICE: Textile's _hosted_ Hub infrastructure will be taken off-line on January 9th, 2023. At this time, all ThreadDB and Bucket data will no longer be available, and will subsequently be removed. See https://github.com/textileio/textile/issues/578 for further details.\n\n# textile\n\n[![Made by Textile](https://img.shields.io/badge/made%20by-Textile-informational.svg?style=popout-square)](https://textile.io)\n[![Chat on Slack](https://img.shields.io/badge/slack-slack.textile.io-informational.svg?style=popout-square)](https://slack.textile.io)\n[![GitHub license](https://img.shields.io/github/license/textileio/textile.svg?style=popout-square)](./LICENSE)\n[![GitHub action](https://github.com/textileio/textile/workflows/Tests/badge.svg?style=popout-square)](https://github.com/textileio/textile/actions)\n[![standard-readme compliant](https://img.shields.io/badge/readme%20style-standard-brightgreen.svg?style=popout-square)](https://github.com/RichardLitt/standard-readme)\n\n\u003e Textile hub services and buckets lib\n\nTextile connects and extends [Libp2p](https://libp2p.io/), [IPFS](https://ipfs.io/), and [Filecoin](https://filecoin.io/). Three interoperable technologies makeup Textile:\n\n* [**ThreadDB**](https://github.com/textileio/go-threads): A server-less p2p database built on Libp2p\n* [**Powergate**](https://github.com/textileio/powergate): File storage built on Filecoin and IPFS\n* [**Buckets**](https://github.com/textileio/textile/tree/master/buckets): File and dynamic directory storage built on ThreadDB, Powergate, and [UnixFS](https://github.com/ipfs/go-unixfs).\n\nJoin us on our [public Slack channel](https://slack.textile.io/) for news, discussions, and status updates. [Check out our blog](https://medium.com/textileio) for the latest posts and announcements.\n\n## Table of Contents\n\n- [textile](#textile)\n  - [Security](#security)\n  - [Background](#background)\n  - [Install](#install)\n    - [The Hub](#the-hub)\n      - [`hubd`](#hubd)\n      - [`hub`](#hub)\n    - [Buckets](#buckets)\n      - [`buckd`](#buckd)\n      - [`buck`](#buck)\n    - [The Buckets Library](#the-buckets-library)\n  - [Getting Started](#getting-started)\n    - [The Hub](#the-hub-1)\n    - [Running Buckets](#running-buckets)\n    - [Creating a bucket](#creating-a-bucket)\n    - [Creating a private bucket](#creating-a-private-bucket)\n    - [Adding files and folders to a bucket](#adding-files-and-folders-to-a-bucket)\n    - [Recreating an existing bucket](#recreating-an-existing-bucket)\n    - [Creating a bucket from an existing Cid](#creating-a-bucket-from-an-existing-cid)\n    - [Exploring bucket contents](#exploring-bucket-contents)\n    - [Resetting bucket contents](#resetting-bucket-contents)\n    - [Watching a bucket for changes](#watching-a-bucket-for-changes)\n    - [Protecting a file with a password](#protecting-a-file-with-a-password)\n    - [Sharing bucket files and folders](#sharing-bucket-files-and-folders)\n    - [Creating a Filecoin bucket archive](#creating-a-filecoin-bucket-archive)\n    - [Multi-writer buckets](#multi-writer-buckets)\n    - [Deleting a bucket](#deleting-a-bucket)\n    - [Using the Buckets Library](#using-the-buckets-library)\n      - [Creating a bucket](#creating-a-bucket-1)\n      - [Getting an existing bucket](#getting-an-existing-bucket)\n      - [Pushing local files](#pushing-local-files)\n      - [Pulling remote changes](#pulling-remote-changes)\n    - [Using the Mail Library](#using-the-mail-library)\n      - [Creating a mailbox](#creating-a-mailbox)\n      - [Getting an existing mailbox](#getting-an-existing-mailbox)\n      - [Sending a message](#sending-a-message)\n      - [Watching for new messages](#watching-for-new-messages)\n  - [Developing](#developing)\n  - [Contributing](#contributing)\n  - [Changelog](#changelog)\n  - [License](#license)\n\n## Security\n\nTextile is still under heavy development and no part of it should be used before a thorough review of the underlying code and an understanding APIs and protocols may change rapidly. There may be coding mistakes, and the underlying protocols may contain design flaws. Please [let us know](mailto:contact@textile.io) immediately if you have discovered a security vulnerability.\n\nPlease also read the [security note](https://github.com/ipfs/go-ipfs#security-issues) for [go-ipfs](https://github.com/ipfs/go-ipfs).\n\n## Background\n\nGo to [the docs](https://docs.textile.io/) for more about the motivations behind Textile.\n\n## Install\n\nThis repo contains two service daemons with CLIs and a Buckets Library for building local-first apps and services.\n\n### The Hub\n\n#### `hubd`\n\n-   **Prebuilt package**: See [release assets](https://github.com/textileio/textile/releases/latest)\n-   **Docker image**: See the `latest` tag on [Docker Hub](https://hub.docker.com/r/textile/textile/tags)\n-   **Build from the source**:\n\n```\ngit clone https://github.com/textileio/textile\ncd textile\ngo get ./cmd/hubd\n```\n\n#### `hub`\n\n-   **Prebuilt package**: See [release assets](https://github.com/textileio/textile/releases/latest)\n-   **Build from the source**: \n\n```\ngit clone https://github.com/textileio/textile\ncd textile\ngo get ./cmd/hub\n```\n\n**Note**: `hub` _includes_ `buck` as a subcommand: `hub buck`. This is because `hubd` hosts `buckd`, along with other services.\n\n`hub` is built in part on the [gRPC client](https://pkg.go.dev/github.com/textileio/textile/v2/api/hub/client), which can be imported to an existing project:\n\n```\nimport \"github.com/textileio/textile/v2/api/hub/client\"\n```\n\n### Buckets\n\n#### `buckd`\n\n-   **Prebuilt package**: See [release assets](https://github.com/textileio/textile/releases/latest)\n-   **Docker image**: See the `buckets` tag on [Docker Hub](https://hub.docker.com/r/textile/textile/tags)\n-   **Build from the source**: \n\n```\ngit clone https://github.com/textileio/textile\ncd textile\ngo get ./cmd/buckd\n```\n\n#### `buck`\n\n-   **Prebuilt package**: See [release assets](https://github.com/textileio/textile/releases/latest)\n-   **Build from the source**: \n\n```\ngit clone https://github.com/textileio/textile\ncd textile\ngo get ./cmd/buck\n```\n\n`buck` is built in part on the [gRPC client](https://pkg.go.dev/github.com/textileio/textile/v2/api/buckets/client), which can be imported in an existing project:\n\n```\nimport \"github.com/textileio/textile/v2/api/buckets/client\"\n```\n\n### The Buckets Library\n\n```\nimport \"github.com/textileio/textile/v2/buckets/local\"\n```\n\nThe full spec is available [here](https://pkg.go.dev/github.com/textileio/textile/v2/buckets/local).\n\n## Getting Started\n\n### The Hub\n\nThe Hub daemon (`hubd`), a.k.a. _The Hub_, is a hosted wrapper around other Textile services that includes developer accounts for individuals and organizations. You are encouraged to run your own, and we strongly discourage the use of the hosted [Textile Hub](https://docs.textile.io/hub/) as it will soon be shutting down.\n\nThe layout of the `hub` client CLI mirrors the services wrapped by `hubd`:\n\n-   `hub threads` provides limited access to ThreadDB.\n-   `hub buck` provides access to Buckets (`buckd`) by wrapping the standalone `buck` CLI.\n-   `hub buck archive` provides limited access to The Hub's hosted Powergate instance, and the Filecoin network.\n\nTry `hub --help` for more usage.\n\n```\nThe Hub Client.\n\nUsage:\n  hub [command]\n\nAvailable Commands:\n  billing     Billing management\n  buck        Manage an object storage bucket\n  destroy     Destroy your account\n  fil         Interact with Filecoin related commands.\n  help        Help about any command\n  init        Initialize account\n  keys        API key management\n  login       Login\n  logout      Logout\n  orgs        Org management\n  threads     Thread management\n  update      Update the hub CLI\n  version     Show current version\n  whoami      Show current user\n\nFlags:\n      --api string        API target (default \"api.hub.textile.io:443\")\n  -h, --help              help for hub\n      --identity string   User identity\n      --key string        User API key\n      --newIdentity       Generate a new user identity\n  -o, --org string        Org username\n      --secret string     User API secret\n  -s, --session string    User session token\n      --token string      User identity token\n\nUse \"hub [command] --help\" for more information about a command.\n```\n\nRead more about The Hub, including how to [create an account](https://docs.textile.io/hub/accounts/#account-setup), in the [docs](https://docs.textile.io/hub/).\n\n### Running Buckets\n\nMuch like [`threadsd`](https://github.com/textileio/go-threads/tree/master/threadsd), the `buckd` daemon can be run as a server or alongside desktop apps or command-line tools. The easiest way to run `buckd` is by using the provided Docker Compose files. If you're new to Docker and/or Docker Compose, get started [here](https://docs.docker.com/compose/gettingstarted/). Once you are setup, you should have `docker-compose` in your `PATH`.\n\nCreate an `.env` file and add the following values:  \n\n```\nREPO_PATH=~/myrepo\nBUCK_LOG_DEBUG=true\n```\n\nCopy [this compose file](https://github.com/textileio/textile/blob/master/cmd/buckd/docker-compose.yml) and run it with the following command.\n\n```\ndocker-compose -f docker-compose.yml up\n```\n\nCongrats! Now you have Buckets running locally.\n\nThe Docker Compose file starts an IPFS node, which is used to pin bucket files and folders. You could point `buckd` to a different (possibly remote) IPFS node by setting the `BUCK_ADDR_IPFS_API` variable to a different multiaddress.  \n\nBy default, this approach does not start [Powergate](https://github.com/textileio/powergate). If you do, be sure to set the `BUCK_ADDR_POWERGATE_API` variable to the multiaddress of your Powergate. `buckd` must be configured with Powergate to enable Filecoin archiving with `buck archive`.\n\n### Creating a bucket\n\nSince `hub buck` and `buck` are functionally identical, this section will focus on `buck` and the Buckets Library using a locally running `buckd`.\n\nFirst off, take a look at `buck --help`.\n\n```\nThe Bucket Client.\n\nManages files and folders in an object storage bucket.\n\nUsage:\n  buck [command]\n\nAvailable Commands:\n  add         Add a UnixFs DAG locally at path\n  archive     Create a Filecoin archive\n  cat         Cat bucket objects at path\n  decrypt     Decrypt bucket objects at path with password\n  destroy     Destroy bucket and all objects\n  encrypt     Encrypt file with a password\n  help        Help about any command\n  init        Initialize a new or existing bucket\n  links       Show links to where this bucket can be accessed\n  ls          List top-level or nested bucket objects\n  pull        Pull bucket object changes\n  push        Push bucket object changes\n  root        Show bucket root CIDs\n  status      Show bucket object changes\n  watch       Watch auto-pushes local changes to the remote\n\nFlags:\n      --api string   API target (default \"127.0.0.1:3006\")\n  -h, --help         help for buck\n\nUse \"buck [command] --help\" for more information about a command.\n```\n\nA Textile bucket functions a bit like an S3 bucket. It's a virtual filesystem where you can push, pull, list, and cat files. You can share them via web links or render the whole thing as a website or web app. They also function a bit like a Git repository. The point of entry is from a folder on your local machine that is synced to a _remote_.\n\nTo get started, initialize a new bucket.\n\n```\nmkdir mybucket \u0026\u0026 cd mybucket\nbuck init\n```\n\nWhen prompted, give your bucket a name and either opt-in or decline bucket encyption (see [Creating a private bucket](#creating-a-private-bucket) for more about bucket encryption).\n\nYou should now see two links for the new bucket on the locally running gateway.\n\n```\n\u003e http://127.0.0.1:8006/thread/bafkq3ocmdkrljadlgybtvocytpdw4hbnzygxecxehdp7pfj32lxp34a/buckets/bafzbeifyzfm3kosie25s5qthvvcjrr42ivd7doqhwvu5m4ks7uqv4j5lyi Thread link\n\u003e http://127.0.0.1:8006/ipns/bafzbeifyzfm3kosie25s5qthvvcjrr42ivd7doqhwvu5m4ks7uqv4j5lyi IPNS link (propagation can be slow)\n\u003e Success! Initialized /path/to/mybucket as a new empty bucket\n```\n\nThe first URL is the link to the ThreadDB instance. Internally, a collection named `buckets` is created. Each new instance in this collection amounts to a new bucket. However, when you visit this link, you'll notice a custom file browser. This is because the gateway considers the built-in `buckets` collection a special case. You can still view the raw ThreadDB instance by appending `?json=true` to the URL.\n\nThe second URL is the bucket's unique IPNS address, which is auto-updated when you add, modify, or delete files.\n\nIf you have configured the daemon with DNS settings, you will see a third URL that links to the bucket's WWW address, where it is rendered as a static website / client-side application. See `buckd --help` for more info.\n\n**Important**: If your bucket is private (encrypted), an access token (JWT) will be appended to these links. This token represents your _identity_ across ***all buckets*** and should not be shared without caution.\n\n`buck init` created a configuration folder in `mybucket` called `.textile`. This folder is somewhat like a `.git` folder, as it contains information about the bucket's remote address and local state.\n\n`.textile/config.yml` will look something like,\n\n```\nkey: bafzbeifyzfm3kosie25s5qthvvcjrr42ivd7doqhwvu5m4ks7uqv4j5lyi\nthread: bafkq3ocmdkrljadlgybtvocytpdw4hbnzygxecxehdp7pfj32lxp34a\n```\n\nWhere `key` is the bucket's unique key, and `thread` is it's ThreadDB ID.\n\nAdditionally, `.textile/repo` contains a repository describing the current file structure, which is used to stage changes against the remote.\n\n### Creating a private bucket\n\nBucket encryption (AES-CTR + AES-512 HMAC) happens entirely within the `buckd`, meaning your data gets encrypted on the way in, and decrypted on the way out. This type of encryption has two goals:\n\n- Obfuscate bucket data / files (the normal goal of encryption)\n- Obfuscate directory structure, which amounts to encrypting [IPLD](https://ipld.io/) nodes and their links.\n\nAs a result of these goals, we refer to encrypted buckets as _private buckets_. Read more about bucket encryption [here](https://docs.textile.io/buckets/#encryption).\n\nTo create a new private bucket, use the `--private` flag with `buck` init or respond `y` when prompted.\n\nIn addition to bucket-level encryption, you can also [protect a file with a password](#protecting-a-file-with-a-password).\n\n### Adding files and folders to a bucket\n\nBucket files and folders are content-addressed by Cids. Check out [the spec](https://github.com/multiformats/cid) if you're unfamiliar with Cids.\n\n`buck` stages new files as additions:\n\n```\necho \"hello world\" \u003e hello.txt\nbuck status\n\u003e new file:  hello.txt\n```\n\n`buck status` is powered by DAG-based diffing. Much like `git`, this allows buck to only push and pull _changes_. Read more about bucket diffing in the [docs](https://docs.textile.io/buckets/#diffing-and-synching), or check out this [in-depth blog post](https://blog.textile.io/buckets-diffing-syncing-archiving/).\n\nUse `push` to sync the change.\n\n```\nbuck push\n+ hello.txt: bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4\n\u003e bafybeihm4zrnrsdroazwsvk3i65ooqzdftaugdkjiedr6ocq65u3ap4wni\n```\n\nThe output shows the Cid of the added file and the bucket's new root Cid.\n\n`push` will sync all types of file changes: _Additions, modifications, and deletions_.\n\n### Recreating an existing bucket\n\nIt's often useful to recreate a bucket from the remote. This is somewhat like re-cloning a Git repo. This can be done in a different location on the same machine, or, if `buckd` has a public IP address, from a completely different machine.\n\nLet's recreate the bucket from the previous step in a new directory outside of the original bucket.\n\n```\nmkdir mybucket2 \u0026\u0026 cd mybucket2\nbuck init --existing\n```\n\nThe `--existing` flag allows for interactively selecting an existing bucket to initialize from.\n\n```\n? Which exiting bucket do you want to init from?:\n  ▸ MyBucket bafzbeifyzfm3kosie25s5qthvvcjrr42ivd7doqhwvu5m4ks7uqv4j5lyi\n```\n\nAt this point, there's only one bucket to choose from. \n\n**Note**: If `buckd` was running inside The Hub (`hubd`), you would be able to choose from buckets belonging to your Organizations and well as your individual Developer account by using the `--org` flag. Read more about Hub Accounts and Organizations [here](https://docs.textile.io/hub/accounts/).\n\n```\n\u003e Selected bucket MyBucket\n+ hello.txt: bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4\n+ .textileseed: bafkreifbdzttoqsch5j66hfmcbsic6qvwrikibgzfbg3tn7rc3j63ukk3u\n\u003e Your bucket links:\n\u003e http://127.0.0.1:8006/thread/bafkq3ocmdkrljadlgybtvocytpdw4hbnzygxecxehdp7pfj32lxp34a/buckets/bafzbeifyzfm3kosie25s5qthvvcjrr42ivd7doqhwvu5m4ks7uqv4j5lyi Thread link\n\u003e http://127.0.0.1:8006/ipns/bafzbeifyzfm3kosie25s5qthvvcjrr42ivd7doqhwvu5m4ks7uqv4j5lyi IPNS link (propagation can be slow)\n\u003e Success! Initialized /path/to/mybucket2 from an existing bucket\n```\n\nJust as before, the output shows the bucket's remote links. However, in this case `init` also pulled down the content.\n\n**Note**: `.textileseed` is used to randomize a bucket's top level Cid and cannot be modified.\n\nThe `--existing` flag is really just a helper that sets the `--thread` and `--key` flags, which match the config values we saw earlier. We could have used those flags directly to achieve the same result.\n\n```\nbuck init --thread bafkq3ocmdkrljadlgybtvocytpdw4hbnzygxecxehdp7pfj32lxp34a --key bafzbeifyzfm3kosie25s5qthvvcjrr42ivd7doqhwvu5m4ks7uqv4j5lyi\n```\n\nLastly, we could have just copied `.textile/config.yml` to a new directory and used `buck pull` to pull down the existing content.\n\n### Creating a bucket from an existing Cid\n\nSometimes it's useful to create a bucket from a [UnixFS](https://github.com/ipfs/go-unixfs) directory that is already on the IPFS network.\n\nWe can simulate this scenario by adding a local folder to IPFS and then using its root Cid to create a bucket with the `--cid` flag. Here's a local directory.\n\n```\n.\n├── a\n│   ├── bar.txt\n│   ├── foo.txt\n│   └── one\n│       ├── baz.txt\n│       ├── buz.txt\n│       └── two\n│           ├── boo.txt\n│           └── fuz.txt\n├── b\n│   ├── foo.txt\n│   └── one\n│       ├── baz.txt\n│       ├── muz.txt\n│       ├── three\n│       │   └── far.txt\n│       └── two\n│           └── fuz.txt\n└── c\n    ├── one.jpg\n    └── two.jpg\n```\n\nUse the recursvie flag `-r` with `ipfs add`.\n\n```\nipfs add -r .\nadded QmcDkcMJXZsNnExehsE1Yh6SRWucHa9ruVT82gpL83431W mydir/a/bar.txt\nadded QmYiUq2U6euWnKag23wFppG12hon4EBDswdoe4MwrKzDBn mydir/a/foo.txt\nadded QmXrd35ja3kknnmgj5kyDM74jfG8GLJJQGtRpEQpXCLTR3 mydir/a/one/baz.txt\nadded QmSWJvCzotB3CbdxVu8mBvmLqpSuEQgUoJHTFy1azRfwhT mydir/a/one/buz.txt\nadded QmT6h1eaBV74Sh75upE7ugFLkBnmyGr3WsQ8w8yx5NjgPV mydir/a/one/two/boo.txt\nadded QmTdg1b5eWEx4zJtrgvew1inkkZ29fp9mbQ4uHyKurW8Ub mydir/a/one/two/fuz.txt\nadded QmYiQAk1seXrmuQkpGE83AxJyNZDK1RNSaLyp3Z4r1zsrB mydir/b/foo.txt\nadded QmXrd35ja3kknnmgj5kyDM74jfG8GLJJQGtRpEQpXCLTR3 mydir/b/one/baz.txt\nadded QmSWJvCzotB3CbdxVu8mBvmLqpSuEQgUoJHTFy1azRfwhT mydir/b/one/muz.txt\nadded QmYs12A3CGSTHX4QrsvBe2AvLHEThrapXoTFQpyh8AzpFa mydir/b/one/three/far.txt\nadded QmTdg1b5eWEx4zJtrgvew1inkkZ29fp9mbQ4uHyKurW8Ub mydir/b/one/two/fuz.txt\nadded QmaLpwNPwftSQY3w4ZtMfZ8k38D5EgK2bcDuU4UwzREJpi mydir/c/one.jpg\nadded QmYLiWv2WXQd1m8YyHx4dMoj8B3Kuiuu7pCCoYibkqKyVj mydir/c/two.jpg\nadded QmT5YXeCfbMuVjanbHjQhECUQSACJLecfmjRBZHvmu5FDU mydir/a/one/two\nadded QmWh2Wx9Lec4wbEvFbsq4HmYjFmgUFtxNJ8wEVwXjhJ2uk mydir/a/one\nadded QmSujVHvG8Y3Jv21AbMFNQPphjyqNamh6cvdyXSD1jAtSZ mydir/a\nadded QmUGSorWDy2JiKYvQuJzEb4TnYDuDNLcdFyR6NhMwnwdvy mydir/b/one/three\nadded QmWvX7UVexbjXJtxKMyMSgGpPesFQD7teNTqUcDsP2mzW6 mydir/b/one/two\nadded QmPyMD67EgSZS1WpvgudHkxbA5zgjqmse8srPpFb9sVefT mydir/b/one\nadded QmQdAtg5NkwkvLtTbka3eci58UGj3m9AehC2sbksGSbjPZ mydir/b\nadded QmcjtVAF9PQfMKTc57vcvZeBrzww3TLxPcQfUQW7cXXLJL mydir/c\nadded QmcvkGF2t8Z94UqhdtdFRokGoqypbGyKkzRPVF4owmjVrE mydir\n```\n\nAfter adding the entire directory, we see the root Cid is `QmcvkGF2t8Z94UqhdtdFRokGoqypbGyKkzRPVF4owmjVrE`. Let's create the bucket using this Cid.\n\n```\nbuck init --cid QmcvkGF2t8Z94UqhdtdFRokGoqypbGyKkzRPVF4owmjVrE\n```\n\nThe files behind the Cid will be pulled into the new bucket.\n\n```\n+ a/bar.txt: QmcDkcMJXZsNnExehsE1Yh6SRWucHa9ruVT82gpL83431W\n+ a/foo.txt: QmYiUq2U6euWnKag23wFppG12hon4EBDswdoe4MwrKzDBn\n+ a/one/two/fuz.txt: QmTdg1b5eWEx4zJtrgvew1inkkZ29fp9mbQ4uHyKurW8Ub\n+ a/one/baz.txt: QmXrd35ja3kknnmgj5kyDM74jfG8GLJJQGtRpEQpXCLTR3\n+ c/two.jpg: QmYLiWv2WXQd1m8YyHx4dMoj8B3Kuiuu7pCCoYibkqKyVj\n+ b/foo.txt: QmYiQAk1seXrmuQkpGE83AxJyNZDK1RNSaLyp3Z4r1zsrB\n+ a/one/buz.txt: QmSWJvCzotB3CbdxVu8mBvmLqpSuEQgUoJHTFy1azRfwhT\n+ a/one/two/boo.txt: QmT6h1eaBV74Sh75upE7ugFLkBnmyGr3WsQ8w8yx5NjgPV\n+ b/one/muz.txt: QmSWJvCzotB3CbdxVu8mBvmLqpSuEQgUoJHTFy1azRfwhT\n+ b/one/three/far.txt: QmYs12A3CGSTHX4QrsvBe2AvLHEThrapXoTFQpyh8AzpFa\n+ b/one/baz.txt: QmXrd35ja3kknnmgj5kyDM74jfG8GLJJQGtRpEQpXCLTR3\n+ b/one/two/fuz.txt: QmTdg1b5eWEx4zJtrgvew1inkkZ29fp9mbQ4uHyKurW8Ub\n+ c/one.jpg: QmaLpwNPwftSQY3w4ZtMfZ8k38D5EgK2bcDuU4UwzREJpi\n\u003e Your bucket links:\n\u003e http://127.0.0.1:8006/thread/bafk3k3itq2rsybcvhf6wuvumruw3j6cw7ixhrtx4ek45qgvp3e7u2xa/buckets/bafzbeiawo6ghgsqjlorii4wghdl4tzz54x2kiwtcgtaq7b3h5gta2yok2i Thread link\n\u003e http://127.0.0.1:8006/ipns/bafzbeiawo6ghgsqjlorii4wghdl4tzz54x2kiwtcgtaq7b3h5gta2yok2i IPNS link (propagation can be slow)\n\u003e Success! Initialized /path/to/mybucket3 as a new bootstrapped bucket\n```\n\nCurrently, UnixFS in `go-ipfs` uses Cid version 0, which is why we see all these old-style Cids started with `Qm`. Of course, you can also use UnixFS directories that use Cid version 1.\n\nSimilar to initializing a new bucket from an existing Cid, `buck add` allows you to _add_ and/or _merge in_ an existing UnixFS directory to an _existing bucket_. Like adding new files locally, this works by pulling down the UnixFS content from the IPFS network into the local bucket. Sync the changes with `buck push` as normal.\n\nPulling an existing UnixFS directory into a new or existing private bucket is also possible. Just opt-in to encryption during initialization as normal. `buckd` will recursively encrypt (without duplicating) the Cid's IPLD file and directory nodes as they are pulled into the new bucket.\n\n### Exploring bucket contents\n\nUse `buck ls [path]` to explore bucket contents. Omitting `[path]` will list the top-level directory.\n\n```\nbuck ls\n\n  NAME          SIZE     DIR    OBJECTS  CID\n  .textileseed  32       false  n/a      bafkreiezexkrnk7yew6glm6sulhur66bbecc2aeaitf7uz4ymmp442lepu\n  a             3726     true   3        QmSujVHvG8Y3Jv21AbMFNQPphjyqNamh6cvdyXSD1jAtSZ\n  b             3191     true   2        QmQdAtg5NkwkvLtTbka3eci58UGj3m9AehC2sbksGSbjPZ\n  c             1537626  true   2        QmcjtVAF9PQfMKTc57vcvZeBrzww3TLxPcQfUQW7cXXLJL\n```\n\nUse `[path]` to drill into directories, e.g.,\n\n```\nbuck ls a\n\n  NAME     SIZE  DIR    OBJECTS  CID\n  bar.txt  517   false  n/a      QmcDkcMJXZsNnExehsE1Yh6SRWucHa9ruVT82gpL83431W\n  foo.txt  557   false  n/a      QmYiUq2U6euWnKag23wFppG12hon4EBDswdoe4MwrKzDBn\n  one      2502  true   3        QmWh2Wx9Lec4wbEvFbsq4HmYjFmgUFtxNJ8wEVwXjhJ2uk\n```\n\n`buck cat` functions a lot like `ls`, but cats file contents to stdout.\n\n### Resetting bucket contents\n\nSimilar to a `git reset --hard`, you can use `buck pull --hard` to discard local changes that have not been pushed.\n\nContinuing with the bucket above, add, modify, and/or delete some files. `buck status` should show your staged changes.\n\n```\nbuck status\n\u003e modified:  a/bar.txt\n\u003e deleted:   a/one/baz.txt\n\u003e new file:  b/one/three/car.txt\n\u003e deleted:   b/foo.txt\n```\n\nNormally, `buck pull` will move your local changes to temporary `.buckpatch` files, apply the remote / upstream changes, then reapply your local changes. However, the `--hard` flag will prune all local changes, resetting the local bucket contents to match the remote exactly.\n\n```\nbuck pull --hard\n+ a/one/baz.txt: QmXrd35ja3kknnmgj5kyDM74jfG8GLJJQGtRpEQpXCLTR3\n+ b/foo.txt: QmYiQAk1seXrmuQkpGE83AxJyNZDK1RNSaLyp3Z4r1zsrB\n+ a/bar.txt: QmcDkcMJXZsNnExehsE1Yh6SRWucHa9ruVT82gpL83431W\n- b/one/three/car.txt\n\u003e QmTz6HoC18QQqAEtYhfLc4Fse3LPbSCKV8vouvE88MKjFj\n```\n\nNow `buck status` will report `\u003e Everything up-to-date`.\n\nTry `buck pull --help` for more options when pulling the remote.\n\n### Watching a bucket for changes\n\nSo far we've seen how a bucket can change locally, but the remote can also change. This could happen for a couple reasons:\n\n* Changes are pushed from a different bucket copy against the same `buckd`.\n* Changes are pushed from a different `buckd` at the ThreadDB layer. This is known as a multi-writer scenario. See [Multi-writer buckets](#multi-writer-buckets) for more.\n\nIn either case, it is possible to listen for and apply the remote changes using `buck watch`. This will also watch for local changes and auto-push them to the remote. In this way, multiple copies of the same bucket can be kept in sync.\n\n`watch` will block until it's cancelled with a Ctrl-C.\n\n```\nbuck watch\n\u003e Success! Watching /path/to/mybucket for changes...\n```\n\n`watch` will survive network interruptions, reconnecting when possible.\n\n```\n\u003e Not connected. Trying to connect...\n\u003e Not connected. Trying to connect...\n\u003e Not connected. Trying to connect...\n\u003e Success! Watching /path/to/mybucket for changes...\n```\n\nWhile `watch` is active, file and folders dropped into the bucket will be automatically pushed.\n\n### Protecting a file with a password\n\nPrivate buckets handle encryption entirely within `buckd`, but you can use an additional client-side encryption layer with `buck encrypt` to password protect files. This encryption is also AES-CTR + AES-512 HMAC, which means you can efficiently encrypt large file streams. However, unlike bucket-wide encryption in private buckets, client-side encryption is only available for files, not IPLD directory nodes.\n\nLet's create an encrypted version of the `hello.txt` file.\n\n```\nbuck encrypt hello.txt supersecret \u003e secret.txt\n```\n\n`encrypt` writes to stdout. So, here we redirect the output to a new file called `secret.txt`. [scrypt](https://pkg.go.dev/golang.org/x/crypto/scrypt?tab=doc) is used to derive the AES and HMAC keys from a password. This carries the normal tradeoff: _The encryption is only as good as the password_. Also, as with all client-side encryption, you must also store or otherwise remember the password!\n\n`encrypt` only works on local files. You'll have to use `push` to sync the new file to the remote.\n\n```\nbuck push --yes\n+ secret.txt: bafkreiayymufgaut3wrfbzfdxiacxn64mxijj54g2osyk7qnco54iftovi\n\u003e bafybeidhffwg5ucwktn7iwyvnkhxpz7b2yrh643bo74cjvsbquzpdgpcd4\n```\n\n`decrypt`, on the other hand, works on remote files. So, after pushing `secret.txt`, we can decrypt it (if we can remember the password) and write the plaintext to stdout.\n\n```\nbuck decrypt secret.txt supersecret\nhello world\n```\n\nLooks like it worked!\n\n### Sharing bucket files and folders\n\nBucket contents can be shared with other Hub accounts and users using the `buck roles` command. Each file and folder in a bucket maintains a set of public-key based access roles: `None`, `Reader`, `Writer`, and `Admin`. Only the `Admin` role can add and remove files and folders from a shared path. See `hub buck roles grant --help` for more about each role. For most applications, access roles only makes sense in the context of the Hub.\n\nBy default, public buckets have two roles located at the top-level path:\n\n```\nhub buck roles ls\n\n  IDENTITY                                                     ROLE\n  *                                                            Reader\n  bbaareibzpb44ahd7oieqevvlqajidd4jajcvx2vdvti6bpw5wkqolwwerm  Admin\n\n\u003e Found 2 access roles\n```\n\nSince access roles are inherited down a bucket path, the single admin role grants the owner full access to all current and future files and folders. The default (`*`) `Read` role indicates that the entire bucket is open to the world. This is merely a reflection of the fact that the underlying UnixFS directory behind public (non-encrypted) buckets are discoverable on the IPFS Network.\n\nPrivate buckets are not open to the world and are created with only the single admin role. However, we can still grant default (`*`) `Read` access to individual files, folders, or the entire bucket posteriori.\n\n```\nhub buck roles grant \"*\" myfolder\nUse the arrow keys to navigate: ↓ ↑ → ←\n? Select a role:\n  None\n  ▸ Reader\n  Writer\n  Admin\n```\n\nWe can now see a new role added to `myfolder`.\n\n```\n hub buck roles ls myfolder\n\n  IDENTITY  ROLE\n  *         Reader\n\n\u003e Found 1 access roles\n```\n\nSimilarly, grant the `None` role to revoke access.\n\nManipulating access roles for a single Hub account or user (public key) can be cumbersome with the `buck` CLI. Applications in need of this level of granular access control should do so programmatically using the [Go client](https://pkg.go.dev/github.com/textileio/textile/v2/api/buckets/client), [JavaScript client](https://textileio.github.io/js-hub/docs/hub.buckets).\n\n### Creating a Filecoin bucket archive\n\nBucket archiving requires a Powergate to be running in `buckd`. If you're curious how to do this, take a look at [this Docker Compose file](https://github.com/textileio/textile/blob/master/integrationtest/pg/docker-compose.yml).\n\nLet's try archiving the bucket from the [Creating a bucket](#creating-a-bucket) section.\n\n```\nbuck archive\n\u003e Warning! Archives are Filecoin Mainnet. Use with caution.\n? Proceed? [y/N]\n```\n\nPlease take note of the warning. Archiving should be considered experimental since Filecoin `mainnet` has not yet launched, and Powergate will either be running a `localnet` or `mainnet`.\n\nYou should see a success message if you proceed.\n\n```\n\u003e Success! Archive queued successfully\n```\n\nThis means that archiving has been initiated. It may take some time to complete...\n\n```\nbuck archive status\n\u003e Archive is currently executing, grab a coffee and be patient...\n```\n\nUse the `archive status` command with `-w` to watch the progress of your archive as it moves through the Filecoin market deal stages.\n\n```\nbuck archive status -w\n\u003e Archive is currently executing, grab a coffee and be patient...\n\u003e    Pushing new configuration...\n\u003e    Configuration saved successfully\n\u003e    Executing job 1006707f-efa8-48c2-98af-a1b320a59780...\n\u003e    Ensuring Hot-Storage satisfies the configuration...\n\u003e    No actions needed in Hot Storage.\n\u003e    Hot-Storage execution ran successfully.\n\u003e    Ensuring Cold-Storage satisfies the configuration...\n\u003e    Current replication factor is lower than desired, making 10 new deals...\n\u003e    Calculating piece size...\n\u003e    Estimated piece size is 256 bytes.\n\u003e    Proposing deal to miner t01459 with 0 fil per epoch...\n\u003e    Proposing deal to miner t0117734 with 500000000 fil per epoch...\n\u003e    Proposing deal to miner t0120993 with 500000000 fil per epoch...\n\u003e    Proposing deal to miner t0120642 with 500000000 fil per epoch...\n\u003e    Proposing deal to miner t0121477 with 500000000 fil per epoch...\n\u003e    Proposing deal to miner t0119390 with 500000000 fil per epoch...\n\u003e    Proposing deal to miner t0101180 with 10000000 fil per epoch...\n\u003e    Proposing deal to miner t0117803 with 500000000 fil per epoch...\n\u003e    Proposing deal to miner t0121852 with 500000000 fil per epoch...\n\u003e    Proposing deal to miner t0119822 with 500000000 fil per epoch...\n\u003e    Watching deals unfold...\n\u003e    Deal with miner t0117803 changed state to StorageDealClientFunding\n\u003e    Deal with miner t0121852 changed state to StorageDealClientFunding\n\u003e    Deal with miner t0121477 changed state to StorageDealClientFunding\n\u003e    Deal with miner t0101180 changed state to StorageDealClientFunding\n\u003e    Deal with miner t0119822 changed state to StorageDealClientFunding\n\u003e    Deal with miner t0119390 changed state to StorageDealClientFunding\n\u003e    Deal with miner t0120642 changed state to StorageDealClientFunding\n\u003e    Deal with miner t0117734 changed state to StorageDealClientFunding\n\u003e    Deal with miner t01459 changed state to StorageDealClientFunding\n\u003e    Deal with miner t0120993 changed state to StorageDealClientFunding\n\u003e    Deal with miner t0121477 changed state to StorageDealWaitingForDataRequest\n\u003e    Deal with miner t0119822 changed state to StorageDealWaitingForDataRequest\n\u003e    Deal with miner t0117734 changed state to StorageDealWaitingForDataRequest\n\u003e    Deal with miner t0121852 changed state to StorageDealWaitingForDataRequest\n\u003e    Deal with miner t01459 changed state to StorageDealWaitingForDataRequest\n\u003e    Deal with miner t0120642 changed state to StorageDealWaitingForDataRequest\n\u003e    Deal with miner t0120993 changed state to StorageDealWaitingForDataRequest\n\u003e    Deal with miner t0117803 changed state to StorageDealWaitingForDataRequest\n\u003e    Deal with miner t0101180 changed state to StorageDealWaitingForDataRequest\n\u003e    Deal with miner t0119390 changed state to StorageDealWaitingForDataRequest\n\u003e    Deal with miner t01459 changed state to StorageDealProposalAccepted\n\u003e    Deal with miner t01459 changed state to StorageDealSealing\n```\n\nThe output will look something like the above. With a little luck, you will start seeing some successful storage deals.\n\nBucket archiving allows you to leverage the purely decentralized nature of Filecoin in your buckets. Check out [this video](https://www.youtube.com/watch?v=jiBUxIi1zko\u0026feature=emb_title) from a [blog post](https://blog.textile.io/buckets-diffing-syncing-archiving/) demonstrating Filecoin bucket recovery using the [Lotus client](https://github.com/filecoin-project/lotus).\n\n### Multi-writer buckets\n\nMulti-writer buckets leverage the distributed nature of ThreadDB by allowing multiple identities to write to the same bucket hosted by different Libp2p hosts. Since buckets are ThreadDB collection _instances_, this is no different than normal ThreadDB peer collaboration.\n\nTo-do: Demonstrate joining a bucket from a ThreadDB invite.\n\n### Deleting a bucket\n\nDeleting a bucket is easy... and permanent! `buck destroy` will delete your local bucket as well as the remote, making it unrecoverable with `buck init --existing`.\n\n### Using the Buckets Library\n\nThe `buckets/local` library powers both the `buck` and `hub buck` CLIs. Everything possible in `buck`, from bucket diffing, pushing, pulling, watching, archiving, etc., is available to you in existing projects by importing the Buckets Library.\n\n```\ngo get github.com/textileio/textile/v2/buckets/local\n```\n\nVisit the [GoDoc](https://pkg.go.dev/github.com/textileio/textile/v2/buckets/local) for a complete list of methods and more usage descriptions.\n\n#### Creating a bucket\n\nCreate a new bucket by constructing a configuration object. Only `Path` is required.\n\n```\n// Setup the buckets lib\nbuckets := local.NewBuckets(cmd.NewClients(\"api.textile.io:443\", false), local.DefaultConfConfig())\n\n// Create a new bucket with config\nmybuck, err := buckets.NewBucket(context.Background(), local.Config{\n    Path: \"path/to/bucket/folder\"\n})\n\n// Check current status\ndiff, err := mybuck.DiffLocal() // diff contains staged changes\n```\n\n`buckets.NewBucket` will write a local config file and data repo.\n\nSee `local.WithName`, `local.WithStrategy`, `local.WithPrivate`, `local.WithCid`, `local.WithInitPathEvents` for more options when creating buckets.\n\nTo create a bucket from an existing remote, use its thread ID and instance ID (bucket `key`) in the config.\n\n#### Getting an existing bucket\n\n`GetLocalBucket` returns the bucket at path.\n\n```\nmybuck, err := buckets.GetLocalBucket(context.Background(), \"path/to/bucket/folder\")\n```\n\n#### Pushing local files\n\n`PushLocal` pushes all staged changes to the remote and returns the new local and remote root Cids. These roots will only be different if the bucket is private (the remote is encrypted).\n\n```\nnewRoots, err := mybuck.PushLocal()\n```\n\nSee `local.PathOption` for more options when pushing.\n\n#### Pulling remote changes\n\n`PullRemote` pulls all remote changes locally and returns the new root Cids.\n\n```\nnewRoots, err := mybuck.PullRemote()\n```\n\nSee `local.PathOption` for more options when pulling.\n\n### Using the Mail Library\n\nThe `mail/local` library provides mechanisms for sending and receiving messages between Hub users. Mailboxes are built on ThreadDB.\n\n```\ngo get github.com/textileio/textile/v2/mail/local\n```\n\nVisit the [GoDoc](https://pkg.go.dev/github.com/textileio/textile/v2/mail/local) for a complete list of methods and more usage descriptions.\n\n#### Creating a mailbox\n\nLike creating a bucket, create a new mailbox by constructing a configuration object. All fields are required.\n\n```\n// Setup the mail lib\nmail := local.NewMail(cmd.NewClients(\"api.textile.io:443\", true), local.DefaultConfConfig())\n\n// Create a libp2p identity (this can be any thread.Identity)\nprivKey, _, err := crypto.GenerateEd25519Key(rand.Reader)\nid := thread.NewLibp2pIdentity(privKey)\n\n// Create a new mailbox with config\nmailbox, err := mail.NewMailbox(context.Background(), local.Config{\n    Path: \"path/to/mail/folder\", // Usually a global location like ~/.textile/mail\n    Identity: id,\n    APIKey: \u003cAPI_SECRET\u003e,\n    APISecret: \u003cAPI_KEY\u003e,\n})\n```\n\n`APIKey` and `APISecret` are User Group API Keys. Read more about [creating API Keys](https://docs.textile.io/hub/app-apis/#creating-user-group-keys).\n\nTo recreate a user's mailbox, specify the same identity and API Key in the config.\n\n#### Getting an existing mailbox\n\n`GetLocalMailbox` returns the mailbox at path.\n\n```\nmailbox, err := mail.GetLocalMailbox(context.Background(), \"path/to/mailbox/folder\")\n```\n\n#### Sending a message\n\nWhen a mailbox sends a message to another mailbox, the message is encrypted for the recipient's inbox _and_ for the senders sentbox. This allows both parties to control the message's lifecycle.\n\n```\n// Create two mailboxes (for most applications, this would not happen on the same machine)\nbox1, err := mail.NewMailbox(context.Background(), local.Config{...})\nbox2, err := mail.NewMailbox(context.Background(), local.Config{...})\n\n// Send a message from the first mailbox to the second\nmessage, err := box1.SendMessage(context.Background(), box2.Identity().GetPublic(), []byte(\"howdy\"))\n\n// List the recipient's inbox\ninbox, err := box2.ListInboxMessages(context.Background())\n\n// Open decrypts the message body\nbody, err := inbox[0].Open(context.Background(), box2.Identity())\n\n// Mark the message as read\nerr = box2.ReadInboxMessage(context.Background(), inbox[0].ID)\n```\n\n#### Watching for new messages\n\nApplications may watch for mailbox events in the inbox and/or sentbox.\n\n```\n// Handle mailbox events as they arrive\nevents := make(chan MailboxEvent)\ndefer close(events)\ngo func() {\n    for e := range events {\n        switch e.Type {\n        case NewMessage:\n            // handle new message\n        case MessageRead:\n            // handle message read (inbox only)\n        case MessageDeleted:\n            // handle message deleted\n        }\n    }\n}()\n\n// Start watching (the third param indicates we want to keep watching when offline)\nstate, err := mailbox.WatchInbox(context.Background(), events, true)\nfor s := range state {\n    // handle connectivity state\n}\n```\n\nSimilarly, use `WatchSentbox` to watch a sentbox.\n\n## Developing\n\nThe easiest way to develop against `hubd` or `buckd` is to use the Docker Compose files found in `cmd`. The `-dev` flavored files do not persist repos via Docker Volumes, which may be desirable in some cases.\n\n## Contributing\n\nPull requests and bug reports are very welcome ❤️\n\nThis repository falls under the Textile [Code of Conduct](./CODE_OF_CONDUCT.md).\n\nFeel free to get in touch by:\n-   [Opening an issue](https://github.com/textileio/textile/issues/new)\n-   Joining the [public Slack channel](https://slack.textile.io/)\n-   Sending an email to contact@textile.io\n\n## Changelog\n\nA changelog is published along with each [release](https://github.com/textileio/textile/releases).\n\n## License\n\n[MIT](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftextileio%2Ftextile","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftextileio%2Ftextile","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftextileio%2Ftextile/lists"}