{"id":22998436,"url":"https://github.com/tablelandnetwork/basin-cli","last_synced_at":"2025-06-28T23:37:11.285Z","repository":{"id":191684603,"uuid":"677533217","full_name":"tablelandnetwork/basin-cli","owner":"tablelandnetwork","description":"Continuously publish data from your database to the Tableland network.","archived":false,"fork":false,"pushed_at":"2024-03-28T15:22:39.000Z","size":578,"stargazers_count":12,"open_issues_count":2,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-03-28T16:42:38.828Z","etag":null,"topics":["basin","depin","filecoin"],"latest_commit_sha":null,"homepage":"https://tableland.xyz","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tablelandnetwork.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-08-11T20:31:23.000Z","updated_at":"2023-12-22T18:01:47.000Z","dependencies_parsed_at":"2023-09-28T21:55:47.446Z","dependency_job_id":"55495e88-ab65-4a61-979b-0b70680921cc","html_url":"https://github.com/tablelandnetwork/basin-cli","commit_stats":null,"previous_names":["tablelandnetwork/basin-cli"],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tablelandnetwork%2Fbasin-cli","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tablelandnetwork%2Fbasin-cli/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tablelandnetwork%2Fbasin-cli/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tablelandnetwork%2Fbasin-cli/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tablelandnetwork","download_url":"https://codeload.github.com/tablelandnetwork/basin-cli/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229786993,"owners_count":18124014,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["basin","depin","filecoin"],"created_at":"2024-12-15T06:13:25.472Z","updated_at":"2024-12-15T06:13:26.120Z","avatar_url":"https://github.com/tablelandnetwork.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# vaults-cli\n\n[![License](https://img.shields.io/github/license/tablelandnetwork/basin-cli.svg)](./LICENSE)\n[![standard-readme compliant](https://img.shields.io/badge/standard--readme-OK-green.svg)](https://github.com/RichardLitt/standard-readme)\n\n\u003e Continuously publish data from your database or file uploads to the Tableland Vaults network.\n\n## Table of Contents\n\n- [Background](#background)\n- [Usage](#usage)\n  - [Install](#install)\n  - [Postgres setup](#postgres-setup)\n    - [Self-hosted](#self-hosted)\n    - [Amazon RDS](#amazon-rds)\n    - [Supabase](#supabase)\n  - [Create a vault](#create-a-vault)\n  - [Start replicating a database](#start-replicating-a-database)\n  - [Write files](#write-files)\n  - [Listing vaults](#listing-vaults)\n  - [Listing events](#listing-events)\n  - [Retrieving data](#retrieving-data)\n  - [HTTP APIs](#http-apis)\n    - [Create a vault](#create-a-vault-1)\n    - [Write files](#write-files-1)\n    - [Listing vaults](#listing-vaults-1)\n    - [List events](#list-events)\n    - [Retrieving data](#retrieving-data-1)\n- [Development](#development)\n  - [Running](#running)\n  - [Run tests](#run-tests)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Background\n\nTextile Vaults is a secure and verifiable open data platform. The Vaults CLI is a tool that allows you to continuously replicate a table or view from your database to the network (currently, only PostgreSQL is supported). Or, you can directly upload files to the vault (currently, parquet is only supported)\n\n\u003e 🚧 Vaults is currently not in a production-ready state. Any data that is pushed to the network may be subject to deletion. 🚧\n\n## Usage\n\n### Install\n\nYou can either install the CLI from the remote source:\n\n```bash\ngo install github.com/tablelandnetwork/basin-cli/cmd/vaults@latest\n```\n\nOr clone from source and run the Makefile `install` command:\n\n```bash\ngit clone https://github.com/tablelandnetwork/basin-cli.git\ncd basin-cli\nmake install\n```\n\n### Postgres setup\n\nYou can either write files directly to the network, _or_ you can replicate one or more tables from Postgres database. The replication process requires a few configuration steps before you can create a vault and start streaming data.\n\n#### Self-hosted\n\n- Make sure you have access to a superuser role. For example, you can create a new role such as `CREATE ROLE vaults WITH PASSWORD NULL LOGIN SUPERUSER;`.\n- Check that your Postgres installation has the [wal2json](https://github.com/eulerto/wal2json) plugin installed.\n- Check if logical replication is enabled:\n\n  ```sql\n  SHOW wal_level;\n  ```\n\n  The `wal_level` setting must be set to logical: `ALTER SYSTEM SET wal_level = logical;`.\n\n- Restart the database in order for the new `wal_level` to take effect (be careful!).\n\n#### Amazon RDS\n\n- Make sure you have a user with the `rds_superuser` role, and use `psql` to connect to your database.\n\n  ```console\n  psql -h [HOST] -U [USER] -d [DATABASE]\n  ```\n\n- Check if logical replication is enabled:\n\n  ```sql\n  SELECT name, setting\n  FROM pg_settings\n  WHERE name = 'rds.logical_replication';\n  ```\n\n- If it's on, go to [Create a vault](#create-a-vault)\n- If it's off, follow the next steps:\n  - [Create a custom RDS parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithDBInstanceParamGroups.html#USER_WorkingWithParamGroups.Creating)\n  - After creation, edit it and set the `rds.logical_replication` parameter to `1`\n  - [Associate the recently created parameter group with you DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithDBInstanceParamGroups.html#USER_WorkingWithParamGroups.Associating)\n    - You can choose **Apply immediately** to apply the changes immediately\n    - You'll probably need to reboot the instance for changes to take effect (be careful!)\n- After reboot, check if logical replication is enabled\n\n#### Supabase\n\n- Log into the [Supabase](https://supabase.io/) dashboard and go to your project, or create a new one.\n- Check if logical replication is enabled. This should be the default setting, so you shouldn't have to change anything. You can do this in the `SQL Editor` section on the left hand side of the Supabase dashboard by running `SHOW wal_level;` query, which should log `logical`.\n- You can find the database connection information on the left hand side under `Project Settings` \u003e `Database`. You will need the `Host`, `Port`, `Database`, `Username`, and `Password` to connect to your database.\n\n### Create a vault\n\n_Vaults_ define the place you push data into.\n\nVaults uses public key authentication, so you will need an Ethereum style (ECDSA, secp256k1) wallet to create a new vault. You can use an existing wallet or set up a new one with `vaults wallet create`. Your private key is only used locally for signing.\n\n```bash\nvaults account create [FILENAME]\n```\n\nA new private key will be written to `FILENAME`.\n\nThe name of a vault contains a `namespace` (e.g. `my_company`) and an identifier (e.g., `my_data`), separated by a period (`.`). Use `vaults create` to create a new vault. See `vaults create --help` for more info.\n\n```bash\nvaults create --account [WALLET_ADDRESS] [namespace.identifier]\n```\n\nTo crete a vault with a Time-to-Live (TTL) cache period (in minutes), use the `--cache` flag:\n\n```bash\nvaults create --account [WALLET_ADDRESS] --cache 10 [namespace.identifier]\n```\n\n### Start replicating a database\n\nUse `vaults stream` to start a daemon that will continuously push changes to the underlying table/view to the network. See `vaults stream --help` for more info.\n\n```bash\nvaults stream --dburi [DB_URI] --tables t1,t2 --private-key [PRIVATE_KEY] [namespace.identifier]\n```\n\nThe `--dburi` should follow this format:\n\n```sh\npostgresql://[USER]:[PASSWORD]@[HOST]:[PORT]/[DATABASE]\n```\n\n\u003e 🚧 Vaults currently only replicates `INSERT` statements, which means that it only replicates append-only data (e.g., log-style data). Row updates and deletes will be ignored. 🚧\n\n### Write files\n\nBefore writing a file, you need to [Create a vault](#create-a-vault), if not already created. Then, use `vaults write` to write a Parquet file.\n\n```bash\nvaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] filepath\n```\n\nYou can attach a timestamp to that file write, e.g.\n\n```bash\nvaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] --timestamp 1699984703 filepath\n\n# or use data format\nvaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] --timestamp 2006-01-02 filepath\n\n# or use RFC3339 format\nvaults write --vault [namespace.identifier] --private-key [PRIVATE_KEY] --timestamp 2006-01-02T15:04:05Z07:00 filepath\n```\n\nIf a timestamp is not provided, the CLI will assume the timestamp is the current client epoch in UTC.\n\n### Listing vaults\n\nYou can list the vaults from an account by running:\n\n```bash\nvaults list --account [ETH_ADDRESS]\n```\n\n### Listing events\n\nYou can list events of a given vault by running:\n\n```bash\nvaults events --vault [VAULT_NAME] --latest 5\n```\n\nEvents command accept `--before`,`--after` , and `--at` flags to filter events by timestamp\n\n```bash\n# examples\nvaults events --vault demotest.data --at 1699569502\nvaults events --vault demotest.data --before 2023-11-09T19:38:23-03:00\nvaults events --vault demotest.data --after 2023-11-09\n```\n\n### Retrieving data\n\nYou can retrieve a file from a vault by running:\n\n```bash\nvaults retrieve bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq\n```\n\nYou can specify the file where the retrieved content will be save:\n\n```bash\nvaults retrieve --output [FILENAME] bafybeifr5njnrw67yyb2h2t7k6ukm3pml4fgphsxeurqcmgmeb7omc2vlq\n```\n\n### HTTP APIs\n\nInstead of using the CLI, you can use the HTTP APIs directly. All requests use the following base URL:\n\n- `https://basin.tableland.xyz`\n\n#### Create a vault\n\n`POST /vaults/{vault_id}`\n\n- Note: the `vault_id` must contain a namespace and identifier, separated by a `.`—such as `test_vault.data`.\n\nParams:\n\n- `account` (required)\n- `cache` (optional)\n\n**Examples**\n\n**Without cache**\n\n```bash\ncurl --data 'account=0x78C61e68f9f985C43e36dD5ced3f5a24aD0c503e' \\\n'https://basin.tableland.xyz/vaults/test_vault.data'\n```\n\n**With cache (in minutes)**\n\n```bash\ncurl --data 'account=0x78C61e68f9f985C43e36dD5ced3f5a24aD0c503e\u0026cache=10' \\\n'https://basin.tableland.xyz/vaults/test_vault.data'\n```\n\n#### Write files\n\n`POST /vaults/{vault_id}/events`\n\nHeaders:\n\n- `filename`: The name to store the file as.\n\nParams:\n\n- `signature` (required): hex encoded keccak hash of the event\n- `timestamp` (optional): unix timestamp\n\n**Examples**\n\n```bash\ncurl -H \"filename: data.parquet\" --data-binary \"@data.parquet\" \\\n'https://basin.tableland.xyz/vaults/test_vault.data/event?timestamp=1708987192\u0026signature=a4cb49a595988e2a3b20e6ee468d50a8d3c3cb01a278754c07efda3a89a7e60527545deb512204b034100d6d6b9d169a2d22f5e6286c9c0272e8dc920981941a00'\n```\n\nNote about implementation:\n\n- In this example, the `timestamp` and `signature` are query parameters.\n- You **must sign the event before sending**. For example, you can use CLI's `sign` command to get the signature string needed in the request above:\n  ```\n  vaults sign --private-key 0x1234abcd /path/to/file\n  ```\n\n#### Listing vaults\n\n`GET /vaults?account={address}`\n\n**Examples**\n\n```bash\ncurl 'https://basin.tableland.xyz/vaults?account=0x78C61e68f9f985C43e36dD5ced3f5a24aD0c503e'\n```\n\n#### List events\n\n`GET /vaults/{vault_id}/events`\n\nIt supports the `limit`, `offset`, `before` and `after` as optional params.\n\n**Examples**\n\n```bash\ncurl 'https://basin.tableland.xyz/vaults/cache_long.test/events'\n```\n\n- Note: the default value for `limit` is `10`, so be sure to request more events or filter with `before` or `after`, in case there are more than 10 events in a vault.\n\n#### Retrieving data\n\n`GET /events/{event_id}`\n\n**Examples**\n\n```bash\ncurl 'https://basin.tableland.xyz/events/bafkreibq2xwn6xejhdemvoqqjzvbns5agqob4f2zhegutshbzpz6zyemv4' -o data.parquet\n```\n\n\u003e Currently, this endpoint can **only download data from the cache**.\n\u003e\n\u003e If the cache has expired (or was never set), nothing will be downloaded via the HTTP API. Only the CLI `retrieve` command is able to retrieve a non-cached file directly from IPFS/Filecoin. We’re working to implement ways to make the HTTP API as seamless as the CLI flow.\n\n## Development\n\n### Running\n\nYou can make use of the scripts inside `scripts` to facilitate running the CLI locally without building.\n\n```bash\n# Starting the Provider Server\nPORT=8888 ./scripts/server.sh\n\n# Create an account\n./scripts/run.sh account create pk.out\n\n# Start replicating\n./scripts/run.sh vaults stream --dburi [DB_URI] --tables t1,t2 --private-key [PRIVATE_KEY] namespace.identifier\n```\n\n### Run tests\n\n```bash\nmake test\n```\n\nNote: One of the tests requires Docker Engine to be running.\n\n## Contributing\n\nPRs accepted.\n\nSmall note: If editing the README, please conform to the\n[standard-readme](https://github.com/RichardLitt/standard-readme) specification.\n\n## License\n\nMIT AND Apache-2.0, © 2021-2024 Tableland Network Contributors\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftablelandnetwork%2Fbasin-cli","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftablelandnetwork%2Fbasin-cli","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftablelandnetwork%2Fbasin-cli/lists"}