{"id":13562467,"url":"https://github.com/application-research/estuary","last_synced_at":"2025-05-15T05:07:17.805Z","repository":{"id":37028272,"uuid":"344230147","full_name":"application-research/estuary","owner":"application-research","description":"A custom IPFS/Filecoin node that makes it easy to pin IPFS content and make Filecoin deals.","archived":false,"fork":false,"pushed_at":"2025-03-27T11:28:00.000Z","size":6516,"stargazers_count":241,"open_issues_count":162,"forks_count":66,"subscribers_count":13,"default_branch":"dev","last_synced_at":"2025-04-10T10:09:57.568Z","etag":null,"topics":["filecoin","ipfs","p2p","storage"],"latest_commit_sha":null,"homepage":"https://docs.estuary.tech","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/application-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-03T18:45:48.000Z","updated_at":"2025-03-27T11:27:57.000Z","dependencies_parsed_at":"2024-01-14T03:46:23.496Z","dependency_job_id":"70f3f356-7705-467a-831b-278d39c0c3c8","html_url":"https://github.com/application-research/estuary","commit_stats":null,"previous_names":["whyrusleeping/estuary"],"tags_count":34,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/application-research%2Festuary","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/application-research%2Festuary/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/application-research%2Festuary/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/application-research%2Festuary/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/application-research","download_url":"https://codeload.github.com/application-research/estuary/tar.gz/refs/heads/dev","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254276447,"owners_count":22043867,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["filecoin","ipfs","p2p","storage"],"created_at":"2024-08-01T13:01:08.995Z","updated_at":"2025-05-15T05:07:12.780Z","avatar_url":"https://github.com/application-research.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"\u003e [!WARNING]\n\u003e  # Project Discontinued\n\u003e \n\u003e The instance and its related API endpoints hosted at \u003chttps://estuary.tech\u003e are discontinued and no longer maintained.\n\u003e \n\u003e The codebase is provided as-is for anyone interested in forking or learning.\n\u003e \n----\n\n# Estuary\n\n\u003e An experimental [IPFS](https://ipfs.tech) node\n\nQuestions? Reach out! [![slack](https://img.shields.io/badge/slack-filecoin-blue.svg?logo=slack)](https://filecoinproject.slack.com/)\n\n## Building\n\nRequirements:\n\n- go 1.18\n- [jq](https://stedolan.github.io/jq/)\n- [hwloc](https://www.open-mpi.org/projects/hwloc/)\n- opencl\n- [rustup](https://rustup.rs/)\n- postgresql\n\n1. Run `make clean all` inside the estuary directory\n\n## Running your own node\n\nTo run locally in a 'dev' environment, first run:\n\n```sh\n./estuary setup --username=\u003cuname\u003e --password=\u003cpword\u003e\n```\n\nSave the credentials you use here, you will need them to login to the estuary-www frontend.\n\nNOTE: if you want to use a different database than a sqlite instance stored in your local directory, you will need to configure that with the `--database` flag, like so: `./estuary setup --username=\u003cuname\u003e --password=\u003cpword\u003e --database=XXXXX`\n\nOnce you have the setup complete, choose an appropriate directory for estuary to keep its data, and use it as your datadir flag when running estuary.\nYou will also need to tell estuary where it can access a lotus gateway api, we recommend using:\n\n```sh\nexport FULLNODE_API_INFO=wss://api.chain.love\n```\n\nThen run:\n\n```sh\n./estuary --datadir=/path/to/storage --database=IF-YOU-NEED-THIS --logging\n```\n\nNOTE: Estuary makes only verified deals by default and this requires the wallet address to have datacap(see https://verify.glif.io/). To make deals without datacap, it will require the wallet to have FIL, and the run command will need the `--verified-deal` option set to `false`.\n\n```sh\n./estuary --datadir=/path/to/storage --database=IF-YOU-NEED-THIS --logging --verified-deal=false\n```\n\n## Running as daemon with Systemd\n\nThe Makefile has a target that will install a generic but workable systemd service for estuary.\n\nRun `make install-estuary-service` on the machine you wish to run estuary on.\n\nMake sure to follow the instructions output by the `make` command as configuration is required before the service can run successfully.\n\n## Running estuary using docker\n\n- View the guidelines on how to run estuary using docker [here](https://github.com/application-research/estuary-docker).\n\n## Using Estuary\n\nThe first thing you'll likely want to do with Estuary is upload content. To upload your first file, use the `/content/add`\nendpoint:\n\n```\ncurl -X POST http://localhost:3004/content/add -H \"Authorization: Bearer REPLACE_ME_WITH_API_KEY\" -H \"Accept: application/json\" -H \"Content-Type: multipart/form-data\" -F \"data=@PATH_TO_FILE_BUT_REMEMBER_THE_@_SYMBOL_IS_REQUIRED\"\n```\n\nYou can verify this worked with the `/content/list` endpoint:\n\n```\ncurl -X GET -H \"Authorization: Bearer REPLACE_ME_WITH_API_KEY\" http://localhost:3004/content/list\n```\n\nYou may find the API documentation at [docs.estuary.tech](https://docs.estuary.tech/) useful as you explore Estuary's capabilities.\n\n### Sealing a Deal\n\nEstuary will automatically make a deal with Filecoin miners after 8 hours. If you upload more than 3.57 GiB of data\nit will make the deal sooner.\n\nTo keep tabs on the status of your uploaded content and Filecoin deals, you can use [estuary-www](https://github.com/application-research/estuary-www).\nClone the `estuary-www` repository and run:\n\n```\nnpm install\nnpm run dev\n```\n\nAnd then head to [localhost:4444/staging](http://localhost:4444/staging) to see the status of your deal.\n\n## Contributing\n\nSee [`CONTRIBUTING.md`](CONTRIBUTING.md) for contributing and development instructions.\n\n## Troubleshooting\n\nMake sure to install all dependencies as indicated above. Here are a few issues that one can encounter while building estuary\n\n### Guide for: `route ip+net: netlinkrib: too many open files`\n\n#### Error\n\nIf you get the following error:\n\n```sh\n/ERROR basichost basic/basic_host.go:328 failed to resolve local interface addresses {\"error\": \"route ip+net: netlinkrib: too many open files\"}\n```\n\nIt is because you do not have enough open file handles available.\n\n#### Solution\n\nUpdate this with the following command:\n\n```sh\nulimit -n 10000\n```\n\n### Guide for: Missing `hwloc` on M1 Macs\n\nThe Portable Hardware Locality (hwloc) software package provides a portable abstraction of the hierarchical structure of current architectures, including NUMA memory nodes, sockets, shared caches, cores, and simultaneous multi-threading (across OS, versions, architectures, etc.).\n\n`lhwloc` is used by libp2p-core. Estuary uses libp2p for the majority of its features including network communication, pinning, replication and resource manager.\n\n#### Error\n\n```\n`ld: library not found for -lhwloc`\n```\n\n#### Solution\n\nFor M1 Macs, here's the following steps needed\n\n- Step 1: `brew install go bzr jq pkg-config rustup hwloc` - Uninstall rust as it would clash with rustup in case you have installed.\n- Step 2: `export LIBRARY_PATH=/opt/homebrew/lib`\n- Step 3: Follow the steps as per the docs.\n\nOn Ubuntu, install `libhwloc-dev`.\n\n### Guide for: `cannot find -lfilcrypto collect2`\n\nRelated issue [here](https://github.com/application-research/estuary/issues/71)\n\n#### Error\n\nWhen trying to build estuary in an ARM machine, it returns an error\n\n```\n# github.com/filecoin-project/filecoin-ffi/generated /usr/bin/ld: skipping incompatible extern/filecoin-ffi/generated/../libfilcrypto.a when searching for -lfilcrypto /usr/bin/ld: skipping incompatible extern/filecoin-ffi/generated/../libfilcrypto.a when searching for -lfilcrypto /usr/bin/ld: skipping incompatible extern/filecoin-ffi/generated/../libfilcrypto.a when searching for -lfilcrypto /usr/bin/ld: skipping incompatible extern/filecoin-ffi/generated/../libfilcrypto.a when searching for -lfilcrypto /usr/bin/ld: skipping incompatible extern/filecoin-ffi/generated/../libfilcrypto.a when searching for -lfilcrypto /usr/bin/ld: cannot find -lfilcrypto collect2: error: ld returned 1 exit status make: *** [Makefile:67: estuary] Error 2\n```\n\n#### Solution\n\nRelated solution [here](https://github.com/filecoin-project/lotus/issues/1779#issuecomment-629932097)\n\n```\nRUSTFLAGS=\"-C target-cpu=native -g\" FFI_BUILD_FROM_SOURCE=1 make clean deps bench\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapplication-research%2Festuary","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapplication-research%2Festuary","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapplication-research%2Festuary/lists"}