Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/filecoin-station/fil-deal-ingester
A set of tools to manually ingest Filecoin storage deals and produce a list of retrieval tasks
https://github.com/filecoin-station/fil-deal-ingester
Last synced: 1 day ago
JSON representation
A set of tools to manually ingest Filecoin storage deals and produce a list of retrieval tasks
- Host: GitHub
- URL: https://github.com/filecoin-station/fil-deal-ingester
- Owner: filecoin-station
- License: other
- Created: 2023-11-13T15:27:18.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-02-19T09:44:11.000Z (11 months ago)
- Last Synced: 2024-04-01T15:06:04.613Z (9 months ago)
- Language: JavaScript
- Size: 43.9 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
README
# fil-deal-ingester
A set of tools to manually ingest Filecoin storage deals and produce a list of retrieval tasks
## Basic use
Make sure you have a recent Rust toolchain set up.
Make sure you have the latest Node.js LTS version installed. Remember to install dependencies after cloning this repository locally.
```
❯ npm install
```
Setup port forwarding between your local computer and Postgres instance hosted by Fly.io
([docs](https://fly.io/docs/postgres/connecting/connecting-with-flyctl/)). Remember to use a
different port if you have a local Postgres server for development!
```sh
fly proxy 5454:5432 -a spark-db
```
Find spark-db entry in 1Password and get the user and password from the connection string.
Run the following command to fetch the market deals and update SPARK DB:
```sh
DATABASE_URL=postgres://user:password@localhost:5454/spark ./run.sh
```
**NOTE: This will download & store several gigabytes of data and can take up to 3-5 hours to complete.**
> If you want to free up disk space, `generated/StateMarketDeals.ndjson` can weigh in at 40GB or more.
> Feel free to delete all files in the `generated` folder after the script finished running.
## Deployment to fly.io
**NOTE: Make sure you have the fly.io CLI installed and are logged in.**
### One-time setup
_Creating scheduled machine has to be done by hand as it's not possible to set up a schedule inside the fly.toml file._
Set up an app, volume and secrets:
```sh
fly apps create --name=fil-deal-ingester --org=
fly volumes create fil_deal_ingester_data --size=80 --app=fil-deal-ingester --region= --snapshot-retention=1
fly secrets set DATABASE_URL= --app=fil-deal-ingester
fly secrets set SLACK_WEBHOOK_URL= --app=fil-deal-ingester
```
Finally, create the machine with the following command:
```sh
fly machine run . \
--app=fil-deal-ingester \
--schedule=daily \
--region= \
--volume fil_deal_ingester_data:/usr/src/app/generated \
--env JSON_CONVERTER_BIN=/usr/src/app/fil-deal-ingester \
--env ENVIRONMENT=docker \
--vm-size=shared-cpu-1x
```
### Updating existing machine
If you want to update the existing machine with new build, you'd have to get machine id first by running:
```sh
fly machine ls --app fil-deal-ingester
```
And then update the machine with the following command:
```sh
fly machine update --dockerfile Dockerfile
```