{"id":25322421,"url":"https://github.com/checkernetwork/fil-deal-ingester","last_synced_at":"2025-10-29T01:31:16.268Z","repository":{"id":207002941,"uuid":"718196354","full_name":"CheckerNetwork/fil-deal-ingester","owner":"CheckerNetwork","description":"A set of tools to manually ingest Filecoin storage deals and produce a list of retrieval tasks","archived":false,"fork":false,"pushed_at":"2025-01-21T13:29:25.000Z","size":100,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-01-29T12:36:28.018Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CheckerNetwork.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-13T15:27:18.000Z","updated_at":"2025-01-21T13:29:27.000Z","dependencies_parsed_at":"2025-01-29T12:47:50.640Z","dependency_job_id":null,"html_url":"https://github.com/CheckerNetwork/fil-deal-ingester","commit_stats":null,"previous_names":["filecoin-station/fil-deal-ingester","checkernetwork/fil-deal-ingester"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CheckerNetwork%2Ffil-deal-ingester","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CheckerNetwork%2Ffil-deal-ingester/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CheckerNetwork%2Ffil-deal-ingester/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CheckerNetwork%2Ffil-deal-ingester/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CheckerNetwork","download_url":"https://codeload.github.com/CheckerNetwork/fil-deal-ingester/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238754216,"owners_count":19524904,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-13T23:40:45.306Z","updated_at":"2025-10-29T01:31:16.263Z","avatar_url":"https://github.com/CheckerNetwork.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fil-deal-ingester\n\nA set of tools to manually ingest Filecoin storage deals and produce a list of retrieval tasks\n\n## Basic use\n\nMake sure you have a recent Rust toolchain set up.\n\nMake sure you have the latest Node.js LTS version installed. Remember to install dependencies after cloning this repository locally.\n\n```\n❯ npm install\n```\n\nSetup port forwarding between your local computer and Postgres instance hosted by Fly.io\n  ([docs](https://fly.io/docs/postgres/connecting/connecting-with-flyctl/)). Remember to use a\n  different port if you have a local Postgres server for development!\n\n```sh\nfly proxy 5454:5432 -a spark-db\n```\n\nFind spark-db entry in 1Password and get the user and password from the connection string.\n\nRun the following command to fetch the market deals and update SPARK DB:\n\n```sh\nDATABASE_URL=postgres://user:password@localhost:5454/spark ./run.sh\n```\n\n**NOTE: This will download \u0026 store several gigabytes of data and can take up to 3-5 hours to complete.**\n\n\u003e If you want to free up disk space, `generated/StateMarketDeals.ndjson` can weigh in at 40GB or more.\n\u003e Feel free to delete all files in the `generated` folder after the script finished running.\n\n\n## Deployment to fly.io\n\n**NOTE: Make sure you have the fly.io CLI installed and are logged in.**\n\n### One-time setup\n\n_Creating scheduled machine has to be done by hand as it's not possible to set up a schedule inside the fly.toml file._\n\nSet up an app, volume and secrets:\n\n```sh\nfly apps create --name=fil-deal-ingester --org=\u003corg-name\u003e\nfly volumes create fil_deal_ingester_data --size=80 --app=fil-deal-ingester --region=cdg --snapshot-retention=1\nfly secrets set DATABASE_URL=\u003cpostgres-connection-string\u003e --app=fil-deal-ingester\nfly secrets set SLACK_WEBHOOK_URL=\u003cslack-webhook-url\u003e --app=fil-deal-ingester\n```\n\nFinally, create the machine with the following command:\n\n```sh\nfly machine run . \\\n--app=fil-deal-ingester \\\n--schedule=daily \\\n--region=cdg \\\n--volume fil_deal_ingester_data:/usr/src/app/generated \\\n--env JSON_CONVERTER_BIN=/usr/src/app/fil-deal-ingester \\\n--env ENVIRONMENT=docker \\\n--vm-size=shared-cpu-2x \\\n--vm-memory=512\n```\n\n### Updating existing machine\n\nIf you want to update the existing machine with new build, you'd have to get machine id first by running:\n\n```sh\nfly machine ls --app fil-deal-ingester\n```\n\nAnd then update the machine with the following command:\n\n```sh\nfly machine update \u003cmachine-id\u003e  --dockerfile Dockerfile\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcheckernetwork%2Ffil-deal-ingester","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcheckernetwork%2Ffil-deal-ingester","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcheckernetwork%2Ffil-deal-ingester/lists"}