{"id":13781047,"url":"https://github.com/davidgasquez/filecoin-data-portal","last_synced_at":"2026-04-02T18:38:57.038Z","repository":{"id":174998049,"uuid":"653047302","full_name":"davidgasquez/filecoin-data-portal","owner":"davidgasquez","description":"🧮 Open, serverless, and local friendly Data Platform for the Filecoin Ecosystem","archived":false,"fork":false,"pushed_at":"2026-03-23T12:09:52.000Z","size":5673,"stargazers_count":34,"open_issues_count":17,"forks_count":10,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-03-24T09:46:37.357Z","etag":null,"topics":["data-analysis","data-platform","filecoin"],"latest_commit_sha":null,"homepage":"https://filecoindataportal.xyz","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davidgasquez.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":"FUNDING.json","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null},"funding":{}},"created_at":"2023-06-13T09:58:40.000Z","updated_at":"2026-03-23T12:09:56.000Z","dependencies_parsed_at":"2026-02-12T05:09:12.574Z","dependency_job_id":null,"html_url":"https://github.com/davidgasquez/filecoin-data-portal","commit_stats":null,"previous_names":["davidgasquez/filecoin-analytics","davidgasquez/filecoin-data-portal"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/davidgasquez/filecoin-data-portal","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidgasquez%2Ffilecoin-data-portal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidgasquez%2Ffilecoin-data-portal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidgasquez%2Ffilecoin-data-portal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidgasquez%2Ffilecoin-data-portal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davidgasquez","download_url":"https://codeload.github.com/davidgasquez/filecoin-data-portal/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidgasquez%2Ffilecoin-data-portal/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31077754,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-28T00:48:58.392Z","status":"online","status_checked_at":"2026-03-28T02:00:07.791Z","response_time":79,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-analysis","data-platform","filecoin"],"created_at":"2024-08-03T18:01:22.464Z","updated_at":"2026-04-02T18:38:57.015Z","avatar_url":"https://github.com/davidgasquez.png","language":"Python","funding_links":[],"categories":["Projects","Learning Material"],"sub_categories":["Projects"],"readme":"\u003cp align=\"center\"\u003e\n  \u003cdiv align=\"center\"\u003e\n    \u003cimg width=\"80px\" src=\"https://filecoindataportal.xyz/logo.svg\"\u003e\n  \u003c/div\u003e\n  \u003ch1 align=\"center\"\u003eFilecoin Data Portal\u003c/h1\u003e\n  \u003cp align=\"center\"\u003eOpen, serverless, and local friendly Data Platform for the Filecoin Ecosystem.\u003c/a\u003e \u003c/p\u003e\n\u003c/p\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg alt=\"GitHub Workflow Status\" src=\"https://img.shields.io/github/actions/workflow/status/davidgasquez/filecoin-data-portal/pipeline.yml?style=flat-square\"\u003e\n  \u003cimg alt=\"GitHub Workflow Status\" src=\"https://img.shields.io/github/actions/workflow/status/davidgasquez/filecoin-data-portal/tables.yml?style=flat-square\"\u003e\n  \u003cimg alt=\"GitHub\" src=\"https://img.shields.io/github/license/davidgasquez/filecoin-data-portal?style=flat-square\"\u003e\n  \u003cimg alt=\"GitHub Repo stars\" src=\"https://img.shields.io/github/stars/davidgasquez/filecoin-data-portal?style=flat-square\"\u003e\n  \u003ca href=\"https://drips.network/app/projects/github/davidgasquez/filecoin-data-portal\" target=\"_blank\"\u003e\u003cimg src=\"https://filecoin.drips.network/api/embed/project/https%3A%2F%2Fgithub.com%2Fdavidgasquez%2Ffilecoin-data-portal/support.png?background=light\u0026style=drips\u0026text=project\u0026stat=none\" alt=\"Support filecoin-data-portal on drips.network\" height=\"21\"\u003e\u003c/a\u003e\n\u003c/div\u003e\n\n\u003cbr\u003e\n\n## 📖 Overview\n\nThis repository contains all the code and related artifacts to process Filecoin data from [diverse sources](portal/docs/data-sources.md) (on-chain and off-chain). You can go directly to the processed datasets or explore the different metrics and pipelines.\n\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://filecoindataportal.xyz/data\" target=\"_blank\"\u003e\n      \u003cimg src=\"https://img.shields.io/badge/GET_THE_DATA-0090ff?style=for-the-badge\" alt=\"GET THE DATA\"\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\n### 📦 Key Features\n\n- **Open**: Code and data are open source and relies on open standards and formats.\n- **Permissionless Collaboration**: Collaborate on data, models, and pipelines. Fork the repo and run the platform locally in mintures. No constraints or platform lock-ins.\n- **Decentralization Options**: Runs on a laptop, server, CI runner, or even on decentralized compute networks like Bacalhau. No local setup required; it even works seamlessly in GitHub Codespaces.\n- **Data as Code**: Each commit generates and pushes all table files to R2.\n- **Modular Flexibility**: Replace, extend, or remove individual components. Compatible with tons of tools. At the end of the day, tables are Parquet files.\n- **Low Friction Data Usage**: Raw and processed data is available to anyone openly. Use whatever tool you want!\n- **Modern Data Engineering**: Supports data engineering essentials such as typing, testing, materialized views, and development branches. Utilizes best practices, including declarative transformations, and utilizes state-of-the-art tools like DuckDB.\n\n## 🚀 Powering\n\n- [Filecoin In Numbers](https://numbers.filecoindataportal.xyz/)\n- [Filecoin Pulse Website](https://pulse.filecoindataportal.xyz/)\n- [Filecoin Metrics Dune Dashboard](https://dune.com/kalen/filecoin-daily-metrics)\n- [Filecoin Storage Providers Market Dashboard](https://filecoin-project.github.io/filecoin-storage-providers-market/)\n\n## 🛠️ Contributing\n\nThis project is in active development. You can help by giving ideas, answering questions, reporting bugs, proposing enhancements, improving the documentation, and fixing bugs. Feel free to open issues and pull requests!\n\nSome ways you can contribute to this project:\n\n- Adding new data sources.\n- Improving the data quality of existing datasets.\n- Adding tests to the data pipelines.\n\n## ⚙️ Development\n\nYou can run the Filecoin Data Portal locally using Python Virtual Environment or VSCode Development Containers. You'll need the following secrets in your environment:\n\n- A `SPACESCOPE_TOKEN` to access [Spacescope](https://spacescope.io/) API.\n- A Google Cloud Platform `GOOGLE_APPLICATION_CREDENTIALS` for accessing BigQuery.\n- A `SPARK_API_BEARER_TOKEN` for accessing Spark retrievals API.\n- A `DUNE_API_KEY` for accessing Dune Analytics.\n\n### 🐍 Python Virtual Environment\n\nClone the repository and run the following commands (or `make setup`) from the root folder:\n\n```bash\n# Create a virtual environment\npip install uv \u0026\u0026 uv venv\n\n# Install the package and dependencies\nuv pip install -U -e .[dev]\n```\n\nNow, you should be able to spin up Dagster UI (`make dev`) and [access it locally](http://127.0.0.1:3000).\n\n### 🐳 Dev Container\n\nYou can jump into the repository [Development Container](https://code.visualstudio.com/docs/remote/containers). Once inside the develpment environment, you'll only need to run `make dev` to spin up the [Dagster UI locally](http://127.0.0.1:3000). The development environment can also run in your browser thanks to GitHub Codespaces!\n\n[![Development Container Badge](https://github.com/codespaces/badge.svg)](https://codespaces.new/davidgasquez/filecoin-data-portal)\n\n## 📃 Disclaimer\n\nThe datasets provided by this service are made available \"as is\", without any warranties or guarantees of any kind, either expressed or implied. By using these datasets, you agree that you do so at your own risk.\n\n## 📝 License\n\n[MIT](https://choosealicense.com/licenses/mit/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidgasquez%2Ffilecoin-data-portal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidgasquez%2Ffilecoin-data-portal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidgasquez%2Ffilecoin-data-portal/lists"}