{"id":23123253,"url":"https://github.com/lupusruber/crypto_stats","last_synced_at":"2026-05-15T13:03:24.285Z","repository":{"id":253109529,"uuid":"842493522","full_name":"lupusruber/crypto_stats","owner":"lupusruber","description":"A project that provides a cloud-native solution for ingesting, transforming, and visualizing cryptocurrency data, utilizing modern tools and workflows for scalability and automation.","archived":false,"fork":false,"pushed_at":"2024-12-07T15:04:21.000Z","size":549,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-19T12:05:20.053Z","etag":null,"topics":["data-engineering","data-streaming","etl-pipeline","gcp","terraform"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lupusruber.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-14T13:16:03.000Z","updated_at":"2024-12-07T15:05:57.000Z","dependencies_parsed_at":"2024-08-20T11:17:06.350Z","dependency_job_id":"9178f00b-59a7-49fe-96ff-0e9c2411fddb","html_url":"https://github.com/lupusruber/crypto_stats","commit_stats":null,"previous_names":["lupusruber/crypto_stats"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lupusruber/crypto_stats","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lupusruber%2Fcrypto_stats","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lupusruber%2Fcrypto_stats/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lupusruber%2Fcrypto_stats/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lupusruber%2Fcrypto_stats/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lupusruber","download_url":"https://codeload.github.com/lupusruber/crypto_stats/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lupusruber%2Fcrypto_stats/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33067476,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-15T11:35:32.926Z","status":"ssl_error","status_checked_at":"2026-05-15T11:35:31.362Z","response_time":103,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-engineering","data-streaming","etl-pipeline","gcp","terraform"],"created_at":"2024-12-17T07:33:15.842Z","updated_at":"2026-05-15T13:03:24.258Z","avatar_url":"https://github.com/lupusruber.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Crypto Statistics Data Engineering Project\n\nThis project is designed to process and visualize cryptocurrency data through a cloud-based architecture using Google Cloud Platform (GCP). The project includes end-to-end workflows for data ingestion, transformation, and visualization, with tools like Terraform for infrastructure setup, Mage for workflow orchestration, dbt for data transformation, and Metabase for interactive dashboards. \n\nIt aims to provide an automated, scalable solution for ingesting, transforming, and analyzing large volumes of cryptocurrency data in real-time.\n\n## Technologies used\n\n- **Cloud:** GCP (Google Cloud)\n- **Infrastructure as code (IaC):** Terraform\n- **Workflow orchestration:** Mage\n- **Data Warehouse:** Google BigQuery\n- **Data Lake:** Google Cloud Storage\n- **Data Transofrmations:** dbt (Data Build Tool)\n- **Data Visualizations:** Metabase\n\n## Data Ingestion DAG (Source -\u003e Bucket)\n![Data Ingestion DAG](https://github.com/lupusruber/crypto_stats/blob/master/Images/Data%20Ingestion%20DAG.png)\n\n## ETL (Bucket -\u003e DWH)\n![ETL DAG](https://github.com/lupusruber/crypto_stats/blob/master/Images/ETL%20Pipeline.png)\n\n## Dashboards\n![Dashboard](https://github.com/lupusruber/crypto_stats/blob/master/Images/Dashboard.png.png)\n\n## How to replicate this project?\n\n### 1. Clone the repo\n```bash\ngit clone https://github.com/lupusruber/crypto_stats.git\n```\n\n### 2. Create the needed infrastructure\n```bash\ncd terraform\nterraform init\nterraform plan\nterraform apply\n```\n### 3. Get Mage and run the pipelines\n```bash\ndocker run -it -p 6789:6789 -v $(pwd):/home/src mageai/mageai /app/run_app.sh mage start [project_name]\n```\nCopy the pipeline scripts inside the cointainer.\n\n### 4. Get dbt and run the models\nFor this project dbt cloud was used.\nCreate a new dbt project and add the models from the repo to the project directory.\nRun the command:\n```bash\ndbt build\n```\nThe staged models and the facts should be part of your big query dataset now.\n\n### 5. Get Metabase and create dashboards\n```bash\ndocker run -d -p 3000:3000 --name metabase metabase/metabase\n```\nCreate the dashboards using the data from Big Query.\n\n## Notes\n- You need to have a GCS account\n- Create a service account and download credentials\n- Store credentails in [project_name]/keys/credentials.json, they are used by Terraform, Mage, dbt and Metabase\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flupusruber%2Fcrypto_stats","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flupusruber%2Fcrypto_stats","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flupusruber%2Fcrypto_stats/lists"}