{"id":16085676,"url":"https://github.com/iamraphson/coincap-streaming-data-pipeline","last_synced_at":"2025-04-05T13:42:10.547Z","repository":{"id":232697835,"uuid":"784554658","full_name":"iamraphson/coincap-streaming-data-pipeline","owner":"iamraphson","description":null,"archived":false,"fork":false,"pushed_at":"2024-06-14T01:49:12.000Z","size":620,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-11T09:49:32.864Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iamraphson.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-10T04:38:37.000Z","updated_at":"2024-06-14T01:49:15.000Z","dependencies_parsed_at":"2024-04-11T02:10:18.571Z","dependency_job_id":"25544833-79eb-4cbd-ac6c-e25ec9d9f46e","html_url":"https://github.com/iamraphson/coincap-streaming-data-pipeline","commit_stats":null,"previous_names":["iamraphson/coincap-streaming-data-pipeline"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamraphson%2Fcoincap-streaming-data-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamraphson%2Fcoincap-streaming-data-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamraphson%2Fcoincap-streaming-data-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamraphson%2Fcoincap-streaming-data-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iamraphson","download_url":"https://codeload.github.com/iamraphson/coincap-streaming-data-pipeline/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247345713,"owners_count":20924098,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-09T13:08:40.930Z","updated_at":"2025-04-05T13:42:10.517Z","avatar_url":"https://github.com/iamraphson.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Coincap Streaming pipeline.\n\nThis project demonstrates my expertise in building a streaming data pipeline using Coincap's websocket for real-time cryptocurrency prices and market capitalizations.\n\n[CoinCap API 2.0](https://docs.coincap.io/) is an invaluable tool for accessing real-time pricing and market activity for over 1,000 cryptocurrencies. By aggregating exchange data from thousands of markets, it provides transparent and accurate information on asset prices and availability. The API also offers insights into the specific exchanges and markets that contribute to pricing.\n\nThe primary goal is to design a robust streaming data pipeline that ensures low latency, scalability, and availability. This pipeline aims to track and display trends in cryptocurrency prices and market capitalizations.\n\n### Built With\n\n- Containerization - [**Docker**](https://www.docker.com), [**Docker Compose**](https://docs.docker.com/compose/)\n- Stream Processing - [**Redpanda**](https://redpanda.com/), [**Spark Streaming**](https://spark.apache.org/docs/latest/streaming-programming-guide.html)\n- Data Warehouse - [**Apache Cassandra**](https://cassandra.apache.org/)\n- Data Visualization - [**Grafana**](https://grafana.com/)\n- Language - [**Python**](https://www.python.org)\n\n## Project Architecture\n\n![architecture](Screenshot/application_flow.png)\n\nAll applications are containerized into Docker containers.\n\n## Dashboard\n\n### Assets pricing \n![Assets](Screenshot/All_Asset_Price.png)\n\n### BNB pricing \n![Assets](Screenshot/BNB.png)\n\n### BTC pricing \n![Assets](Screenshot/BTC.png)\n\n### ETH pricing \n![Assets](Screenshot/ETH.png)\n\n\n## Getting Started\n\n### Prerequisites\n\n1. Install VSCode or [Zed](https://zed.dev/) or any other IDE that works for you.\n\n2. [Install Docker Desktop](https://docs.docker.com/get-docker/)\n\n3. Clone this repository onto your local machine.\n\n\n### Set up Pipeline\n\n- created an `.env` file.\n```\nREDPANDA_BROKERS='coincap-redpanda:29092'\nASSET_PRICES_TOPIC=data.asset_prices\n```\n\n- Run `docker-compose up` .\n- Access the Grafana dashboard by visiting `http://localhost:1244/` in your web browser. \n- Import `./Grafana/dashboard.json` into Grafana to view the dashboard.\n\n\n## Contact\n\nTwitter: [@iamraphson](https://twitter.com/iamraphson)\n\n🦅","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiamraphson%2Fcoincap-streaming-data-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiamraphson%2Fcoincap-streaming-data-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiamraphson%2Fcoincap-streaming-data-pipeline/lists"}