{"id":17780710,"url":"https://github.com/hy0g0/crypto-data","last_synced_at":"2026-04-29T17:01:42.252Z","repository":{"id":258713944,"uuid":"860315072","full_name":"Hy0g0/Crypto-Data","owner":"Hy0g0","description":"This project aim to build a reliable application that can handle a lot of crypto data then display it in dashboard","archived":false,"fork":false,"pushed_at":"2024-10-24T23:19:01.000Z","size":119,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-25T08:04:49.495Z","etag":null,"topics":["clickhouse","grafana","kafka","python","redpanda"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Hy0g0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-20T07:57:34.000Z","updated_at":"2024-10-24T23:19:05.000Z","dependencies_parsed_at":"2024-10-26T22:55:04.641Z","dependency_job_id":null,"html_url":"https://github.com/Hy0g0/Crypto-Data","commit_stats":null,"previous_names":["hy0g0/crypto-data"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hy0g0%2FCrypto-Data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hy0g0%2FCrypto-Data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hy0g0%2FCrypto-Data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Hy0g0%2FCrypto-Data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Hy0g0","download_url":"https://codeload.github.com/Hy0g0/Crypto-Data/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229701812,"owners_count":18110218,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clickhouse","grafana","kafka","python","redpanda"],"created_at":"2024-10-27T03:03:37.300Z","updated_at":"2026-04-29T17:01:42.184Z","avatar_url":"https://github.com/Hy0g0.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Crypto-Data\n\n# TODO\n\n- [x] Create basic readme to allow all group to use and work on project\n- [x] Create python scraper to get cryptos\n- [x] Make scraper push in redpanda using kafka API\n- [x] Add monitoring to Kube cluster and Redpanda\n- [x] Deploy redpanda cluster with wright configuration\n- [ ] Add distributed Database to save transformed data (cassandra,scyllaDB,CockroachDB)\n- [ ] Update readme with redpanda dashboard creation\n- [ ] Investigate Clickhouse project to find if it fits our plans\n- [ ] Create final front to display our datas\n- [ ] Create Consumer to transform data and push it in DB\n- [ ] Improve cluster security to protect our datas (follow kubescore advices add security scans and enhance docker image creation)\n\n## Diagram ## \n\n![Diagram](./images/schema.svg)\n\n\n## Setup ## \n\nInstall kafka using Helm:\n\n```bash\nhelm install kafka oci://registry-1.docker.io/bitnamicharts/kafka -n kafka\\\n  --set metrics.jmx.enabled=true \\\n  --set metrics.kafka.enabled=true \\\n  --set metrics.serviceMonitor.enabled=true \\\n  --set metrics.serviceMonitor.namespace=default \\\n  --set metrics.prometheusRule.enabled=true \\\n  --set metrics.prometheusRule.namespace=\"default\"\n```\n\n```bash\nhelm repo add prometheus-community https://prometheus-community.github.io/helm-charts\nhelm repo update\nhelm install prometheus-operator prometheus-community/kube-prometheus-stack \\\n  --namespace default \\\n  --create-namespace\n  --values prometheus-values.yaml\n```\n\n\nget grafana admin password\n```bash\nsudo kubectl get secret --namespace default prometheus-operator-grafana  -o jsonpath=\"{.data.admin-password}\" | base64 --decode ; echo\n```\n\nInstall the Redpanda Helm chart to deploy a Redpanda cluster and Redpanda Console.\n\n```bash\nhelm install my-release oci://registry-1.docker.io/bitnamicharts/kafka \\\n  --namespace kafka \\\n  --create-namespace \\\n  --values kafka-values.yaml\n```\n\n## OPS ##\n\n```bash\nhelm upgrade --install redpanda redpanda/redpanda \\\n  --namespace redpanda \\\n  --create-namespace \\\n  --values redpanda-values.yaml\n```\n\n## SETUP CLUSTER ##\n\n### monitoring\n\n```bash\nkubectl apply -f redpanda-servicemonitor.yaml\n```\n\n### create topic \n\n```bash\nkubectl exec -it redpanda-0 -n redpanda -- rpk topic create crypto-prices --partitions 3 -c compression.type=lz4 \n```\n\n### create producer\n\n``` bash\ndocker build -t redpanda-producer:1.0.0 ./scraper  \n```\n\n``` bash\nkubectl apply -f redpanda-producer.yaml -n redpanda \n```\n\nNote : this way of creating producer work for `orbstack` kube cluster for any other you may need to find a way to push image to the cluster repository\n\n## Sources ##\n\n- ### [Redpanda](https://docs.redpanda.com/current/deploy/deployment-option/self-hosted/kubernetes/k-production-deployment/)\n- ### [Helm](https://helm.sh/docs/)\n- ### [minikube](https://minikube.sigs.k8s.io/docs/start/?arch=%2Fmacos%2Farm64%2Fstable%2Fbinary+download)\n- ### [kind](https://kind.sigs.k8s.io/)\n- ### [orbstack](https://orbstack.dev/download)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhy0g0%2Fcrypto-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhy0g0%2Fcrypto-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhy0g0%2Fcrypto-data/lists"}