Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/hy0g0/crypto-data
This project aim to build a reliable application that can handle a lot of crypto data then display it in dashboard
https://github.com/hy0g0/crypto-data
clickhouse grafana kafka python redpanda
Last synced: 11 days ago
JSON representation
This project aim to build a reliable application that can handle a lot of crypto data then display it in dashboard
- Host: GitHub
- URL: https://github.com/hy0g0/crypto-data
- Owner: Hy0g0
- Created: 2024-09-20T07:57:34.000Z (about 2 months ago)
- Default Branch: main
- Last Pushed: 2024-10-24T23:19:01.000Z (13 days ago)
- Last Synced: 2024-10-25T08:04:49.495Z (13 days ago)
- Topics: clickhouse, grafana, kafka, python, redpanda
- Language: Python
- Homepage:
- Size: 116 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Crypto-Data
# TODO
- [x] Create basic readme to allow all group to use and work on project
- [x] Create python scraper to get cryptos
- [x] Make scraper push in redpanda using kafka API
- [x] Add monitoring to Kube cluster and Redpanda
- [x] Deploy redpanda cluster with wright configuration
- [ ] Add distributed Database to save transformed data (cassandra,scyllaDB,CockroachDB)
- [ ] Update readme with redpanda dashboard creation
- [ ] Investigate Clickhouse project to find if it fits our plans
- [ ] Create final front to display our datas
- [ ] Create Consumer to transform data and push it in DB
- [ ] Improve cluster security to protect our datas (follow kubescore advices add security scans and enhance docker image creation)
## Diagram ##
## Setup ##
Install cert-manager using Helm:
```bash
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
--set crds.enabled=true \
--namespace cert-manager \
--create-namespace
```
The Redpanda Helm chart enables TLS by default and uses cert-manager to manage TLS certificates.
```bash
kubectl create namespace redpanda
kubectl create configmap redpanda-io-config --namespace redpanda --from-file=io-config.yaml
```
```bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus-operator prometheus-community/kube-prometheus-stack \
--namespace default \
--create-namespace
--values prometheus-values.yaml
```
get grafana admin password
```bash
sudo kubectl get secret --namespace default prometheus-operator-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
Install the Redpanda Helm chart to deploy a Redpanda cluster and Redpanda Console.
```bash
helm repo add redpanda https://charts.redpanda.com
helm install redpanda redpanda/redpanda \
--version 5.9.4 \
--namespace redpanda \
--create-namespace \
--values redpanda-values.yaml
```
```bash
helm install my-release oci://registry-1.docker.io/bitnamicharts/kafka \
--namespace kafka \
--create-namespace \
--values kafka-values.yaml
```
## OPS ##
```bash
helm upgrade --install redpanda redpanda/redpanda \
--namespace redpanda \
--create-namespace \
--values redpanda-values.yaml
```
## SETUP CLUSTER ##
### monitoring
```bash
kubectl apply -f redpanda-servicemonitor.yaml
```
### create topic
```bash
kubectl exec -it redpanda-0 -n redpanda -- rpk topic create crypto-prices --partitions 3 -c compression.type=lz4
```
### create producer
``` bash
docker build -t redpanda-producer:1.0.0 ./scraper
```
``` bash
kubectl apply -f redpanda-producer.yaml -n redpanda
```
Note : this way of creating producer work for `orbstack` kube cluster for any other you may need to find a way to push image to the cluster repository
## Sources ##
- ### [Redpanda](https://docs.redpanda.com/current/deploy/deployment-option/self-hosted/kubernetes/k-production-deployment/)
- ### [Helm](https://helm.sh/docs/)
- ### [minikube](https://minikube.sigs.k8s.io/docs/start/?arch=%2Fmacos%2Farm64%2Fstable%2Fbinary+download)
- ### [kind](https://kind.sigs.k8s.io/)
- ### [orbstack](https://orbstack.dev/download)