{"id":18452889,"url":"https://github.com/miry/wattx_top_coins","last_synced_at":"2025-04-22T11:54:56.684Z","repository":{"id":136895282,"uuid":"166538117","full_name":"miry/wattx_top_coins","owner":"miry","description":"Top Coins","archived":false,"fork":false,"pushed_at":"2019-02-17T19:01:32.000Z","size":1596,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-16T14:25:32.877Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/miry.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-19T11:16:16.000Z","updated_at":"2023-11-13T10:50:31.000Z","dependencies_parsed_at":"2023-04-14T01:30:57.997Z","dependency_job_id":null,"html_url":"https://github.com/miry/wattx_top_coins","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miry%2Fwattx_top_coins","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miry%2Fwattx_top_coins/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miry%2Fwattx_top_coins/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miry%2Fwattx_top_coins/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/miry","download_url":"https://codeload.github.com/miry/wattx_top_coins/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250237815,"owners_count":21397400,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T07:34:48.291Z","updated_at":"2025-04-22T11:54:56.666Z","avatar_url":"https://github.com/miry.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![CircleCI](https://circleci.com/gh/miry/wattx_top_coins.svg?style=svg)](https://circleci.com/gh/miry/wattx_top_coins)\n[![Go Report Card](https://goreportcard.com/badge/github.com/miry/wattx_top_coins)](https://goreportcard.com/report/github.com/miry/wattx_top_coins)\n\n\n# Top Coins Problem\n\n# Development\n\n## Create module\n\n```shell\n$ go mod init github.com/miry/wattx_top_coins\n```\n\n## Docker\n\nImage: https://cloud.docker.com/repository/docker/miry/top_coins/general\n\n## Run\n\nSetup all dependicies:\n\n```shell\n$ ./bin/setup\n```\n\nRun application on local environment\n\n```shell\n$ go run cmd/top_coins/main.go\n$ curl localhost:8080/version\n```\n\nUse containers to run application\n\n```shell\n$ ./bin/local\n```\n\n### Coinmarketcap\n\nCommandline Client to the service https://coinmarketcap.com/api/documentation/v1/\n\n```shell\n$ go run cmd/coinmarketcapctl/main.go --api-key=xxxx\n```\n\n## Requirements\n\n- It should have scalable components\n- Deploy each component independently\n- Have Service Discovery\n- Monitoring\n- Logging\n- Tests\n- Tracing\n- [Go modules](https://github.com/golang/go/wiki/Modules)\n\n## Design\n\n### Naive approach\n\nCollect data on demand.\n\n```mermaid\nsequenceDiagram\n    participant User\n    User-\u003e\u003eAPIServer: GET /?limit=200\n    APIServer-\u003e\u003ePriceService: go gRPC GetPrices\n    APIServer-\u003e\u003eRankingService: go gRPC GetRanks\n    PriceService-\u003e\u003eExternal: GET https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest\n    Note right of PriceService: Wait for data\u003cbr/\u003eThen build response\n    PriceService-\u003e\u003eAPIServer: Prices\n    RankingService-\u003e\u003eExternal: GET https://www.cryptocompare.com/api\n    Note right of RankingService: Wait for data\u003cbr/\u003eThen build response\n    RankingService-\u003e\u003eAPIServer: Ranks\n    Note right of APIServer: Wait for data\u003cbr/\u003eThen merge results\u003cbr/\u003e and build response\n    APIServer-\u003e\u003eUser: HTTP 200 JSON\n```\n![First](img/first_approach.png)\n\nThis approach is good for debug and as skeleton for application. Have big problems:\n- high latency\n- rely on external services, if something wrong could not build correct results\n- duplicate requests\n\n### Cache layers\n\nPre build API response.\n\n```mermaid\nsequenceDiagram\n    participant User\n    User-\u003e\u003eAPIServer: GET /?limit=200\n    APIServer-\u003e\u003eUser: HTTP 200 JSON from Cache\n\n    loop sync: Get prices every 1 minute\n        APIServer--\u003e\u003ePriceService: go gRPC GetPrices\n        PriceService-\u003e\u003eExternal: GET https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest\n        PriceService--\u003e\u003eAPIServer: go gRPC Prices\n    end\n\n    loop sync: Get ranks every 1 minute\n        APIServer--\u003e\u003eRankingService: go gRPC GetRanks\n        RankingService-\u003e\u003eExternal: GET https://www.cryptocompare.com/api\n        RankingService--\u003e\u003eAPIServer: go gRPC Ranks\n    end\n```\n\n![Cache](img/cache.png)\n\nThis approach would allow us to serve requests, even when the services are down. It would add Availability, instead of consistency.\nSame approach we can do with services. Base on rerquirerments, memory usage would not be high. If it would be required to process a big amount of data, than it could be extracted to differnt storage, base on load and  scalibility. Example distributed cache for go applications or use FoundationDB (or similar DataStorages depends on load and availability).\n\n### Update cache on changes\n\nAnother approach to update local cache by notification on changes. Everytime a service get updates of new data. It cacluclattes Hash of it and sends to subscribers on new changes. Subscribers can verify that hash is changed, and requests new records back.\n\n# Walking Skeleton\n\nBefore work on main feature. I first build a walking skeleton. Where built sample services, with fake data and responses. It helps developers to build CI/CD with concurent developing features.\n\n1. Build API server with static response\n1. Build Kubernetes cluster with all resources required to serve multi branch deployment of the web application\n1. Automate run tests\n1. Build pipeline for delivery a new changes to staging. Check tasks: Tests, Lint(Languages and Pronto), Deploy, Run e2e tests (optional)\n\n## Folder structure\n\nThe project tries to follow the almost standard [go project layout](https://medium.com/golang-learn/go-project-layout-e5213cdcfaa2).\n\n## Kubernetes\n\nKubernetes is more than just a container orchestrator at this point. What it really does is provide a complete platform for running a distributed application in the cloud.\nBecause people on hype about it, so most of hosting providers built kubernetes services and manage it for developers.\n\nFeatures that are usefull:\n\n- Ingress controller: Allow to manage subdomains base on code and integration wit cloud provider LB.\n- Prometheus operator: Get metrics from services (apps, ingress controllers, cloud autoscaler and etc).\n- Service: Gives ability to load balance requests betweend containers(IPVS gives more performance) and service discovery.\n- Autoscale containers base on metrics\n- Allow to run staging/production env on local machine with minikube or docker kubernetes for mac\n- There are many pipelines tools for kubernetes\n- Authorisation with Google, Azure or other OpenID providers\n\n## Monitoring\n\nDepends on programming language. Some common tools: Newrelic, Prometheus, Influxdb. For tracing: Zippkin, Jaeger. Most of such services have integration with PagerDuty to send notifications via SMS, Phone call or chat channels. Grafana as free tool to build dashboards.\n\n## Message Queue\n\nI prefer working with Nats, because it is simple to setup and debug. There is Liftbridge project, that makes nats streaming reliable as Kafka. Kafka is a complex system. Requires understands and maintain of Zookeeper. Then tune kafka brokers. For this project it could be used for tracking requests.\n\n## Datastore\n\nBecuase use of container solution there are multiple projects, that could be deployed safe to it: CockroachDB, TiKV(TiDB), Couchbase, Postgres (with stolon), Vitess.\n\n## Logging\n\nKubernetes provide logs per pod. It is possible to deploy Elasticsearch cluster with Fluentd and Kibana. For small projects it makes sense to use services.\n\n## CI/CD pipeline\n\nAll scripts to build and deliver applications should be inside repo. CircleCI provides ability to have config in the repo to automate. In cluster solutions: Teamcity,DroneIO, Argo and similar. To elliminate tolls - evertyhing should be automated.\n\n## RPC\n\nFor microservices it always need to have ability run different services written in different laguages. For this reason I choose gRPC. It uses protobuf, in my opnion it is complex and would prefer msgpack. In same time protobuf provide a lot of specification about messages. gRPC support SSL communication that gives us ability to build zero trust architecture. Complexity - manage SSL certificates.\n\n## Discovery\n\nBy default it is enough to have Kubernetes discovery solution. Also it could be used Consul.\n\n\n# API server\n\nIt should be stateless applicaiton. Depends on SLA: Latency, Threoughput, Budget and Developers skills could be implemented on different language. This information is missing. Go would be a good choice because of same hype. Alternatives: Erlang(Elixir), Ruby, Rust, Crystal(missing grpc), Python.\nDepends on traffic, Serveless approach could be used.\n\n# Services streaming\n\nYou can pick any language, that makes less memory consumption and has grpc protocol implementation: Go, Ruby, Python.\n\n# Short live tasks\n\nKubernetes gives solution to run con tasks. For solution where need only scrape data from external services and put to DB. Then API server can read from DB directly without grpc. Any language.\n\n\n![Cluster](img/structure.jpg)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiry%2Fwattx_top_coins","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmiry%2Fwattx_top_coins","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiry%2Fwattx_top_coins/lists"}