{"id":17227421,"url":"https://github.com/plutov/surv","last_synced_at":"2026-04-14T23:32:33.359Z","repository":{"id":64304922,"uuid":"145877116","full_name":"plutov/surv","owner":"plutov","description":"SURV Project 🏄","archived":false,"fork":false,"pushed_at":"2018-08-25T05:40:06.000Z","size":19,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-19T00:41:00.040Z","etag":null,"topics":["docker","docker-compose","go"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/plutov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-08-23T16:02:52.000Z","updated_at":"2018-11-24T01:08:29.000Z","dependencies_parsed_at":"2023-01-15T10:15:24.072Z","dependency_job_id":null,"html_url":"https://github.com/plutov/surv","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/plutov/surv","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plutov%2Fsurv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plutov%2Fsurv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plutov%2Fsurv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plutov%2Fsurv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/plutov","download_url":"https://codeload.github.com/plutov/surv/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/plutov%2Fsurv/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31819718,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-14T18:05:02.291Z","status":"ssl_error","status_checked_at":"2026-04-14T18:05:01.765Z","response_time":153,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","go"],"created_at":"2024-10-15T04:19:14.117Z","updated_at":"2026-04-14T23:32:33.320Z","avatar_url":"https://github.com/plutov.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## SURV Project 🏄 [![Build Status](https://travis-ci.org/plutov/surv.svg?branch=master)](https://travis-ci.org/plutov/surv) \n\nWhy SURV? I believe projects with a name have a soul.\n\nThe SURV project consists of few main components: multiple survey services and a centralized dashboard.\n\nHigh level diagram:\n\n![surv high level diagram](https://s3.amazonaws.com/pliutau.com/High+Level+Diagram.png)\n\nIn this project we use JSON over HTTP for communication as survey services can be external ones and HTTP is the most common format nowadays. For internal services I'd go with protocol buffers as they are type safe and beter serialized.\n\n### Data\n\nDashboard displays all survey submissions from all survey services.\n\n![surv data diagram](https://s3.amazonaws.com/pliutau.com/Data+Diagram.png)\n\n### Survey services\n\nTwo survey services use the same codebase (`./survey`) as they solve the similar problem, but they have different configuration and deployed separately. We can scale only one survey service if necessary and easily redeploy one without affecting other. If we develop 2 completely different survey services it makes sense to have a separate codebase so different teams can work on them. \n\nWe separate survey services by domain, for example one service for gaming, another for online store.\n\nDashboard may access these services via known REST API endpoints. If later we have a lot of services it makes sense to have a service discovery using Consul or etcd to know how to connect to services.\n\n#### Survey service configuration:\n\n - Name\n - Address\n\nI manage local configuration in environment variables of Docker Compose, later the production config can be stored in Vault / KMS / etc.\n\n#### API\n\nBasically each survey service is a separate REST API service. Now it is a public API, but in real world it's more safe to guard it with OAuth 2.0 or at least Basic Auth.\n\nAPI is documented using Swagger UI.\n\n### Dashboard service\n\nDashboard service (`./dashboard`) pulls data from different (but known) surveys. It has a connector mechanism (client) (`./dashboard/pkg/api/connectors.go`) for each type of survey, then it pulls data, aggregate it and stores in Dashboard storage. Here I assume that survey service API is almost always up, in real world we have to think about circuit breakers.\n\nAs data fetch / aggregation may take some time Dashboard has a Queue and processes data requests 1 by 1. Data from survey service can be requested by simple POST API request to a Dashboard service (or a button in future interface).\n\nAfter aggregation data is available for GET requests.\n\n#### API\n\nFor now dashboard service consists only of REST API and connectors, it doesn't have GUI / Web interface yet. Now it is a public API, but in real world it's more safe to guard it with OAuth 2.0 or at least Basic Auth.\n\nAPI is documented using Swagger UI.\n\nIn future it may be better to use GraphQL for dashboard so we can decide which data do we need.\n\n#### Queue\n\nI use queue to process all survey data fetch sequentially. In real world it may help to separate Dashboard API and Dashboard Consumers and scale only Consumers for example if necessary.\n\nIn real world I'd use RabbitMQ / Beanstalkd or Cloud-based queue, in this project I simply use Go channels.\n\nI implemented very simple consumers, without retries, statuses updates. It should be more robust in real world, for example if API call to survey service failed - we have to delay this message.\n\n#### Dashboard configuration\n\n- List of survey services: address, name, connector name\n\nThe local configuration is stored in `./dashboard/config.json` file and contains 2 local services. If we want to make list of connected services more flexible we may store this list in Dashboard relational storage.\n\n### Storage\n\nFor now each service has in-memory storage, I decided not to go with real database just not to overengineer. The real world system must have persistent database or cloud-based storage. Also each service configured to connect to any storage instance.\n\n### Local environment\n\nTo orchestrate services locally I used Docker Compose as it's commonly used between developers. However for testing / production environments I'd create a Kubernetes configurations. Depends on the cloud choice I'd setup Continuous Delivery to those environments: AWS EKS + AWS Code Pipeline or GCP Kubernetes Engine + Google Cloud Build. No manual builds / pushes!\n\n#### Run it locally\n\nRequirements:\n\n - Docker\n - Docker Compose\n\n*Tested with Docker 18.06.0-ce-mac70 (26399)*\n\n```\ndocker-compose up -d --build\n```\n\nServices configured locally via env files\nServices:\n\n- [localhost:7771](http://localhost:7771) - Dashboard API\n- [localhost:7772](http://localhost:7772) - Gaming Survey Service API\n- [localhost:7773](http://localhost:7773) - Online Store Survey Service API\n\nAPI specs:\n\n- [Dashboard Swagger UI](http://petstore.swagger.io/?url=http://localhost:7771/swagger)\n- [Gaming Survey Swagger UI](http://petstore.swagger.io/?url=http://localhost:7772/swagger)\n- [Store Survey Swagger UI](http://petstore.swagger.io/?url=http://localhost:7773/swagger)\n\n#### How to verify that it works with Swagger UI\n\n- Go to [Gaming Survey Swagger UI](http://petstore.swagger.io/?url=http://localhost:7772/swagger), submit an answer using `POST /answers`.\n\nExample JSON:\n```\n{\n\t\"survey_id\": 1,\n\t\"values\": {\"age\": \"21\"},\n\t\"user\": \"John\"\n}\n```\n\n- Go to [Store Survey Swagger UI](http://petstore.swagger.io/?url=http://localhost:7773/swagger), submit an answer using `POST /answers` using similar JSON from gaming survey service.\n- Go to [Dashboard Swagger UI](http://petstore.swagger.io/?url=http://localhost:7771/swagger), request data fetch from survey services using `POST /request`.\n- After some time you will be able to get dashboard data using `GET /dashboard`.\n\nVoilà!\n\n### Development\n\nI use Go to implement REST APIs as it is efficient language, easy to distribute via single binary file and type safe, which is crutial for APIs.\n\n#### Testing\n\nUnit tests are executed using TravisCI.\n\nRun them manually:\n```\ngo test -v -race ./...\n```\n\n### Monitoring\n\nI'd go with StackDriver if project is deployed to Kubernetes Engine, because it has good integration with different metrics.\n\n### TODO List\n\n- Persistent volume storage instead of in-memory one.\n- Separate storages for each survey and dashboard.\n- API Auth.\n- Use RabbitMQ (or similar technology) for queue management instead of go channels.\n- As Survey service may have a lot of data, we need a faster way to request only latest data so we don't check duplicates. Use Pub/Sub mechanism where it is possible.\n- Currently aggregation is very simple. In real world we need to define models which will make sense and make an aggregation from raw data into structured one.\n- Use GraphQL for Daashboard API, as we can decide what data do we need.\n- Write more Unit tests.\n- Setup Prometheus monitoring.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fplutov%2Fsurv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fplutov%2Fsurv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fplutov%2Fsurv/lists"}