{"id":15470776,"url":"https://github.com/comtihon/surveyor","last_synced_at":"2026-04-28T08:38:15.029Z","repository":{"id":45466680,"uuid":"105589441","full_name":"comtihon/surveyor","owner":"comtihon","description":"Repo for integration test, doc, docker-compose.yml and start-all cluster scripts","archived":false,"fork":false,"pushed_at":"2018-11-11T17:39:13.000Z","size":29,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-02T13:17:07.264Z","etag":null,"topics":["catcher","docker-compose","documentation","integration-test"],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/comtihon.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-10-02T21:57:38.000Z","updated_at":"2022-07-13T20:10:37.000Z","dependencies_parsed_at":"2022-07-14T18:30:43.881Z","dependency_job_id":null,"html_url":"https://github.com/comtihon/surveyor","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/comtihon%2Fsurveyor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/comtihon%2Fsurveyor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/comtihon%2Fsurveyor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/comtihon%2Fsurveyor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/comtihon","download_url":"https://codeload.github.com/comtihon/surveyor/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246034276,"owners_count":20712851,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["catcher","docker-compose","documentation","integration-test"],"created_at":"2024-10-02T02:06:46.126Z","updated_at":"2026-04-28T08:38:10.004Z","avatar_url":"https://github.com/comtihon.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Surveyor [![Build Status](https://travis-ci.org/comtihon/surveyor.svg?branch=master)](https://travis-ci.org/comtihon/surveyor)\nService that allows survey creation, questions answering and statistics calculation.  \n## Architecture\nSurveyor service contains 4 microservices and 3 third-party services:\n1. [Manager](https://github.com/comtihon/survey_manager) - is responsible for creating/editing surveys, questions and \nanswers configuration. Writes surveys to `PostgreSQL`.\n2. [Requests](https://github.com/comtihon/survey_requests) service - is responsible for gathering survey's answers from\nusers. Reads surveys configuration from the same `PostgreSQL` as `Manager`. Writes all answers to `Kafka`.\n3. [Collector](https://github.com/comtihon/survey_collector) service. Reads answers from `Kafka` and aggregates statistics\nto `MongoDB`.\n4. [Statistics](https://github.com/comtihon/survey_statistics) service. Returns answers statistics for question. Reads\nthe same `MongoDB` as `Collector`.\n5. [PostgreSQL](https://www.postgresql.org) for storing surveys, questions and answers configuration created with \n`Manager`.\n6. [Kafka](https://kafka.apache.org/) for streaming answers data.\n7. [MongoDB](https://www.mongodb.com/) for online statistics. It doesn't store full statistics, instead is increments \ncounters for every answer in question. Dynamic documents are used.\n\n```\n    Survey/Question/Answer CRUD --\u003e Manager --\u003e  Postgres\n    \u0026 Step-by-step attach                           ||\n                                                    || Surveys, Questions, Answers configuration\n                                                    ||\n                                                    \\/\n                            Answered Survey ---\u003e  Requests  ---------\u003e Kafka --------\u003e Collector --------\u003e MongoDB\n                                                             answers          answers            aggregated   ||\n                                                                                                 statistics   || statistics\n                                                                                                              ||\n                                                                                                              \\/\n                                                        Get statistics request (question) --------\u003e       Statistics\n```\n## Waiting for a highload\n1. index `question_id` in `MongoDB`\n2. switch to [avro](https://avro.apache.org/) or [msgpack](http://msgpack.org/index.html) in `Kafka`\n3. `Requests` read from `PostgreSQL` slave\n4. add caching Surveys configuration from `Postgres` for `Requests`\n5. `Statistics` read from `MongoDB` slave\n6. `Kafka` and `Requests` to autoscale groups\n7. Put answers in different topics (by Survey's country code) and start more `Collector`\n\n## Limitations\n1. old questions are not deleted.\nIn current version statistics with old (deleted) questions is also fetched from statistics backend. \nIt is frontend's job to filter deleted questions. It can be done with information, fetched from manager backend.  \nIn future we can make sending notifications with schema changes to collector.\n2. statistics in database is not full (only counters).\nIt was made to allow real time (live on frontend) statistics showing. It would be impossible if we saved all statistics \nto sql database and query it with joins. If we will need whole statistics in future - we can easily add kafka to \npostgres stream.\n\n## Run\n\n    make run\nWill clone and build Docker images of services (if not built) and start all services via docker-compose.   \n__Important__: Avoid port conflicts with your running services.  \nSee `docker-compose.yml` for details.  \n__Requirements__:  \n* [docker](https://www.docker.com/)\n* [docker-compose](https://docs.docker.com/compose/)\n\n\n    make stop\nwill stop all the services\n## Integration testing\nRun integrations tests for surveyor microservises.\n### Full test\n1. create survey in `manager`\n2. check created survey in `postgres`\n3. get survey from `manager`\n4. compare surveys\n5. send answers to `requests`\n6. check `kafka` for answers\n7. check answer statistics in `mongodb` (sometimes test is too fast and fails here. Just re-run)\n8. check answer statistics via `statistics`\n### Run\n\n    make build \u0026\u0026 make run \u0026\u0026 make test\n\n__Important__: Services should be accessible for tests. \n\n__Configuration__:\nEnvironment configuration is available in `inventory/`\n\n### Adding your test\nAdd your test in two steps:\n * add your script in  `script/tests/`.\n * and... that's all. Now second step","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcomtihon%2Fsurveyor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcomtihon%2Fsurveyor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcomtihon%2Fsurveyor/lists"}