{"id":22449239,"url":"https://github.com/konh/analyticspipeline","last_synced_at":"2026-04-16T03:32:37.264Z","repository":{"id":230012142,"uuid":"766859272","full_name":"KonH/AnalyticsPipeline","owner":"KonH","description":"The opinionated approach combines a client-side SDK, operative + historical data storage, and a data mart","archived":false,"fork":false,"pushed_at":"2024-05-06T11:08:55.000Z","size":1428,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-09-04T04:45:33.570Z","etag":null,"topics":["analytics","business-intelligence","csharp","data-driven","data-storage","dotnet","unity"],"latest_commit_sha":null,"homepage":"","language":"ShaderLab","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KonH.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-03-04T09:06:46.000Z","updated_at":"2024-08-07T06:47:39.000Z","dependencies_parsed_at":"2025-09-04T04:33:30.243Z","dependency_job_id":"4c85dbc5-5ae2-4373-8e64-04f5ff29c7e0","html_url":"https://github.com/KonH/AnalyticsPipeline","commit_stats":null,"previous_names":["konh/analyticspipeline"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/KonH/AnalyticsPipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KonH%2FAnalyticsPipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KonH%2FAnalyticsPipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KonH%2FAnalyticsPipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KonH%2FAnalyticsPipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KonH","download_url":"https://codeload.github.com/KonH/AnalyticsPipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KonH%2FAnalyticsPipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31870508,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-15T15:24:51.572Z","status":"online","status_checked_at":"2026-04-16T02:00:06.042Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","business-intelligence","csharp","data-driven","data-storage","dotnet","unity"],"created_at":"2024-12-06T05:08:19.515Z","updated_at":"2026-04-16T03:32:37.226Z","avatar_url":"https://github.com/KonH.png","language":"ShaderLab","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AnalyticsPipeline\n\n## Summary\n\nAs organizations increasingly rely on data-driven insights, creating a robust analytics system becomes paramount.\nMy project aims to deliver an opinionated approach to building such a system, combining efficiency, scalability, and practicality.\nLet’s dive into the key aspects:\n\n### Objective and Scope:\n\n- Analytics system serves as a bridge between raw data and actionable insights. It empowers decision-makers by providing real-time and historical data analysis.\n- That system focus on a medium-scale solution, striking a balance between agility and depth.\n\n### Components:\n\n- **Client-Side SDK**: our SDK facilitates seamless data collection from various sources (Unity game engine: standalone, web, mobile apps). It ensures minimal impact on user experience while capturing essential events.\n- **Operative and Historical Data Storage:**\n  - **Operative Data:** real-time data storage for immediate processing. Minimal latency, but limited volume / time range.\n  - **Historical Data:** long-term storage for trend analysis, reporting, and compliance. Enables retrospective insights and predictive modeling. Full data retention, but higher latency.\n  - **Data mart** - both data storages allows to query data using SQL and set up useful views for business users.\n\n### Technology Stack:\n\nI’ve chosen the .NET ecosystem for several reasons:\n- **Unity Game Developers:** .NET is familiar to Unity developers, allowing seamless transitions between game development and analytics.\n- **Language Consistency:** Using C# for both client and server-side development streamlines code maintenance and collaboration.\n- **Rich Libraries:** Leverage .NET libraries for data manipulation, security, and scalability.\n\n### Business Impact:\n\nThat analytics system empowers stakeholders to:\n- Optimize user experiences based on real-time data.\n- Identify bottlenecks, anomalies, and growth opportunities.\n- Make informed decisions backed by historical trends.\n- Enhance operational efficiency and resource allocation.\n\nIn summary, my opinionated analytics system combines technical excellence with practicality, enabling data-driven success.\nLet’s turn insights into action! 🚀📊\n\n## Architecture chart legend\n\n```mermaid\nflowchart TD\nICA(Internal Component A: tech descripotion) --\u003e |Sync operation| EC\u003eExternal Component]\nEC -.-\u003e |Async operation| ICB(Internal Component B: tech description)\nICB ==\u003e |Sync write operation| S[(Storage)]\n```\n\n## Abstract architecture\n\n```mermaid\nflowchart TD\n    C(Client SDK) --\u003e |HTTP POST| R(Data Receiver)\n    R --\u003e Q\u003eEvent Queue]\n    Q -.-\u003e |Poll| W(Data Writer)\n    W ==\u003e OD[(Operative Database)]\n    OD -.-\u003e |Retention period| M(Data Migrator)\n    M ==\u003e HD[(Historical Database)]\n    OD --\u003e DM\u003eData Mart]\n    HD --\u003e DM\n```\n\n## Self-hosted architecture\n\n```mermaid\nflowchart TD\nC(Client SDK: C#, Unity) --\u003e |HTTP POST| LB\nsubgraph docker compose\nLB\u003enginx]\nR(Data Receiver: C#, ASP.NET Core)\nQ\u003eredpanda]\nW(Data Writer: C#, ASP.NET Core, EF.Core)\nM(Data Migrator: C#, ASP.NET Core, EF.Core)\nP\u003eApache Presto]\nDM\u003eData Mart: Apache Superset]\nend\nLB --\u003e R\nR --\u003e Q\nQ -.-\u003e |Poll| W\nsubgraph Local Database\nOD[(Operative Database: Postgres)]\nend\nW ==\u003e OD\nOD -.-\u003e |Retention period| M\nsubgraph Local MiniIO\nHD[(Historical database bucket)] --\u003e P\nend\nM ==\u003e HD\nOD --\u003e DM\nP --\u003e DM\n```\n\n## Cloud architecture (AWS-based)\n\n```mermaid\nflowchart TD\nC(Client SDK: C#, Unity) --\u003e |HTTP POST| ALB\u003eLoad balancer: AWS ALB]\nALB --\u003e R \nsubgraph ECS [AWS ECS]\nR(Data Receiver: C#, ASP.NET Core)\nW(Data Writer: C#, ASP.NET Core, EF.Core)\nM(Data Migrator: C#, ASP.NET Core, EF.Core)\nP\u003eApache Presto]\nDM\u003eData Mart: Apache Superset]\nend\nR --\u003e Q\u003eEvent Queue: AWS SQS]\nQ -.-\u003e |Poll| W\nsubgraph RDS [AWS RDS]\nOD[(Operative Database: Postgres)]\nend\nW ==\u003e OD\nOD -.-\u003e |Retention period| M\nsubgraph S3 [AWS S3]\nHD[(Historical database bucket)] --\u003e P\nend\nM ==\u003e HD\nOD --\u003e DM\nP --\u003e DM\n```\n\n## Self-hosted implementation\n\n### Dependencies\n\n- Docker\n\n### Usage\n\nTo run whole system:\n\n```\ndocker-compose up -d\n```\n\nTo run with full services rebuild:\n\n``` \ndocker-compose up -d --build\n```\n\n#### Superset configuration\n\nTo setup Superset, you need to run the following commands:\n\n```\ndocker exec -it analyticspipeline-superset sh\nexport FLASK_APP=superset; flask fab create-admin\nsuperset db upgrade\nsuperset init\n```\n\nThen you can access Superset at `http://localhost:8088`.\n\nTo configure access to operative storage, you need to add a new database connection using web UI:\n\n```\nPlus Sign -\u003e Data -\u003e Connect database:\n\u003e Host: postgres\n\u003e Post: 5432\n\u003e Database name: analytics_database\n\u003e Username: analytics_r_user\n\u003e Password: analytics_r_user_password\n```\n\n### Examples\n\n#### Unity Sample Project\n\n![unity sample project](./static/client_prototype.png)\n\n#### Superset Dashboard\n\n![superset dashboard 1](./static/superset_dashboard_1.png)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkonh%2Fanalyticspipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkonh%2Fanalyticspipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkonh%2Fanalyticspipeline/lists"}