{"id":41469618,"url":"https://github.com/doziestar/datavinci","last_synced_at":"2026-01-23T16:45:40.403Z","repository":{"id":246766475,"uuid":"822095099","full_name":"doziestar/datavinci","owner":"doziestar","description":"DataVinci enables you to visualize data from various sources, generate insights, analyze data with AI models, and receive real-time updates on anomalies","archived":false,"fork":false,"pushed_at":"2024-07-07T14:48:40.000Z","size":11515,"stargazers_count":1,"open_issues_count":4,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-07-07T16:06:49.086Z","etag":null,"topics":["data","golang","logs","pipeline"],"latest_commit_sha":null,"homepage":"https://datavinci.so","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/doziestar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-30T09:52:25.000Z","updated_at":"2024-07-07T14:43:50.000Z","dependencies_parsed_at":"2024-07-07T16:11:05.273Z","dependency_job_id":null,"html_url":"https://github.com/doziestar/datavinci","commit_stats":null,"previous_names":["doziestar/datavinci"],"tags_count":1,"template":true,"template_full_name":null,"purl":"pkg:github/doziestar/datavinci","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doziestar%2Fdatavinci","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doziestar%2Fdatavinci/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doziestar%2Fdatavinci/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doziestar%2Fdatavinci/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/doziestar","download_url":"https://codeload.github.com/doziestar/datavinci/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/doziestar%2Fdatavinci/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28695989,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-23T15:57:05.722Z","status":"ssl_error","status_checked_at":"2026-01-23T15:56:27.656Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","golang","logs","pipeline"],"created_at":"2026-01-23T16:45:35.137Z","updated_at":"2026-01-23T16:45:40.396Z","avatar_url":"https://github.com/doziestar.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DataVinci\n\n![alt text](68119.png)\n\nDataVinci is a comprehensive data management and visualization tool designed for the developer community. It enables users to visualize data from various sources, generate insights, analyze data with AI models, and receive real-time updates on anomalies.\n\n[![codecov](https://codecov.io/github/doziestar/datavinci/graph/badge.svg?token=PN33XKZTE5)](https://codecov.io/github/doziestar/datavinci)\n\n## Table of Contents\n\n- [Features](#features)\n- [Architecture](#architecture)\n- [Getting Started](#getting-started)\n- [Development](#development)\n- [Deployment](#deployment)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Features\n\n- Multi-source data integration (PostgreSQL, MongoDB, Cassandra, Elasticsearch, various logs)\n- Interactive data visualization with customizable dashboards\n- AI-powered data analysis and anomaly detection\n- Real-time data processing and alerts\n- Cloud resource management and visualization (e.g., Amazon S3)\n- Report generation and scheduling\n- Collaboration features with version control\n\n## Architecture\n\nDataVinci follows a microservices architecture for scalability and maintainability. Here's a high-level overview of the system:\n\n```mermaid\n\ngraph TB\n    A[Web UI] --\u003e B[API Gateway]\n    B --\u003e C[Authentication Service]\n    B --\u003e D[Data Source Service]\n    B --\u003e E[Visualization Service]\n    B --\u003e F[Report Service]\n    B --\u003e G[AI Analysis Service]\n    B --\u003e H[Real-time Processing Service]\n    D --\u003e I[Data Connectors]\n    I --\u003e J[(Various Data Sources)]\n    E \u0026 F \u0026 G \u0026 H --\u003e K[Data Processing Engine]\n    K --\u003e L[(Data Lake/Warehouse)]\n    M[Background Jobs] --\u003e K\n\n```\n\n## Getting Started\n\n### Prerequisites\n\n- Go 1.16+\n- Node.js 14+\n- Docker and Docker Compose\n- Kubernetes cluster (for production deployment)\n\n### Installation\n\n1. Clone the repository:\n\n```bash\ngit clone https://github.com/doziestar/datavinci.git\ncd datavinci\n```\n\n2. Install the required dependencies:\n\n```bash\ngo mod download\ncd web \u0026\u0026 yarn install \u0026\u0026 cd ..\n```\n\n3. Set up the environment variables:\n\n```bash\ncp .env.example .env\n# Edit .env with your configuration\n```\n\n4. Start the development server:\n\n```bash\ndocker-compose up -d\ngo run cmd/datavinci/main.go\n\ncd web/src-tauri \u0026\u0026 cargo tauri dev\n```\n\n5. Access the web UI at `http://localhost:3000`.\n\n## Development\n\nDataVinci uses a monorepo structure with Go workspaces (go.work) for backend services and Next.js with Tauri for the frontend.\n\n### Folder Structure\n\n```bash\ndatavinci/\n├── cmd/\n│   └── datavinci/\n│       └── main.go\n├── internal/\n│   ├── auth/\n│   ├── datasource/\n│   ├── visualization/\n│   ├── report/\n│   ├── ai/\n│   └── realtime/\n├── pkg/\n│   ├── common/\n│   └── models/\n├── web/\n│   ├── components/\n│   ├── pages/\n│   └── public/\n├── deployments/\n│   ├── docker/\n│   └── k8s/\n├── scripts/\n├── tests/\n├── go.work\n├── go.mod\n├── go.sum\n├── package.json\n├── docker-compose.yml\n├── Dockerfile\n└── README.md\n```\n\n### Service Communication\n\nThe backend services communicate with each other using gRPC. The API Gateway acts as a reverse proxy for the frontend and forwards requests to the appropriate service.\n\n```mermaid\ngraph TB\n    Client[Client] --\u003e APIGateway[API Gateway]\n    subgraph \"Service Mesh\"\n        APIGateway --\u003e Auth[Authentication Service]\n        APIGateway --\u003e DataSource[Data Source Service]\n        APIGateway --\u003e Visualization[Visualization Service]\n        APIGateway --\u003e Report[Report Service]\n        APIGateway --\u003e AI[AI Analysis Service]\n        APIGateway --\u003e RealTime[Real-time Processing Service]\n    end\n    Auth -.-\u003e|gRPC| DataSource\n    DataSource -.-\u003e|gRPC| Visualization\n    Visualization -.-\u003e|gRPC| Report\n    DataSource -.-\u003e|gRPC| AI\n    DataSource -.-\u003e|gRPC| RealTime\n\n    MessageBroker[Message Broker] --\u003e DataSource\n    MessageBroker --\u003e Visualization\n    MessageBroker --\u003e Report\n    MessageBroker --\u003e AI\n    MessageBroker --\u003e RealTime\n\n    EventStore[(Event Store)] --\u003e MessageBroker\n\n    DataSource --\u003e DB[(Data Sources)]\n    RealTime --\u003e DB\n```\n\n### Testing\n\nRun the tests with:\n\n```bash\ngo test ./...\ncd web \u0026\u0026 yarn test \u0026\u0026 cd ..\n\n// or\n\ngo test -v -race -coverprofile=pkg/coverage.txt -covermode=atomic ./internal/auth/...\n```\n\nTo ensure that the code meets our standards, run the pre-commit hooks:\n\n```bash\npre-commit run --all-files\n```\n\n### Linting\n\nLint the Go code with:\n\n```bash\ngolangci-lint run\n```\n\nLint the JavaScript code with:\n\n```bash\ncd web \u0026\u0026 yarn lint \u0026\u0026 cd ..\n```\n\n## Deployment\n\nDataVinci can be deployed on any cloud provider or on-premises infrastructure. For production deployments, we recommend using Kubernetes with Helm charts.\n\n### Docker\n\nBuild the Docker image with:\n\n```bash\ndocker build -t datavinci:latest .\n```\n\n### Kubernetes\n\nDeploy the application on a Kubernetes cluster with:\n\n```bash\nkubectl apply -f deployments/k8s\n```\n\n### Helm\n\nInstall the Helm chart with:\n\n```bash\nhelm install datavinci deployments/helm\n```\n\n## Contributing\n\nContributions are welcome! Please read the [contributing guidelines](CONTRIBUTING.md) before submitting a pull request.\n\n## Pre-commit Hooks\n\nWe use pre-commit hooks to ensure code quality and consistency. These hooks run automatically before each commit, checking your changes against our coding standards and running various linters.\n\n### Setup\n\n1. Install pre-commit:\n\n   ```bash\n   pip install pre-commit\n   ```\n\n2. Install the git hook scripts:\n   ```bash\n   pre-commit install\n   ```\n\n### Running pre-commit\n\nThe hooks will run automatically on `git commit`. If you want to run the hooks manually (for example, to test them or run them on all files), you can use:\n\n```bash\npre-commit run --all-files\n```\n\n### Our pre-commit hooks\n\nWe use the following hooks:\n\n- **For Go:**\n\n  - `go-fmt`: Formats Go code\n  - `go-vet`: Reports suspicious constructs\n  - `go-imports`: Updates import lines\n  - `go-cyclo`: Checks function complexity\n  - `golangci-lint`: Runs multiple Go linters\n  - `go-critic`: Provides extensive code analysis\n  - `go-unit-tests`: Runs Go unit tests\n  - `go-build`: Checks if the code builds\n  - `go-mod-tidy`: Runs `go mod tidy`\n\n- ** ensure that you have the following tools installed:**\n\n  - `golangci-lint`\n  - `go-critic`\n  - `go-cyclo`\n  - `go-unit-tests`\n  - `go-build`\n  - `go-mod-tidy`\n\n    ```bash\n    go install github.com/fzipp/gocyclo/cmd/gocyclo@latest\n    go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest\n    go install github.com/go-critic/go-critic/cmd/gocritic@latest\n    go install github.com/hexdigest/gounit/cmd/gounit@latest\n    go install github.com/securego/gosec/v2/cmd/gosec@latest\n    ```\n\n- **For TypeScript/JavaScript:**\n\n  - `prettier`: Formats code\n  - `eslint`: Lints JavaScript and TypeScript code\n\n- **General:**\n  - `trailing-whitespace`: Trims trailing whitespace\n  - `end-of-file-fixer`: Ensures files end with a newline\n  - `check-yaml`: Checks yaml files for parseable syntax\n  - `check-added-large-files`: Prevents giant files from being committed\n\n### Skipping hooks\n\nIf you need to bypass the pre-commit hooks (not recommended), you can use:\n\n```bash\ngit commit -m \"Your commit message\" --no-verify\n```\n\nHowever, please use this sparingly and ensure your code still meets our standards.\n\n### Updating hooks\n\nTo update the pre-commit hooks to the latest versions, run:\n\n```bash\npre-commit autoupdate\n```\n\nThen commit the changes to `.pre-commit-config.yaml`.\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoziestar%2Fdatavinci","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdoziestar%2Fdatavinci","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoziestar%2Fdatavinci/lists"}