{"id":17436072,"url":"https://github.com/yindia/cloud","last_synced_at":"2025-12-30T21:06:06.949Z","repository":{"id":257963577,"uuid":"868224027","full_name":"yindia/cloud","owner":"yindia","description":"Build a mini orchestrationtool from zero, new versions every month , Lean Stack ","archived":false,"fork":false,"pushed_at":"2024-10-16T22:48:56.000Z","size":775,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-03T06:07:13.022Z","etag":null,"topics":["cicd","connectrpc","distributed-systems","docker","golang","grpc","k8s","ml","monorepo","shadcn-ui","tailwindcss","typescript"],"latest_commit_sha":null,"homepage":"https://buf.build/evalsocket/cloud","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yindia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-05T19:50:07.000Z","updated_at":"2024-10-16T22:48:59.000Z","dependencies_parsed_at":"2024-10-20T08:11:36.752Z","dependency_job_id":null,"html_url":"https://github.com/yindia/cloud","commit_stats":null,"previous_names":["yindia/cloud"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/yindia/cloud","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yindia%2Fcloud","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yindia%2Fcloud/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yindia%2Fcloud/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yindia%2Fcloud/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yindia","download_url":"https://codeload.github.com/yindia/cloud/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yindia%2Fcloud/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279007342,"owners_count":26084282,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cicd","connectrpc","distributed-systems","docker","golang","grpc","k8s","ml","monorepo","shadcn-ui","tailwindcss","typescript"],"created_at":"2024-10-17T10:01:58.077Z","updated_at":"2025-10-11T13:44:46.242Z","avatar_url":"https://github.com/yindia.png","language":"TypeScript","readme":"# Task Service - Case Study\n\n\u003ca href=\"https://vimeo.com/1014374705\"\u003e\n  \u003cimg src=\"https://markdown-videos-api.jorgenkh.no/url?url=https%3A%2F%2Fvimeo.com%2F1014374705\" alt=\"Task Service \" title=\"Task Service \" width=\"100%\" height=\"100%\"/\u003e\n\u003c/a\u003e\n\n## Table of Contents\n- [Quick Start](#quick-start)\n- [Manual Setup](#manual-setup)\n- [Project Structure](#project-structure)\n- [Architecture](#architecture)\n- [Database Schema](#database-schema)\n- [API Documentation](#api-documentation)\n- [CLI Usage](#cli-usage)\n- [Additional Information](#additional-information)\n- [Plugin Model](#plugin-model)\n- [Testing in Kubernetes with Kind](#testing-in-kubernetes-with-kind)\n- [Improvement](https://github.com/yindia/iteration/tree/postgres?tab=readme-ov-file#future-improvements)\n\n## Quick Start\nTo quickly get started with the Task Service, use the following command:\n\n```bash\nmake bootstrap\n\ndocker-compose -f docker-compose.demo.yaml up\n```\n\nThis will start all the necessary services, including the server, database, and worker.\n\n## Manual Setup\n\nFor a more detailed setup process, follow these steps:\n\n### 1. Development Environment Setup\nInstall Pixi and activate the shell:\n```bash\nmake bootstrap\n\n# Run Database \ndocker-compose up -d\n```\n\n\n### 2. Server (Control Plane)\nStart the server:\n```bash\nmake run-server\n```\nAccess at https://127.0.0.1:8080\n\n### 3. CLI Tool\nBuild and test:\n```bash\nmake build-cli\n./bin/task-cli --help\n```\n\n### 4. Dashboard Setup\nInstall dependencies and build:\n```bash\nnpm install\nnpm run dev\n```\nAccess at https://127.0.0.1:3000\n\n\u003cimg width=\"1548\" alt=\"Screenshot 2024-09-29 at 4 22 37 PM\" src=\"https://github.com/user-attachments/assets/7ec5f60a-e51d-44f7-bd55-9d9f761252a6\"\u003e\n\n### 5. Worker (Data Plane)\nStart worker instances:\n```bash\n./bin/task-cli serve  --log-level debug\n```\n\n## Project Structure\n```\ntask/\n├── cmd/\n│   ├── cli/            # CLI for task management\n│   └── server/         # Server entry point\n├── pkg/\n│   ├── config/         # Configuration management\n│   ├── gen/            # GRPC generated code\n│   ├── plugins/        # Plugin model\n│   ├── worker/         # Worker code\n│   └── x/              # Utility functions         \n├── idl/\n│   └── proto/          # Protocol buffer definitions\n├── clients/\n│   └── dashboard/      # NextJS Dashboard\n├── charts/\n│   └── task/         # Helm charts for deployment\n├── server/\n│   ├── repository/            # Database ORM\n│   └── root/         # Server Root \n│   └── route/         # All Server Routes\n└── docs/               # Documentation files\n```\n\n## Architecture\n\nThe Task Service follows a distributed architecture with separate components for the control plane and data plane. Here's a high-level overview of the system:\n\n```mermaid\ngraph TD\n    %% Clients\n    A[Dashboard Client] --\u003e|Sends Request| B(Server)\n    C[CLI Client] --\u003e|Sends Request| B(Server)\n\n    %% Control Plane\n    subgraph Control Plane\n        B(Server) --\u003e|Reads/Writes| D[(PostgreSQL Database)]\n    end\n\n    %% Data Plane\n    subgraph Data Plane\n        E[Agent] --\u003e|Initiates Connection| B[Server]\n        B[Server] --\u003e|Publish W| E[Agent]\n        E --\u003e|Creates CRD| H[CRD]\n        F[Controller] --\u003e|Watches CRD| H\n        F --\u003e|Executes Task| J[Task Execution]\n        F --\u003e|Sends Status Update| B\n    end\n```\n\nThis architecture allows for:\n- Separation of concerns between the control plane (server) and data plane (workers)\n- Scalability of worker nodes to handle increased workloads\n- Asynchronous task execution through message queuing\n- Real-time status updates from workers to the server\n\n\n### Database Operations\n\nThe server interacts with the database for persistent storage of tasks and their history. Here's a summary of the database operations:\n\n1. **Read Operations**\n   - Get Task by ID\n     - Purpose: Retrieve details of a specific task\n     - Frequency: On-demand, triggered by API requests\n   - List All Tasks\n     - Purpose: Retrieve a list of all tasks\n     - Frequency: On-demand, typically for dashboard or reporting\n   - List Task History\n     - Purpose: Retrieve the status change history of a specific task\n     - Frequency: On-demand, for detailed task analysis\n\n2. **Write Operations**\n   - Create New Task\n     - Purpose: Store a newly created task\n     - Frequency: Each time a new task is submitted\n   - Update Task Status\n     - Purpose: Modify the status of an existing task\n     - Frequency: As task states change (e.g., from queued to running to completed)\n   - Create Task History Entry\n     - Purpose: Log task status changes and creation events\n     - Frequency: On task creation and each status change\n\n\n### Database Schema\n\nThe Task Service uses a PostgreSQL database to store task and task history information. Below is an Entity-Relationship Diagram (ERD) representing the database schema:\n\n```mermaid\nerDiagram\n    %% Task Model\n    TASK {\n        int id PK\n        string name\n        int type\n        int status\n        jsonb payload\n        int retries\n        int priority\n        timestamp created_at\n    }\n    \n    %% TaskHistory Model\n    TASK_HISTORY {\n        int id PK\n        int task_id FK\n        int status\n        string details\n        timestamp created_at\n    }\n\n    %% Relationships\n    TASK ||--o{ TASK_HISTORY : has\n\n    %% Indexes (described as comments)\n    %% Indexes for TASK\n    %% - idx_type_status (type, status)\n    %% - idx_created_at (created_at)\n    %% - idx_status_created_at (status, created_at)\n\n    %% Indexes for TASK_HISTORY\n    %% - idx_task_id_created_at (task_id, created_at)\n```\n\nNote:  Ideally, we should create separate tables for tasks 📝 and task executions ⚙️. When a task is created, it should be added to the task table. Upon triggering an execution, a corresponding entry should be created in the execution table, and the execution data should be published to the PostgreSQL queue for processing 📬. This way, the task status remains unchanged, and only the execution status is updated in the execution table ✅.\n\n\n### Table Descriptions\n\n1. **TASK**\n   - Stores information about individual tasks\n   - `id`: Unique identifier for the task (Primary Key)\n   - `name`: Name of the task\n   - `type`: Type of the task (e.g., send_email, run_query)\n   - `status`: Current status of the task (e.g., pending, running, completed)\n   - `payload`: JSON object containing task-specific parameters\n   - `retries`: Number of retry attempts for the task\n   - `priority`: Priority level of the task\n   - `created_at`: Timestamp of task creation\n\n2. **TASK_HISTORY**\n   - Tracks the history of status changes for tasks\n   - `id`: Unique identifier for the history entry (Primary Key)\n   - `task_id`: Foreign Key referencing the TASK table\n   - `status`: Status of the task at the time of the history entry\n   - `details`: Additional details about the status change\n   - `created_at`: Timestamp of the history entry creation\n\n### Relationships\n\n- One TASK can have many TASK_HISTORY entries (one-to-many relationship)\n\n### Indexes\n\nTo optimize query performance, the following indexes are implemented:\n\n1. **TASK table**\n   - `idx_type_status`: Composite index on `type` and `status` columns\n   - `idx_created_at`: Index on `created_at` column\n   - `idx_status_created_at`: Composite index on `status` and `created_at` columns\n\n2. **TASK_HISTORY table**\n   - `idx_task_id_created_at`: Composite index on `task_id` and `created_at` columns\n\nThese indexes improve the efficiency of common queries such as filtering tasks by type and status, sorting by creation time, and retrieving task history.\n\n\n### Worker/Data Plane Process\n\nThe worker process follows a specific flow for task execution and error handling. Here's a detailed view of the worker's operation:\n\n```mermaid\ngraph TD\n    A[Receive Message] --\u003e B{Update Status: RUNNING}\n    B --\u003e|Success| C[Run Task]\n    B --\u003e|Failure| D[Log Error]\n    D --\u003e K[Move to Next Message]\n\n    C --\u003e E{Task Execution}\n    E --\u003e|Success| F[Update Status: SUCCEEDED]\n    E --\u003e|Failure| G[Retry Logic]\n\n    G --\u003e H{Retry Attempt \u003c= 3?}\n    H --\u003e|Yes| I[Backoff]\n    I --\u003e J[Update Status: RETRYING]\n    J --\u003e C\n    H --\u003e|No| K[Update Status: FAILED]\n\n    F --\u003e L[Move to Next Message]\n    K --\u003e L\n```\n\n\n## API Documentation\n- [Proto Docs](https://buf.build/evalsocket/cloud)\n- [Studio](https://buf.build/studio/evalsocket/cloud/cloud.v1.TaskManagementService/CreateTask)\n\n## CLI Usage\n\n### Task Management\n\nThe Task Service CLI provides several commands to manage tasks. Here's a detailed overview of each command and its available flags:\n\n#### Create a Task\n\nCreate a new task with the specified name, type, and parameters.\n\n```bash\ntask-cli task create [task name] --type [task type] --parameter [key=value]\n```\n\nFlags:\n- `--type`, `-t`: Type of the task (e.g., send_email, run_query)\n- `--parameter`, `-p`: Additional parameters for the task as key=value pairs (can be used multiple times)\n\nExample:\n```bash\ntask-cli task create \"Send Newsletter\" --type send_email --parameter recipient=user@example.com --parameter subject=\"Weekly Update\"\n```\n\n#### Get Task Details\n\nRetrieve and display the details of a specific task by its ID.\n\n```bash\ntask-cli  task get --id [task ID] [flags]\n```\n\nFlags:\n- `--id`, `-i`: ID of the task (required)\n- `--output`, `-o`: Output format (table, json, yaml) (default: \"table\")\n\nExample:\n```bash\ntask-cli  task get --id 123 --output json\n```\n\n#### Get Task History\n\nRetrieve and display the history of a specific task by its ID.\n\n```bash\ntask-cli history --id [task ID] [flags]\n```\n\nFlags:\n- `--id`, `-i`: ID of the task (required)\n- `--output`, `-o`: Output format (table, json, yaml) (default: \"table\")\n\nExample:\n```bash\ntask-cli  history --id 123 --output yaml\n```\n\n#### List All Tasks\n\nRetrieve and display a list of all tasks.\n\n```bash\ntask-cli task list [flags]\n```\n\nFlags:\n- `--output`, `-o`: Output format (table, json, yaml) (default: \"table\")\n- `--pageNumber`, `-n`: Page number for pagination (default: 1)\n- `--pageCount`, `-c`: Number of items per page (default: 30)\n\nExamples:\n```bash\ntask-cli task list\ntask-cli task list --output json\ntask-cli task list --pageNumber 2 --pageCount 20\n```\n\n#### Task Status\n\nRetrieve the status counts of all tasks in the system.\n\n```bash\ntask-cli task status\n```\n\nAliases: `s`, `stat`\n\nExample:\n```bash\ntask-cli task status\ntask-cli task s\n```\n\nThis command will display the count of tasks for each status (e.g., PENDING, RUNNING, SUCCEEDED, FAILED).\n\n#### End-to-End Testing\n\nRun end-to-end tests against the system to verify its functionality.\n\n```bash\ntask-cli end2end [flags]\n```\n\nFlags:\n- `--num-tasks`, `-n`: Number of tasks to create for the test (default: 100, max: 100)\n\nExample:\n```bash\ntask-cli end2end\ntask-cli end2end -n 50\n```\n\nThis command will:\n1. Create the specified number of tasks (default 100)\n2. Monitor the tasks' completion status for up to 3 minutes\n3. Display progress every 5 seconds\n4. Report the final result (success or partial completion)\n\nThe test creates a mix of \"run_query\" and \"send_email\" task types to simulate a realistic workload.\n\n\n### Global Flags\n\nThe following flag is available for all task commands:\n\n- `--log-level`: Set the logging level (default: \"error\")\n- `--address`: Control Plane Address (default: \"http://127.0.0.1:8080\")\n\nExample:\n```bash\ntask-cli task list --log-level debug\n```\n\n### Output Formats\n\nAll commands that display task information support three output formats:\n\n- `table`: Displays the information in a formatted table (default)\n- `json`: Outputs the data in JSON format\n- `yaml`: Outputs the data in YAML format\n\nUse the `--output` or `-o` flag to specify the desired format.\n\n## Additional Information\n- Control plane (server) manages task creation, scheduling, and status updates\n- Data plane (workers) executes tasks (Currently part of same binary)\n- RiverQueue used for communication between control and data planes using postgres as queue backend\n- Explore the UI or CLI to create and manage tasks\n\n## Plugin Model\n\nThe Task Service uses a plugin-based architecture to allow for extensibility and customization of task execution. This model enables users to create their own task types and implement custom logic for task execution.\n\n### How It Works\n\n1. **Plugin Interface**: All plugins must implement the `Plugin` interface defined in `@pkg/plugins/plugins.go`. This interface requires a `Run` method:\n\n   ```go\n   type Plugin interface {\n       Run(parameters map[string]string) error\n   }\n   ```\n\n2. **Plugin Registration**: Plugins are registered in the `NewPlugin` function in `@pkg/plugins/plugins.go`. This function acts as a factory, creating the appropriate plugin based on the task type:\n\n   ```go\n   func NewPlugin(pluginType string) (Plugin, error) {\n       switch pluginType {\n       case email.PLUGIN_NAME:\n           return \u0026email.Email{}, nil\n       case query.PLUGIN_NAME:\n           return \u0026query.Query{}, nil\n       // Add more plugin types here\n       default:\n           return nil, fmt.Errorf(\"unknown plugin type: %s\", pluginType)\n       }\n   }\n   ```\n\n3. **Custom Plugin Implementation**: Users can create their own plugins by implementing the `Plugin` interface. For example, the `Email` plugin in `@pkg/email/email.go`:\n\n   ```go\n   var PLUGIN_NAME = \"send_email\"\n   type Email struct {}\n\n   func (e *Email) Run(parameters map[string]string) error {\n       // Implementation of email sending logic\n       return nil\n   }\n   ```\n\n4. **Task Execution**: When a task is executed, the system uses the `NewPlugin` function to create the appropriate plugin based on the task type. It then calls the `Run` method of the plugin, passing any necessary parameters.\n\n### Creating a New Plugin\n\nTo create a new plugin:\n\n1. Create a new package in the `@pkg/plugins` directory for your plugin.\n2. Implement the `Plugin` interface in your new package.\n3. Add your plugin to the `NewPlugin` function in `@pkg/plugins/plugins.go`.\n\nThis modular approach allows for easy extension of the Task Service with new task types and functionalities.\n\n\n## Testing in Kubernetes with Kind\n\nThis section guides you through setting up and testing the Task Service in a local Kubernetes cluster using Kind (Kubernetes in Docker) and Helm charts.\n\n### Prerequisites\n\n- [Docker](https://docs.docker.com/get-docker/)\n- [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation)\n- [kubectl](https://kubernetes.io/docs/tasks/tools/)\n- [Helm](https://helm.sh/docs/intro/install/)\n\n### Setup\n\n1. Create a Kind cluster:\n\n```bash\nkind create cluster --name task-service\n```\n\n2. Set kubectl context to the new cluster:\n\n```bash\nkubectl cluster-info --context kind-task-service\n```\n\n3. Add the necessary Helm repositories:\n\n```bash\nmake helm\n```\n\n### Deploy Task Service\n\nInstall the Task Service Helm chart:\n\n```bash\nhelm install task-service ./charts/task  -n task\n```\n\n### Verify Deployment\n\nCheck that all pods are running:\n\n```bash\nkubectl get pods\n```\n\n## Port Forward\n```\nkubectl port-forward service/task 80 -n task\n```\n\n### Access the Service\n\n1. Port-forward the Task Service:\n\n```bash\nkubectl port-forward -n task svc/task 8080:80\n```\n\n2. Access the service at `http://127.0.0.1:8080`\n\n3. Use CLI to verify the connection:\n```bash\n./bin/task-cli task l --address http://127.0.0.1:8080\n```\n\n### Clean Up\n\nTo delete the Kind cluster and all resources:\n\n```bash\nkind delete cluster --name task-service\n```\n\nThis setup allows you to test the entire Task Service stack, including the server, workers, and dependencies, in a local Kubernetes environment. It's an excellent way to validate the Helm charts and ensure everything works together as expected in a Kubernetes setting.\n\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyindia%2Fcloud","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyindia%2Fcloud","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyindia%2Fcloud/lists"}