{"id":21657258,"url":"https://github.com/zcubbs/spark","last_synced_at":"2026-04-24T20:33:03.858Z","repository":{"id":239764872,"uuid":"797499924","full_name":"zcubbs/spark","owner":"zcubbs","description":"Spark streamlines the process of job execution within a Kubernetes cluster by providing a set of endpoints to schedule, track, and manage jobs programmatically","archived":false,"fork":false,"pushed_at":"2024-05-12T23:08:07.000Z","size":1082,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-20T01:50:06.567Z","etag":null,"topics":["jobs","k8s","scheduler"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zcubbs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-08T00:50:43.000Z","updated_at":"2024-05-14T12:12:41.000Z","dependencies_parsed_at":"2024-05-14T14:43:43.582Z","dependency_job_id":null,"html_url":"https://github.com/zcubbs/spark","commit_stats":null,"previous_names":["zcubbs/spark"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/zcubbs/spark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zcubbs%2Fspark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zcubbs%2Fspark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zcubbs%2Fspark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zcubbs%2Fspark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zcubbs","download_url":"https://codeload.github.com/zcubbs/spark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zcubbs%2Fspark/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261542842,"owners_count":23174644,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["jobs","k8s","scheduler"],"created_at":"2024-11-25T09:20:14.249Z","updated_at":"2026-04-24T20:32:58.838Z","avatar_url":"https://github.com/zcubbs.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# `S P A R K`\n\n\u003e Disclaimer: This project is a work in progress and is not yet ready for production use. Please check back later for updates.\n\n`Spark` streamlines the process of job execution within a Kubernetes cluster by providing a set of endpoints to schedule, track, and manage jobs programmatically. With integrated job tracking and logging capabilities, making it ideal for environments that require dynamic job scheduling and comprehensive monitoring.\n\n![](./docs/assets/splash.png)\n\n## Features\n\n- **Dynamic Job Scheduling**: Automate the deployment of Kubernetes jobs, leveraging a structured approach to define and manage job specifications such as image, commands, and necessary configurations.\n- **Concurrency Management**: Control and limit the number of jobs that can run concurrently, allowing for effective resource utilization and system stability.\n- **Task Queuing System**: Utilize an internal queuing system to manage job tasks, ensuring that job submissions are handled efficiently and executed in order.\n- **Comprehensive Monitoring**: Continuously monitor the status of each job, capturing and reacting to job completions, failures, and timeouts in real-time.\n- **Log Retrieval and Storage**: Automatically fetch and store logs from job executions, providing immediate access to job outputs for debugging and verification purposes.\n- **Rate Limiting and Timeouts**: Implement client-side rate limiting and configurable timeouts to manage the load on the Kubernetes API and ensure jobs complete within expected time frames.\n- **Local Persistence**: Using BuntDB for fast, in-memory data storage to keep track of job statuses and logs, ensuring data persistence across job operations.\n\n## Usage Scenarios\n\n- **Data processing applications**: Managing batch jobs for data transformation, analysis, or MLM training.\n- **General automation**: Running maintenance scripts, backups, and other periodic tasks within a Kubernetes cluster.\n- **CI/CD pipelines**: Automating deployment tasks, testing, and other operations that can be encapsulated as Kubernetes jobs.\n\n## The Runner `k8sJobs` Package\n\nThe -Runner- `k8sJobs` package is a Go library for managing Kubernetes jobs. It provides functionalities for creating, monitoring, and deleting jobs, managing concurrency, and maintaining a record of job statuses and logs in a local BuntDB database.\n\n### Features\n\n- **Dynamic Job Management**: Create and monitor Kubernetes jobs dynamically within your application.\n- **Concurrency Control**: Manage multiple jobs concurrently with a configurable limit.\n- **Task Queuing**: Queue tasks with a channel-based mechanism.\n- **Local Persistence**: Utilize BuntDB to store job statuses and logs.\n- **Timeout Handling**: Automatically handle job execution with configurable timeouts.\n- **Error Handling**: Robust error handling throughout the job lifecycle.\n\n### Structure\n\n#### Runner Struct\n\nOrchestrates job tasks and interacts with the Kubernetes API.\n\n- **Fields**:\n    - `cs`: Kubernetes ClientSet to interact with Kubernetes API.\n    - `maxConcurrentJobs`: Maximum number of jobs that can run concurrently.\n    - `taskChan`: Channel for queuing tasks.\n    - `quit`: Channel to signal the shutdown of task processing.\n    - `namespace`: Namespace in Kubernetes where jobs are deployed.\n    - `db`: BuntDB instance for local data storage.\n\n#### Task Struct\n\nDefines the structure of a job task.\n\n- **Fields**:\n    - `ID`: Unique identifier of the task.\n    - `Command`: Docker container command.\n    - `Image`: Docker image used for the job.\n    - `Timeout`: Execution timeout in seconds.\n    - `Status`: Current status of the task.\n    - `Logs`: Logs generated by the task.\n    - `CreatedAt`: Timestamp of task creation.\n    - `StartedAt`: Timestamp of task start.\n    - `CompletedAt`: Timestamp of task completion.\n\n### Usage Example\n\n```go\n// You can replace \"default\" with the desired path \n// or pass an empty string for the current namespace if in-cluster.\nrunner, err := k8sJobs.New(context.Background(), \n    \"default\", // kubernetes config location\n    5, // max concurrent jobs\n    100, // max job queue size\n    60, // default job timeout\n)\nif err != nil {\n    log.Fatalf(\"Failed to start runner: %v\", err)\n}\n\ntask := k8sJobs.Task{\n    ID:      \"unique-job-id\",\n    Command: []string{\"echo\", \"Hello World\"},\n    Image:   \"busybox\",\n    Timeout: 30,\n}\n\nif err := runner.AddTask(task); err != nil {\n    log.Errorf(\"Failed to add task: %v\", err)\n}\n\n// Perform additional operations...\n\nrunner.Shutdown()\n```\n\n## Dashboard\n\nThe dashboard provides a minimalist \u0026 quick read-only access to the job statuses and logs.\n\n![](./docs/assets/dashboard.png)\n\n## Development\n\n### CLI Commands\n\nThe CLI provides a set of commands to interact with the `Spark` system. To get started, run the following command:\n\n```bash\ngo run .\\cmd\\cli\\main.go --mode rest --image \"busybox\" --cmd \"echo Hello World\" --timeout 60\n```\n\nThe command above will create a new job with the specified image, command, and timeout. You can also use the following flags to customize the job:\n\n- `--mode`: The mode of operation (`rest` or `grpc`).\n- `--image`: The Docker image to use for the job.\n- `--cmd`: The command to run inside the container.\n- `--timeout`: The maximum duration for the job to complete.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzcubbs%2Fspark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzcubbs%2Fspark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzcubbs%2Fspark/lists"}