{"id":26054783,"url":"https://github.com/davidandw190/data-locality-scheduler","last_synced_at":"2026-05-08T02:15:49.765Z","repository":{"id":281257809,"uuid":"944465237","full_name":"davidandw190/data-locality-scheduler","owner":"davidandw190","description":"A work-in-progress, Knative-compatible, K8s scheduler designed for data-intensive workflows across geo-distributed, heterogeneous edge-cloud infrastructures, optimizing data transfer consts against computational migration overhead through adaptive locality-aware placement decisions.","archived":false,"fork":false,"pushed_at":"2025-06-15T17:17:38.000Z","size":5557,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-15T17:25:44.419Z","etag":null,"topics":["cloud-native","data-locality","edge-cloud","edge-computing","faas","go","k8s","knative","kubernetes","scheduler"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davidandw190.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-03-07T11:47:43.000Z","updated_at":"2025-06-15T17:17:44.000Z","dependencies_parsed_at":"2025-12-08T02:03:03.324Z","dependency_job_id":null,"html_url":"https://github.com/davidandw190/data-locality-scheduler","commit_stats":null,"previous_names":["davidandw190/data-locality-scheduler"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/davidandw190/data-locality-scheduler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidandw190%2Fdata-locality-scheduler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidandw190%2Fdata-locality-scheduler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidandw190%2Fdata-locality-scheduler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidandw190%2Fdata-locality-scheduler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davidandw190","download_url":"https://codeload.github.com/davidandw190/data-locality-scheduler/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidandw190%2Fdata-locality-scheduler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32763529,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-07T02:14:30.463Z","status":"online","status_checked_at":"2026-05-08T02:00:05.879Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud-native","data-locality","edge-cloud","edge-computing","faas","go","k8s","knative","kubernetes","scheduler"],"created_at":"2025-03-08T09:59:54.543Z","updated_at":"2026-05-08T02:15:49.746Z","avatar_url":"https://github.com/davidandw190.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data-Locality-Aware Kubernetes Scheduler\n\n\u003e **Bachelor's Thesis Project**  \n\u003e *Data-Locality-Aware Scheduling for Serverless Containerized Workflows across the Edge-Cloud Continuum*  \n\u003e **Author:** Andrei-David Nan  \n\u003e **Supervisor:** Lect. Dr. Adrian Spătaru  \n\u003e **Institution:** West University of Timișoara, Faculty of Mathematics and Computer Science\n\n## Overview\n\nThis project implements a data-locality-aware scheduler extension for Kubernetes that optimizes containerized workload placement across edge-cloud environments. The scheduler uses a Multi-Criteria Decision Making (MCDM) algorithm to balance resource availability, node capabilities, and data locality when making scheduling decisions.\n\n## Architecture\n\nThe system consists of three main components:\n\n### 1. Data-Locality-Aware Scheduler (`cmd/scheduler`)\n- Extends Kubernetes scheduler with MCDM algorithm\n- Maintains Bandwidth Graph and Data Index for locality decisions\n- Registers as alternative scheduler (`schedulerName: \"data-locality-scheduler\"`)\n- Implements dynamic priotity function weight adjustment based on workload classification\n\n### 2. Node Capability Daemon (`cmd/node-daemon`)\n- Runs as DaemonSet across all cluster nodes\n- Detects hardware capabilities and storage characteristics\n\n- Updates node labels with capability information\n- Discovers local data objects (MinIO buckets, volumes), and works together with the Storage Index of the scheduler which accounts for data item placements. \n\n### 3. Knative Integration Webhook (`integration/knative`)\n- Intercepts Knative service deployments\n- Enriches serverless functions with scheduling annotations\n- Enables data-locality optimization for FaaS workloads\n\n## Project Structure\n\n```bash\n├── cmd/                    \n│   ├── node-daemon/        # Node capability detection daemon\n│   └── scheduler/          # Data-locality-aware scheduler\n├── pkg/                    \n│   ├── daemon/             # Node daemon implementation\n│   ├── scheduler/          # Scheduler algorithms and logic\n│   └── storage/            # Storage abstraction and indexing\n├── deployments/            \n│   ├── 00-core/           # Scheduler and daemon deployments\n│   ├── 01-storage/        # MinIO storage setup\n│   ├── 02-test/           # Test workload definitions\n│   └── 03-validation/     # Validation and stress tests\n├── benchmarks/            \n├── integration/knative/   # Knative serverless integration\n├── config/               # Configuration files\n└── build/                # Docker build files\n```\n\n## Quick Start\n\n### Prerequisites\n\n- Kubernetes cluster (v1.23+)\n- `kubectl` configured\n- Docker\n\n\n### Clone the Repository\n\n```bash\n# Clone the data locality scheduler project\ngit clone github.com/davidandw190/data-locality-aware-scheduler\ncd data-locality-aware-scheduler/\n\n# Navigate to the benchmarker framework\ncd benchmarks/simulated\n```\n\n\n\n\n\n### Install Dependencies\n\n```python\n# Install Python dependencies \npip install -r requirements.txt\n\n# Check framework help\npython benchmark_runner.py --help\n```\n\n### Basic Deployment\n\n1. **Deploy core components:**\n   ```bash\n   kubectl apply -f deployments/00-core/\n    ```   \n2. **Set up storage (optional)**\n\n    ```bash\n    kubectl apply -f deployments/01-storage/\n    ```\n\n\n### Using the scheduler\n\nSpecify the scheduler in your pod spec:\n\n```bash\napiVersion: v1\nkind: Pod\nmetadata:\n  name: my-workload\nspec:\n  schedulerName: \"data-locality-scheduler\"\n  containers:\n  - name: app\n    image: my-app:latest\n```\n\n## Configuration\n\nThe scheduler behavior can be customized via `config/scheduler-config.yaml`:\n\n```yaml\nscheduler:\n  weights:\n    default:\n      resource: 0.20\n      affinity: 0.10\n      nodeType: 0.15\n      capabilities: 0.15\n      dataLocality: 0.40\n    dataIntensive:\n      dataLocality: 0.70\n      resource: 0.10\n```\n\n## Knative Integration\n\nFor serverless workloads using Knative:\n\n1. **Deploy the webhook**:\n\n  ```bash\n    kubectl apply -f integration/knative/manifests/\n  ```\n2. **Deploy Knative services normally** - they will automatically use the data-locality scheduler:\n\n  ```bash\n    kubectl apply -f - \u003c\u003cEOF\n      apiVersion: serving.knative.dev/v1\n      kind: Service\n      metadata:\n        name: eo-fmask-processor\n        namespace: default\n        annotations:\n          scheduler.thesis/data-sources: \"landsat-l1,landsat-l2\"\n          scheduler.thesis/workload-type: \"compute-intensive\"\n      spec:\n        template:\n          metadata:\n            annotations:\n              autoscaling.knative.dev/target: \"5\"\n          spec:\n            containers:\n            - name: fmask-processor\n              image: my-registry/eo-fmask:latest\n              ports:\n              - containerPort: 8080\n              env:\n              - name: MINIO_ENDPOINT\n                value: \"minio.storage.svc.cluster.local:9000\"\n              - name: LANDSAT_BUCKET\n                value: \"landsat-l1\"\n              resources:\n                requests:\n                  memory: \"2Gi\"\n                  cpu: \"1000m\"\n                limits:\n                  memory: \"4Gi\"\n                  cpu: \"2000m\"\n    EOF\n  ```\n\n\n3. **Trigger the function with a CloudEvent** containing data dependencies:\n\n```bash\n  curl -X POST http://eo-fmask-processor.default.example.com \\\n    -H \"Content-Type: application/json\" \\\n    -H \"Ce-Specversion: 1.0\" \\\n    -H \"Ce-Type: eo.processing.fmask.request\" \\\n    -H \"Ce-Source: eo-pipeline/landsat-processor\" \\\n    -H \"Ce-Id: fmask-LC08-039037-20241215\" \\\n    -H \"Ce-Subject: landsat-cloud-masking\" \\\n    -d '{\n      \"specversion\": \"1.0\",\n      \"type\": \"eo.processing.fmask.request\",\n      \"source\": \"eo-pipeline/landsat-processor\",\n      \"id\": \"fmask-LC08-039037-20241215\",\n      \"data\": {\n        \"dataInputs\": [\n          {\n            \"urn\": \"landsat-l1/LC08_L1TP_B1.tif\",\n            \"size\": 45097156,\n            \"processingTime\": 30\n          },\n          {\n            \"urn\": \"landsat-l1/LC08_L1TP_B10.tif\",\n            \"size\": 22548578\n          }\n        ],\n        \"processingOptions\": {\n          \"computeIntensive\": true,\n          \"preferRegion\": \"region-1\"\n        }\n      }\n    }'\n```\n\n\n## Benchmarking\n\nThis project includes a benchmarking framework for evaluating scheduler performance across different workload scenarios.\n\n\n**For detailed benchmarking instructions, see:** [`benchmarks/README.md`](benchmarks/README.md)\n\n### Quick Benchmark Run\n\n```bash\ncd benchmarks/simulated/framework\npython benchmark_runner.py --config benchmark_config.yaml\n```\n\nThe benchmarking framework supports:\n- Multiple workload types (ETL, image processing, stream processing, etc.)\n- Comparative analysis vs. default Kubernetes scheduler\n- Data locality and network topology impact analysis\n- Performance metrics and visualizations\n\n## Development\n\n### Building from Source\n\n```bash\n# Build scheduler\ndocker build -f build/Dockerfile.scheduler -t data-locality-scheduler .\n\n# Build node daemon  \ndocker build -f build/Dockerfile.daemon -t node-capability-daemon .\n```\n\n### Running Tests\n\n```bash\n# Apply validation tests\nkubectl apply -f deployments/03-validation/\n\n# Monitor results\nkubectl logs -f deployment/scheduler-validator\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidandw190%2Fdata-locality-scheduler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidandw190%2Fdata-locality-scheduler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidandw190%2Fdata-locality-scheduler/lists"}