{"id":36617704,"url":"https://github.com/dream-horizon-org/darwin","last_synced_at":"2026-01-12T09:14:44.672Z","repository":{"id":327027086,"uuid":"1105886938","full_name":"dream-horizon-org/darwin","owner":"dream-horizon-org","description":"ML Platform at Dream11","archived":false,"fork":false,"pushed_at":"2025-12-30T12:37:54.000Z","size":37714,"stargazers_count":59,"open_issues_count":7,"forks_count":15,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-01T02:53:07.914Z","etag":null,"topics":["ai","ml","mlops","workflow"],"latest_commit_sha":null,"homepage":"https://darwin.dreamhorizon.org/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dream-horizon-org.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-28T09:54:55.000Z","updated_at":"2025-12-29T14:46:42.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/dream-horizon-org/darwin","commit_stats":null,"previous_names":["ds-horizon/darwin"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dream-horizon-org/darwin","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dream-horizon-org%2Fdarwin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dream-horizon-org%2Fdarwin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dream-horizon-org%2Fdarwin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dream-horizon-org%2Fdarwin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dream-horizon-org","download_url":"https://codeload.github.com/dream-horizon-org/darwin/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dream-horizon-org%2Fdarwin/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28337656,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T06:09:07.588Z","status":"ssl_error","status_checked_at":"2026-01-12T06:05:18.301Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ml","mlops","workflow"],"created_at":"2026-01-12T09:14:43.060Z","updated_at":"2026-01-12T09:14:44.658Z","avatar_url":"https://github.com/dream-horizon-org.png","language":"Python","funding_links":[],"categories":["人工智能"],"sub_categories":["机器学习"],"readme":"# Darwin ML Platform\n\n🌐 **[Visit Darwin Platform](https://darwin.dreamhorizon.org/)**\n\n**Darwin** is an enterprise-grade, end-to-end machine learning platform designed for production-scale AI/ML workloads. It provides a unified ecosystem for the complete ML lifecycle—from distributed compute and feature engineering to experiment tracking, model deployment, and real-time inference serving.\n\n---\n\n## 🎯 Why Darwin?\n\nDarwin solves critical challenges in production ML infrastructure:\n\n- **Unified Platform**: Single platform for training, serving, and feature engineering—no context switching between disparate tools\n- **Production-Grade Scalability**: Built on Kubernetes and Ray for elastic, distributed compute at scale\n- **Cost Optimization**: Intelligent auto-scaling, spot instance support, and policy-based auto-termination\n- **Developer Velocity**: SDK-first design with CLI tools for rapid experimentation and deployment\n- **Enterprise Ready**: Multi-tenancy, RBAC, audit logging, and metadata lineage out of the box\n- **Low-Latency Serving**: Sub-10ms feature retrieval and optimized model inference pipelines\n\n---\n\n## 🏗️ Architecture Overview\n\n```mermaid\ngraph TB\n    subgraph \"User Interface Layer\"\n        UI[Workspace UI/Jupyter]\n        CLI[Hermes CLI]\n        SDK[Python SDKs]\n    end\n\n    subgraph \"Orchestration Layer\"\n        Workspace[Workspace Service\u003cbr/\u003eProjects \u0026 Codespaces]\n        MLflow[MLflow\u003cbr/\u003eExperiment Tracking]\n        Chronos[Chronos\u003cbr/\u003eEvent \u0026 Metadata]\n    end\n\n    subgraph \"Compute Layer\"\n        Compute[Darwin Compute\u003cbr/\u003eCluster Management]\n        DCM[Darwin Cluster Manager\u003cbr/\u003eK8s Orchestration]\n        Ray[Ray Clusters\u003cbr/\u003eDistributed Execution]\n    end\n\n    subgraph \"Data Layer\"\n        FS[Feature Store\u003cbr/\u003eOnline/Offline Features]\n        Catalog[Darwin Catalog\u003cbr/\u003eAsset Discovery]\n    end\n\n    subgraph \"Serving Layer\"\n        Serve[ML Serve\u003cbr/\u003eModel Deployment]\n        Builder[Artifact Builder\u003cbr/\u003eImage Building]\n    end\n\n    subgraph \"Infrastructure\"\n        MySQL[(MySQL\u003cbr/\u003eMetadata)]\n        Cassandra[(Cassandra\u003cbr/\u003eFeatures)]\n        OpenSearch[(OpenSearch\u003cbr/\u003eEvents)]\n        S3[(S3\u003cbr/\u003eArtifacts)]\n        Kafka[(Kafka\u003cbr/\u003eStreaming)]\n        K8s[Kubernetes/EKS]\n    end\n\n    UI --\u003e Workspace\n    CLI --\u003e Serve\n    CLI --\u003e Compute\n    SDK --\u003e Compute\n    SDK --\u003e FS\n    SDK --\u003e MLflow\n\n    Workspace --\u003e Compute\n    Workspace --\u003e Chronos\n    MLflow --\u003e S3\n    MLflow --\u003e MySQL\n\n    Compute --\u003e DCM\n    DCM --\u003e Ray\n    Ray --\u003e K8s\n\n    Serve --\u003e Builder\n    Serve --\u003e DCM\n    Builder --\u003e K8s\n\n    FS --\u003e Cassandra\n    FS --\u003e Kafka\n    FS --\u003e MySQL\n    Catalog --\u003e MySQL\n    Catalog --\u003e OpenSearch\n    Chronos --\u003e OpenSearch\n    Chronos --\u003e Kafka\n\n    style Darwin fill:#e1f5ff\n    style Compute fill:#ffe1e1\n    style FS fill:#e1ffe1\n    style Serve fill:#fff5e1\n```\n\n---\n\n## 📦 Platform Components\n\n### 1. Darwin Compute\n**Distributed compute orchestration for ML workloads**\n\n- **Ray Cluster Management**: Create, scale, and manage Ray 2.37.0 clusters on Kubernetes\n- **Multi-Runtime Support**: Pre-configured runtimes (Ray + Python 3.10 + Spark 3.5.1)\n- **Resource Optimization**: \n  - Spot/on-demand instance mixing\n  - Auto-termination policies (idle detection, CPU thresholds)\n  - Cost monitoring with Slack alerts\n- **Package Management**: Dynamic installation of PyPI, Maven, and workspace packages\n- **Jupyter Integration**: Managed Jupyter notebooks with direct cluster access\n- **Job Scheduling**: Ray job submission and monitoring\n\n**SDK**: `darwin-compute`\n```python\nfrom darwin_compute import ComputeCluster\n\ncluster = ComputeCluster(env=\"prod\")\nresult = cluster.create_with_yaml(\"cluster-config.yaml\")\ncluster.start(cluster_id=result['cluster_id'])\n```\n\n---\n\n### 2. Darwin Cluster Manager (DCM)\n**Low-level Kubernetes orchestration service (Go)**\n\n- Helm-based Ray cluster deployment via KubeRay operator\n- Dynamic values.yaml generation for cluster configurations\n- Remote command execution on cluster pods\n- Jupyter pod lifecycle management\n- FastAPI serve deployment orchestration\n\n---\n\n### 3. Feature Store\n**High-performance feature serving and engineering platform**\n\n**Components**:\n- **darwin-ofs-v2** (App): Low-latency online feature serving (\u003c10ms)\n- **darwin-ofs-v2-admin**: Feature group management, schema versioning\n- **darwin-ofs-v2-consumer**: Kafka-based feature materialization\n- **darwin-ofs-v2-populator**: Bulk ingestion from Parquet/Delta tables\n\n**Capabilities**:\n- Real-time feature retrieval with Cassandra backend\n- Point-in-time correctness for training datasets\n- Feature validation and schema evolution\n- Spark integration for batch feature pipelines\n- Multi-tenant feature isolation\n\n**Storage Architecture**:\n- **Cassandra**: High-throughput feature values\n- **MySQL**: Feature metadata and schemas\n- **Kafka**: Real-time feature streaming\n\n**SDK**: `darwin_fs`\n```python\nfrom darwin_fs import FeatureStoreClient\n\nfs = FeatureStoreClient()\nfeatures = fs.fetch_features(\n    feature_group=\"user_engagement\",\n    keys=[123, 456]\n)\n```\n\n---\n\n### 4. ML Serve\n**Production model deployment and serving platform**\n\n- **Serve Lifecycle**: Create, configure, deploy, monitor, undeploy\n- **Multi-Environment**: Dev, staging, UAT, production with environment-specific configs\n- **Backend Support**: \n  - FastAPI serves for REST inference\n  - Ray Serve for distributed model serving (experimental)\n- **Artifact Management**: Git-based Docker image builds\n- **Auto-Scaling**: HPA-based horizontal pod autoscaling\n- **Feature Store Integration**: Native integration for online feature retrieval\n\n**Deployment Workflow**:\n```bash\n# Complete model deployment via Hermes CLI\n\n# 1. Configure authentication\nexport HERMES_USER_TOKEN=admin-token-default-change-in-production\nhermes configure\n\n# 2. Create environment (one-time setup)\nhermes create-environment --name local --domain-suffix .local --cluster-name kind\n\n# 3. Create serve definition\nhermes create-serve --name my-model --type api --space serve --description \"My ML model\"\n\n# 4. Deploy model\nhermes deploy-model \\\n  --serve-name my-model \\\n  --artifact-version v1 \\\n  --model-uri mlflow-artifacts:/1/abc123/artifacts/model \\\n  --cores 4 \\\n  --memory 8 \\\n  --node-capacity spot \\\n  --min-replicas 2 \\\n  --max-replicas 10\n```\n\n\u003e **📖 For detailed Hermes CLI commands and options, see [hermes-cli/CLI.md](hermes-cli/CLI.md)**\n\n---\n\n### 5. Artifact Builder\n**Docker image building service for ML models**\n\n- Build images from GitHub repositories with custom Dockerfiles\n- Queue-based build system with status tracking\n- Container registry integration (ECR, GCR)\n- Integration with ML Serve deployment pipeline\n\n---\n\n### 6. Darwin MLflow\n**Experiment tracking and model registry**\n\n- **MLflow 2.12.2** with custom FastAPI authentication layer\n- Experiment and run tracking (parameters, metrics, artifacts)\n- Model registry with versioning\n- User-based experiment permissions\n- S3/LocalStack artifact storage\n- Custom UI with enhanced authorization\n\n**SDK**: `darwin_mlflow` (wraps MLflow client)\n```python\nimport darwin_mlflow as mlflow\n\nmlflow.log_params({\"lr\": 0.001, \"epochs\": 100})\nmlflow.log_metric(\"accuracy\", 0.95)\nmlflow.sklearn.log_model(model, \"model\")\n```\n\n---\n\n### 7. Chronos (Event Processing \u0026 Metadata)\n**Event ingestion, transformation, and lineage tracking**\n\n- **Event Sources**: REST API for raw events from services\n- **Transformers**: Python/JSONPath-based event processing\n- **Entity Extraction**: Automatic entity creation (clusters, users, jobs)\n- **Relationship Mapping**: Build lineage graphs between entities\n- **Queue Processing**: Async consumption from Kafka/SQS\n\n**Use Cases**:\n- Cluster lifecycle tracking\n- Workflow execution lineage\n- Audit logs and compliance\n- Metadata dependencies (data → model → deployment)\n\n---\n\n### 8. Darwin Workspace\n**Project and development environment management**\n\n- **Project Management**: Multi-user project organization\n- **Codespace Lifecycle**: Create and manage Jupyter/VSCode environments\n- **Compute Integration**: Attach Ray clusters to development environments\n- **Shared Storage**: FSx/EFS integration for persistent workspaces\n- **Event Publishing**: Workspace state changes tracked in Chronos\n\n---\n\n### 9. Darwin Catalog\n**Data asset discovery and governance**\n\n- **Asset Management**: Register datasets, tables, models\n- **Schema Tracking**: Schema evolution and versioning\n- **Lineage**: OpenLineage-based data lineage tracking\n- **Search**: Full-text search across data assets\n- **Metadata**: Tags, descriptions, ownership, quality metrics\n- **Integration**: Spark and Airflow job lineage capture\n\n---\n\n### 10. Hermes CLI\n**Command-line tool for streamlined ML operations**\n\n- Environment configuration and management\n- Model serving project scaffolding (FastAPI templates)\n- One-click model deployment\n- Artifact build and deploy orchestration\n- Configuration management (`.hermes` folder)\n\n**Installation \u0026 Setup**:\n```bash\n# Included with Darwin Distribution\nsource hermes-cli/.venv/bin/activate\n\n# Configure authentication\nexport HERMES_USER_TOKEN=admin-token-default-change-in-production\nhermes configure\n\n# Create environment (one-time setup per environment)\nhermes create-environment \\\n  --name local \\\n  --domain-suffix .local \\\n  --cluster-name kind \\\n  --namespace serve\n```\n\n\u003e **📖 For complete Hermes CLI documentation, see [hermes-cli/CLI.md](hermes-cli/CLI.md)**\n\n---\n\n## 👥 User Personas\n\n### Data Scientists\n**Use Darwin for**: Experimentation, training, model development\n- Launch Ray clusters via SDK for distributed training\n- Track experiments with MLflow\n- Access features from Feature Store\n- Deploy models with one-click Hermes CLI commands\n\n### ML Engineers\n**Use Darwin for**: Production model deployment and monitoring\n- Configure multi-environment serves (dev/staging/prod)\n- Build and deploy artifacts from GitHub\n- Manage auto-scaling policies\n- Monitor model performance and resource usage\n\n### Data Engineers\n**Use Darwin for**: Feature pipelines and data infrastructure\n- Create and manage feature groups in Feature Store\n- Build Spark-based feature engineering pipelines\n- Track data lineage in Catalog\n- Publish features to Kafka for real-time materialization\n\n### Platform Engineers\n**Use Darwin for**: Infrastructure management and operations\n- Deploy and configure Darwin platform via Helm\n- Manage Kubernetes resources and policies\n- Monitor costs and resource utilization\n- Configure multi-tenancy and RBAC\n\n---\n\n## 🚀 Getting Started\n\n### Prerequisites\n- Kubernetes cluster (Kind for local, EKS for production)\n- Helm 3.8+\n- kubectl\n- Docker\n- Python 3.9.7+\n\n### Quick Start: Local Deployment\n\n```bash\n# 1. Initialize configuration (select components to enable)\n./init.sh\n\n# 2. Build platform images and setup Kind cluster\n./setup.sh\n\n# 3. Deploy Darwin platform to Kubernetes\n./start.sh\n```\n\n#### Choosing Your Components\n\nDuring `init.sh`, you'll select which Darwin components to enable. Here's how to decide based on your workflow:\n\n| If you want to... | Enable |\n|-------------------|--------|\n| Run distributed data processing jobs or spin up short-lived compute clusters | **Compute** |\n| Work interactively with persistent code and notebooks attached to scalable clusters | **Workspace** (includes Compute) |\n| Store, version, and serve features for ML training and inference | **Feature Store** |\n| Track experiments, log metrics, and manage model versions | **MLflow** |\n| Deploy trained models as real-time inference endpoints | **Serve** (includes Artifact Builder) |\n| Discover and track lineage across datasets, models, and pipelines | **Catalog** |\n| Capture platform events and build metadata graphs | **Chronos** |\n\n\u003e **Tip**: Dependencies are resolved automatically. For example, enabling **Workspace** will also enable **Compute**, and enabling **Serve** will include **Artifact Builder** and **MLflow**.\n\n**Access Services**:\n- Compute: `http://localhost/compute/*`\n- Feature Store: `http://localhost/feature-store/*`\n- MLflow UI: `http://localhost/mlflow/*`\n- Chronos API: `http://localhost/chronos/*`\n- Catalog API: `http://localhost/darwin-catalog/*`\n- Workspace: `http://localhost/workspace/*`\n\n### Quick Start: Create and Use a Ray Cluster\n\n```bash\n# Create a cluster via REST API\ncurl --location 'http://localhost/compute/cluster' \\\n  --header 'Content-Type: application/json' \\\n  --data-raw '{\n    \"cluster_name\": \"my-first-cluster\",\n    \"tags\": [\"demo\"],\n    \"runtime\": \"0.0\",\n    \"inactive_time\": 30,\n    \"start_cluster\": true,\n    \"head_node_config\": {\n        \"cores\": 4,\n        \"memory\": 8,\n        \"node_capacity_type\": \"ondemand\"\n    },\n    \"worker_node_configs\": [\n        {\n            \"cores_per_pods\": 2,\n            \"memory_per_pods\": 4,\n            \"min_pods\": 1,\n            \"max_pods\": 2,\n            \"disk_setting\": null,\n            \"node_capacity_type\": \"ondemand\"\n        }\n    ],\n    \"user\": \"user@example.com\"\n}'\n\n# Wait for Cluster to become Active\ncurl http://localhost/compute/cluster/{cluster_id}/metadata\n# Wait until the status shows active.\n\n# Response will include cluster_id\n# Get Cluster Dashboards link via below API using cluster_id\ncurl --location 'http://localhost/compute/cluster/{cluster_id}/dashboards'\n# Access Jupyter notebook at the returned jupyter_lab_url\n# Monitor Ray cluster at the ray_dashboard_url\n\n# Stop the cluster when done\ncurl --location --request POST 'http://localhost/compute/cluster/stop-cluster/{cluster_id}' \\\n  --header 'msd-user: {\"email\": \"user@example.com\"}'\n```\n\n**Understanding Runtime Parameter:**\n\nThe `runtime` field specifies which pre-built Docker image to use for your Ray cluster. Darwin supports multiple runtimes with different Python versions and pre-installed libraries:\n\n- `\"0.0\"`: Default runtime with Ray 2.37.0, Python 3.10, Spark 3.5.1, and darwin-sdk\n- Custom runtimes can be registered with specific library combinations\n\nTo check available runtimes:\n```bash\ncurl http://localhost/compute/get-runtimes | python3 -m json.tool\n```\n\n**Or use the Python SDK:**\n\n```python\n# Install SDK\npip install -e darwin-compute/sdk\n\n# Create a cluster\nfrom darwin_compute import ComputeCluster\n\ncluster = ComputeCluster(env=\"darwin-local\")\nresponse = cluster.create_with_yaml(\"examples/cluster-config.yaml\")\ncluster_id = response['cluster_id']\n\n# Check and wait until cluster status becomes active\ncluster.get_info(cluster_id)\n\n# Stop when done\ncluster.stop(cluster_id)\n```\n\n### Quick Start: Deploy a Model\n\n```bash\n# Activate Hermes CLI\nsource hermes-cli/.venv/bin/activate\n\n# 1. Configure Hermes CLI with authentication token\nexport HERMES_USER_TOKEN=admin-token-default-change-in-production\nhermes configure\n\n# 2. Create environment\nhermes create-environment --name local --domain-suffix .local --cluster-name kind\n\n# 3. Create serve\nhermes create-serve \\\n  --name iris-classifier \\\n  --type api \\\n  --space serve \\\n  --description \"Iris classification model\"\n\n# 4. Deploy model (one-click)\nhermes deploy-model \\\n  --serve-name iris-classifier \\\n  --artifact-version v1 \\\n  --model-uri mlflow-artifacts:/1/2b2b1b5727a14c5ca81b44e899979745/artifacts/model \\\n  --cores 2 \\\n  --memory 4 \\\n  --node-capacity spot \\\n  --min-replicas 1 \\\n  --max-replicas 2\n\n# 5. Make predictions\ncurl -X POST http://localhost/iris-classifier/predict \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"features\": [[5.1, 3.5, 1.4, 0.2]]}'\n```\n\n\u003e **📖 For more deployment options, see [hermes-cli/CLI.md](hermes-cli/CLI.md)**\n\n### Quick Start: Use Feature Store\n\n```python\n# Install SDK\npip install -e feature-store/python/darwin_fs\n\n# Fetch features\nfrom darwin_fs import FeatureStoreClient\n\nfs = FeatureStoreClient(env=\"local\")\nfeatures = fs.fetch_features(\n    feature_group_name=\"user_features\",\n    feature_columns=[\"age\", \"tenure\", \"activity_score\"],\n    primary_key_names=[\"user_id\"],\n    primary_key_values=[[123], [456], [789]]\n)\n```\n\n---\n\n## 🎯 Quick Start: Submit a Spark Job Using Darwin SDK\n\nDarwin SDK provides seamless integration with Apache Spark on Ray clusters. Here's how to run distributed Spark workloads using Darwin as your Spark session provider:\n\n### Step 1: Create a Ray Cluster\n\n```bash\ncurl --location 'http://localhost/compute/cluster' \\\n  --header 'Content-Type: application/json' \\\n  --data-raw '{\n    \"cluster_name\": \"spark-demo-cluster\",\n    \"tags\": [\"spark\", \"demo\"],\n    \"runtime\": \"0.0\",\n    \"inactive_time\": 60,\n    \"start_cluster\": true,\n    \"head_node_config\": {\n        \"cores\": 4,\n        \"memory\": 8,\n        \"node_capacity_type\": \"ondemand\"\n    },\n    \"worker_node_configs\": [{\n        \"cores_per_pods\": 2,\n        \"memory_per_pods\": 4,\n        \"min_pods\": 1,\n        \"max_pods\": 2,\n        \"disk_setting\": null,\n        \"node_capacity_type\": \"ondemand\"\n    }],\n    \"user\": \"user@example.com\"\n}'\n```\n\nSave the `cluster_id` from the response.\n\n### Step 2: Wait for Cluster to be Ready\n\n```bash\n# Check cluster status\ncurl http://localhost/compute/cluster/{cluster_id}/metadata\n\n# Wait until status shows \"active\"\n# Then verify pods are running\nkubectl get pods -n ray -l ray.io/cluster={cluster_id}-kuberay\n```\n\n### Step 3: Create Your Spark Job\n\nCreate a file `my_spark_job.py`:\n\n```python\n#!/usr/bin/env python3\n\"\"\"\nDarwin SDK Spark Job Example\n\"\"\"\nimport os\nimport ray\n\n# Initialize Ray (connects to running Ray cluster)\nray.init()\n\n# Set environment variables\nos.environ[\"ENV\"] = \"LOCAL\"\nos.environ[\"CLUSTER_ID\"] = os.getenv(\"CLUSTER_ID\", \"your-cluster-id\")\nos.environ[\"DARWIN_COMPUTE_URL\"] = \"http://darwin-compute.darwin.svc.cluster.local:8000\"\n\nprint(\"=\" * 60)\nprint(\"Darwin SDK Spark Job\")\nprint(f\"Cluster ID: {os.environ['CLUSTER_ID']}\")\nprint(\"=\" * 60)\n\n# Initialize Spark using darwin-sdk\nfrom darwin import init_spark_with_configs\n\nspark_configs = {\n    \"spark.sql.execution.arrow.pyspark.enabled\": \"true\",\n    \"spark.sql.session.timeZone\": \"UTC\",\n    \"spark.sql.shuffle.partitions\": \"10\",\n    \"spark.default.parallelism\": \"10\",\n    \"spark.driver.memory\": \"1g\",\n    \"spark.executor.memory\": \"1g\",\n}\n\nspark = init_spark_with_configs(spark_configs=spark_configs)\nprint(f\"✓ Spark initialized (version: {spark.version})\")\n\n# Create and process DataFrame\ndf = spark.createDataFrame([\n    (1, \"Alice\", 100),\n    (2, \"Bob\", 200),\n    (3, \"Charlie\", 300),\n], [\"id\", \"name\", \"score\"])\n\nprint(\"\\nDataFrame Contents:\")\ndf.show()\n\nprint(f\"\\nTotal records: {df.count()}\")\nprint(f\"Average score: {df.agg({'score': 'avg'}).collect()[0][0]}\")\n\n# Stop Spark cleanly\nfrom darwin import stop_spark\nstop_spark()\n\nprint(\"\\n✓ Job completed successfully!\")\n```\n\n### Step 4: Submit the Job\n\n**Option A: Using submit_spark_job.sh Script**\n\n```bash\ncd darwin-sdk/darwin\n./submit_spark_job.sh \\\n  --cluster-name {cluster_id} \\\n  --namespace ray \\\n  --job-file /path/to/my_spark_job.py \\\n  --wait\n```\n\n**Option B: Using Ray Jobs API**\n\n```bash\n# Port-forward to Ray dashboard\nkubectl port-forward -n ray svc/{cluster_id}-kuberay-head-svc 8265:8265 \u0026\n\n# Submit job\ncurl -X POST \"http://localhost:8265/api/jobs/\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"entrypoint\": \"python my_spark_job.py\",\n    \"runtime_env\": {\n      \"working_dir\": \"./\",\n      \"env_vars\": {\n        \"CLUSTER_ID\": \"'{cluster_id}'\",\n        \"ENV\": \"LOCAL\"\n      }\n    },\n    \"metadata\": {\n      \"name\": \"darwin-spark-demo\"\n    }\n  }'\n```\n\n**Option C: Using Ray Python Client**\n\n```python\nfrom ray.job_submission import JobSubmissionClient\n\nclient = JobSubmissionClient(\"http://localhost:8265\")\n\njob_id = client.submit_job(\n    entrypoint=\"python my_spark_job.py\",\n    runtime_env={\n        \"working_dir\": \"./\",\n        \"env_vars\": {\n            \"CLUSTER_ID\": \"{cluster_id}\",\n            \"ENV\": \"LOCAL\"\n        }\n    }\n)\n\nprint(f\"Submitted job: {job_id}\")\n\n# Wait for completion\nclient.wait_until_status(job_id, \"SUCCEEDED\")\nprint(client.get_job_logs(job_id))\n```\n\n### Step 5: Monitor Job Execution\n\n```bash\n# Check job status\nSUBMISSION_ID=\"raysubmit_xxxxxxxx\"\ncurl \"http://localhost:8265/api/jobs/${SUBMISSION_ID}\"\n\n# View job logs\ncurl \"http://localhost:8265/api/jobs/${SUBMISSION_ID}/logs\"\n\n# Or use Ray Dashboard\nopen http://localhost:8265\n```\n\n### Darwin SDK Spark Functions\n\n| Function | Description |\n|----------|-------------|\n| `init_spark_with_configs(spark_configs)` | Initialize Spark with custom configurations |\n| `start_spark(spark_conf)` | Start Spark with default Glue catalog configs |\n| `get_raydp_spark_session()` | Get existing Spark session |\n| `stop_spark()` | Stop Spark session cleanly |\n\n### Troubleshooting\n\n**Issue: \"Runtime given is incorrect\"**\n```bash\n# Check available runtimes\ncurl http://localhost/compute/get-runtimes\n```\n\n**Issue: Ray job stuck in PENDING**\n```bash\n# Check Ray head pod\nkubectl describe pod {cluster_id}-kuberay-head-xxx -n ray\n```\n\n**Issue: Connection refused when submitting job**\n```bash\n# Restart port-forward\npkill -f \"port-forward.*8265\"\nkubectl port-forward -n ray svc/{cluster_id}-kuberay-head-svc 8265:8265 \u0026\n```\n\n**Issue: Cluster not starting due to long init script**\n\nIf your `init_script` in the cluster configuration is too long, the cluster may fail to start. This happens because init scripts are executed during pod startup and have timeout limitations.\n\n**Solutions:**\n- Use the **Library Installation API** to install packages instead of init scripts\n- Create a **custom runtime** with your dependencies pre-installed\n- Split long scripts into smaller, essential commands\n\n---\n\n## 🧪 Creating Your First Project\n\nThis guide walks you through your first end-to-end experience on Darwin — from compute creation to deployment.\n\n### 🔧 1) Create Compute\n\nCreate a Ray cluster for your ML workload:\n\n```bash\ncurl --location 'http://localhost/compute/cluster' \\\n  --header 'Content-Type: application/json' \\\n  --data-raw '{\n    \"cluster_name\": \"housing-project\",\n    \"tags\": [\"tutorial\", \"housing-prices\"],\n    \"runtime\": \"0.0\",\n    \"inactive_time\": 60,\n    \"head_node_config\": {\n        \"cores\": 4,\n        \"memory\": 8\n    },\n    \"worker_node_configs\": [\n        {\n            \"cores\": 2,\n            \"memory\": 4,\n            \"min_pods\": 1,\n            \"max_pods\": 2\n        }\n    ],\n    \"user\": \"user@example.com\"\n}'\n```\n\nSave the `cluster_id` from the response - you'll need it for the next steps.\n\n### 📊 2) Check Status\n\nCheck your cluster status:\n\n```bash\ncurl http://localhost/compute/cluster/{cluster_id}/metadata\n```\n\n**Wait until the status shows active.**\n\n### 📓 3) Open Jupyter Notebook\n\nOnce the cluster is ready, access the Jupyter notebook at:\n\n```\nhttp://localhost/kind-0/{cluster_id}-jupyter\n```\n\nOpen this URL in your browser to start working in the workspace.\n\n### 🏡 4) Copy and Run Example: housing-prices\n\nIn the Jupyter notebook, copy the example project: /examples/housing-prices/ in Jupyter notebook. The model will be logged automatically to MLflow.\n\n### 🏷️ 5) Check Your Model in the Registry\n\nVerify your trained model in the Darwin MLflow UI:\n\n```\nhttp://localhost/mlflow-app/experiments\n```\n\nNavigate to your experiment to see the registered model with metrics and parameters.\n\n### 🚀 6) Deploy with Hermes CLI\n\nDeploy your trained model (replace `\u003cexperiment_id\u003e` and `\u003crun_id\u003e` with values from MLflow UI):\n\n\u003e **📖 Sample training script for house price prediction: [examples/house-price-prediction/train_house_pricing_model.ipynb](examples/house-price-prediction/train_house_pricing_model.ipynb)**\n\n```bash\n# Activate Hermes CLI\nsource hermes-cli/.venv/bin/activate\n\n# 1. Configure Hermes CLI with authentication token (one-time)\nexport HERMES_USER_TOKEN=admin-token-default-change-in-production\nhermes configure\n\n# 2. Create environment\nhermes create-environment --name local --domain-suffix .local --cluster-name kind\n\n# 3. Create serve\nhermes create-serve \\\n  --name housing-model \\\n  --type api \\\n  --space serve \\\n  --description \"House Price Prediction model\"\n\n# 4. Deploy model (one-click)\nhermes deploy-model \\\n  --serve-name housing-model \\\n  --artifact-version v1 \\\n  --model-uri mlflow-artifacts:/1/\u003cexperiment_id\u003e/\u003crun_id\u003e/artifacts/model \\\n  --cores 2 \\\n  --memory 4 \\\n  --node-capacity spot \\\n  --min-replicas 1 \\\n  --max-replicas 2\n```\n\n### 🌐 7) Test Your Endpoint\n\nTest your deployed model:\n\n```bash\ncurl -X POST http://localhost/housing-model/predict \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"features\": {\n      \"MedInc\": 3.5214,\n      \"HouseAge\": 15.0,\n      \"AveRooms\": 6.575757575757576,\n      \"AveBedrms\": 1.0196969696969697,\n      \"Population\": 1447.0,\n      \"AveOccup\": 3.0144927536231883,\n      \"Latitude\": 37.63,\n      \"Longitude\": -122.43\n    }\n  }'\n```\n\nOnce deployed, your model is accessible as a real-time inference API.\n\n---\n\n### 🌸 Alternative Example: Iris Classification\n\nYou can also try the Iris classification model as an alternative example:\n\n\u003e **📖 Sample training script for iris classification: [examples/iris-classification/train_iris_model.ipynb](examples/iris-classification/train_iris_model.ipynb)**\n\n**Deploy the Iris model:**\n\n```bash\n# Activate Hermes CLI\nsource hermes-cli/.venv/bin/activate\n\n# 1. Configure Hermes CLI with authentication token (one-time)\nexport HERMES_USER_TOKEN=admin-token-default-change-in-production\nhermes configure\n\n# 2. Create environment\nhermes create-environment --name local --domain-suffix .local --cluster-name kind\n\n# 3. Create serve\nhermes create-serve \\\n  --name iris-classifier \\\n  --type api \\\n  --space serve \\\n  --description \"Iris Species Classification model\"\n\n# 4. Deploy model (one-click)\nhermes deploy-model \\\n  --serve-name iris-classifier \\\n  --model-uri mlflow-artifacts:/\u003cexperiment_id\u003e/\u003crun_id\u003e/artifacts/model \\\n  --cores 2 \\\n  --memory 4 \\\n  --node-capacity spot \\\n  --min-replicas 1 \\\n  --max-replicas 2\n```\n\n**Test the Iris model:**\n\n```bash\ncurl -X POST http://localhost/iris-classifier/predict \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"features\": {\n      \"sepal_length\": 5.1,\n      \"sepal_width\": 3.5,\n      \"petal_length\": 1.4,\n      \"petal_width\": 0.2\n    }\n  }'\n```\n\n\u003e **📖 For detailed deployment commands, see [hermes-cli/CLI.md](hermes-cli/CLI.md)**\n\n---\n\n## 🧪 Example Workflows\n\n### End-to-End ML Workflow\n\n```mermaid\nsequenceDiagram\n    participant DS as Data Scientist\n    participant Compute as Darwin Compute\n    participant MLflow as MLflow\n    participant FS as Feature Store\n    participant Serve as ML Serve\n\n    DS-\u003e\u003eCompute: Create Ray cluster\n    Compute--\u003e\u003eDS: Cluster ID + Jupyter link\n    DS-\u003e\u003eFS: Fetch training features\n    FS--\u003e\u003eDS: Feature dataset\n    DS-\u003e\u003eCompute: Train model (Ray/Spark)\n    DS-\u003e\u003eMLflow: Log experiment + model\n    DS-\u003e\u003eServe: Deploy model (Hermes CLI)\n    Serve-\u003e\u003eMLflow: Fetch model artifact\n    Serve-\u003e\u003eFS: Configure feature retrieval\n    Serve--\u003e\u003eDS: Inference endpoint\n    DS-\u003e\u003eCompute: Stop cluster\n```\n\n### Feature Engineering Pipeline\n\n```mermaid\ngraph LR\n    A[Raw Data\u003cbr/\u003eS3/Delta Lake] --\u003e B[Spark Pipeline\u003cbr/\u003eFeature Transform]\n    B --\u003e C[Kafka Topic\u003cbr/\u003efeature-updates]\n    C --\u003e D[Feature Store\u003cbr/\u003eConsumer]\n    D --\u003e E[Cassandra\u003cbr/\u003eMaterialized Features]\n    E --\u003e F[Online Serving\u003cbr/\u003e\u003c 10ms latency]\n    B --\u003e G[Catalog\u003cbr/\u003eLineage Tracking]\n```\n\n---\n\n## 📊 Technology Stack\n\n| Layer | Technologies |\n|-------|-------------|\n| **Languages** | Python 3.9.7, Java 11, Go 1.18 |\n| **Compute** | Ray 2.37.0, Apache Spark 3.5.1 |\n| **Web Frameworks** | FastAPI, Spring Boot, Vert.x |\n| **Orchestration** | Kubernetes (EKS/Kind), Helm 3, KubeRay Operator v1.1.0 |\n| **Databases** | MySQL 8.0, Cassandra 5.0, OpenSearch 2.11 |\n| **Streaming** | Apache Kafka 7.4.0 |\n| **Storage** | S3 (AWS/LocalStack), FSx, EFS |\n| **Experiment Tracking** | MLflow 2.12.2 |\n| **Monitoring** | Prometheus, Grafana, Ray Dashboard |\n| **Container Registry** | ECR, GCR, Local Docker Registry |\n\n---\n\n## 🔧 Configuration\n\nDarwin uses a declarative configuration approach:\n\n### Service Selection (`init.sh`)\nInteractive wizard to select platform components:\n```bash\n./init.sh\n# Prompts for enabling:\n# - Applications (Compute, Feature Store, MLflow, etc.)\n# - Datastores (MySQL, Cassandra, Kafka, etc.)\n# - Ray images and Serve runtimes\n```\n\nGenerates `.setup/enabled-services.yaml` with user selections.\n\n### Environment Variables\nKey configuration via `config.env` (auto-generated):\n```bash\nKUBECONFIG=./kind/config/kindkubeconfig.yaml\nDOCKER_REGISTRY=127.0.0.1:32768\n```\n\n### Helm Values\nCustomize deployments via `helm/darwin/values.yaml`:\n```yaml\nglobal:\n  namespace: darwin\n  \nservices:\n  compute:\n    enabled: true\n    replicas: 2\n  \ndatastores:\n  mysql:\n    enabled: true\n  cassandra:\n    enabled: true\n```\n\n---\n\n## 🏢 Deployment Patterns\n\n### Local Development (Kind)\n- Single-node Kubernetes cluster\n- Local Docker registry\n- HostPath-based persistent storage\n- Nginx Ingress at `localhost/*`\n\n### Production (EKS)\n- Multi-AZ high availability\n- Mixed spot/on-demand node groups\n- Auto-scaling with Karpenter\n- Network policies and security groups\n- S3-backed artifact storage\n- RDS for MySQL (optional)\n- Multi-tenant namespace isolation\n\n---\n\n## 📈 Observability\n\n### Metrics\n- **Prometheus**: Cluster resource utilization, service metrics\n- **Grafana**: Pre-configured dashboards for compute, serving, features\n- **Ray Dashboard**: Job execution, task profiling, resource usage\n\n### Logging\n- Centralized logging via stdout/stderr\n- Application logs in `/app/logs`\n- Structured logging with context\n\n### Events \u0026 Lineage\n- **Chronos**: Event-driven tracking of all platform operations\n- **Catalog**: Data lineage via OpenLineage\n- Elasticsearch-based search and analytics\n\n### Alerts\n- Slack integration for cost alerts\n- Long-running cluster notifications\n- Failed deployment alerts\n\n---\n\n## 📚 SDKs \u0026 APIs\n\n### Available SDKs\n- **darwin-compute**: Ray cluster management\n- **darwin_fs**: Feature Store client\n- **darwin_mlflow**: MLflow wrapper with auth\n- **darwin-workspace** (internal): Workspace orchestration\n\n### REST APIs\nAll services expose FastAPI/Spring Boot REST APIs:\n- Feature Store: `/feature-store/*`, `/feature-store-admin/*`\n- Darwin Compute: `/cluster/*`, `/jupyter/*`\n- ML Serve: `/api/v1/serve/*`, `/api/v1/artifact/*`\n- Chronos: `/api/v1/event/*`, `/api/v1/sources/*`\n- Catalog: `/v1/assets/*`, `/v1/lineage/*`\n\nAPI documentation available at `\u003cservice-url\u003e/docs` (Swagger UI).\n\n---\n\n## 🧩 Extensibility\n\n### Available Runtimes\n\nDarwin provides pre-built Ray runtimes for cluster creation. The **Runtime Name** is what you pass to the API when creating clusters (e.g., `\"runtime\": \"0.1\"`).\n\n| Runtime Name | Image | Ray Version | Python | Class | Type |\n|--------------|-------|-------------|--------|-------|------|\n| 0.0 | ray:2.37.0 | 2.37.0 | 3.10 | CPU | Ray Only |\n| 0.1 | ray:2.53.0 | 2.53.0 | 3.10 | CPU | Ray Only |\n\n### Custom Runtimes\nAdd new Ray runtimes by creating Dockerfiles in `darwin-compute/runtimes/`:\n```dockerfile\n# darwin-compute/runtimes/cpu/Ray2.37_Py3.11_CustomLibs/Dockerfile\nFROM rayproject/ray:2.37.0-py311\nRUN pip install jupyterlab==4.3.0\nRUN pip install custom-library\n```\n\nRegister in `services.yaml`:\n```yaml\nray-images:\n  - image-name: ray:2.37.0-py311-custom\n    dockerfile-path: darwin-compute/runtimes/cpu/Ray2.37_Py3.11_CustomLibs\n```\n\n### Custom Transformers (Chronos)\nCreate Python transformers for event processing:\n```python\n# Chronos transformer\ndef transform(event):\n    return {\n        \"event_type\": \"cluster_created\",\n        \"entities\": [{\"type\": \"cluster\", \"id\": event[\"cluster_id\"]}],\n        \"relationships\": [{\"from\": user, \"to\": cluster, \"type\": \"owns\"}]\n    }\n```\n\n---\n\n## 🤝 Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, coding standards, and contribution guidelines.\n\n---\n\n## 📄 License\n\n[License information to be added]\n\n---\n\n## 📞 Support\n\nFor issues, questions, or feature requests, please open an issue in the repository or contact the platform team.\n\n---\n\n## 🗺️ Project Structure\n\n```\ndarwin-distro/\n├── darwin-compute/          # Ray cluster management service\n├── darwin-cluster-manager/  # Kubernetes orchestration (Go)\n├── feature-store/           # Feature Store (Java)\n├── mlflow/                  # MLflow experiment tracking\n├── ml-serve-app/            # Model serving platform\n├── artifact-builder/        # Docker image builder\n├── chronos/                 # Event processing \u0026 metadata\n├── workspace/               # Project \u0026 codespace management\n├── darwin-catalog/          # Data catalog \u0026 lineage\n├── hermes-cli/              # CLI tool for deployments\n├── helm/                    # Helm charts for deployment\n│   └── darwin/              # Umbrella chart\n│       ├── charts/\n│       │   ├── datastores/  # MySQL, Cassandra, Kafka, etc.\n│       │   └── services/    # Application services\n├── deployer/                # Build scripts and base images\n├── kind/                    # Local Kubernetes setup\n├── examples/                # Example notebooks and configs\n├── init.sh                  # Interactive configuration wizard\n├── setup.sh                 # Build and cluster setup\n├── start.sh                 # Deploy platform\n└── services.yaml            # Service registry\n```\n\n---\n\n**Darwin ML Platform** — Unified, scalable, production-ready machine learning infrastructure.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdream-horizon-org%2Fdarwin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdream-horizon-org%2Fdarwin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdream-horizon-org%2Fdarwin/lists"}