{"id":47628722,"url":"https://github.com/jaehyeon-kim/dynamic-des","last_synced_at":"2026-04-21T02:15:47.943Z","repository":{"id":342896891,"uuid":"1174883175","full_name":"jaehyeon-kim/dynamic-des","owner":"jaehyeon-kim","description":"Real-time SimPy control plane to dynamically update parameters and stream outputs via external systems like Kafka, Redis, or Postgres. Built for event-driven digital twins.","archived":false,"fork":false,"pushed_at":"2026-04-07T07:24:45.000Z","size":3010,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-07T09:22:21.234Z","etag":null,"topics":["data-generation","descrete-event-simulation","digital-twin","industry-4","kafka","postgres","python","redis","simpy","simulation"],"latest_commit_sha":null,"homepage":"https://jaehyeon.me/dynamic-des/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jaehyeon-kim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-07T00:19:19.000Z","updated_at":"2026-04-07T07:23:18.000Z","dependencies_parsed_at":"2026-03-08T03:02:52.253Z","dependency_job_id":null,"html_url":"https://github.com/jaehyeon-kim/dynamic-des","commit_stats":null,"previous_names":["jaehyeon-kim/dynamic-des"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/jaehyeon-kim/dynamic-des","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaehyeon-kim%2Fdynamic-des","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaehyeon-kim%2Fdynamic-des/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaehyeon-kim%2Fdynamic-des/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaehyeon-kim%2Fdynamic-des/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jaehyeon-kim","download_url":"https://codeload.github.com/jaehyeon-kim/dynamic-des/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaehyeon-kim%2Fdynamic-des/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32073509,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-21T01:35:38.224Z","status":"online","status_checked_at":"2026-04-21T02:00:06.111Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-generation","descrete-event-simulation","digital-twin","industry-4","kafka","postgres","python","redis","simpy","simulation"],"created_at":"2026-04-01T23:03:31.612Z","updated_at":"2026-04-21T02:15:47.938Z","avatar_url":"https://github.com/jaehyeon-kim.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dynamic DES\n\n[![CI Pipeline](https://github.com/jaehyeon-kim/dynamic-des/actions/workflows/pipeline.yml/badge.svg)](https://github.com/jaehyeon-kim/dynamic-des/actions/workflows/pipeline.yml)\n[![Documentation](https://img.shields.io/badge/docs-latest-blue.svg)](https://jaehyeon-kim.github.io/dynamic-des/)\n[![PyPI version](https://badge.fury.io/py/dynamic-des.svg)](https://badge.fury.io/py/dynamic-des)\n[![Python Versions](https://img.shields.io/pypi/pyversions/dynamic-des.svg)](https://pypi.org/project/dynamic-des/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n**Real-time SimPy control plane for event-driven digital twins.**\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/jaehyeon-kim/dynamic-des/main/docs/assets/dashboard-preview.gif\" alt=\"Dashboard Screenshot\" width=\"800\" /\u003e\n\u003c/div\u003e\n\nDynamic DES bridges the gap between static discrete-event simulations and the live world. It allows you to update simulation parameters (arrivals, service times, capacities) and stream telemetry via **Kafka**, **Redis**, or **PostgreSQL** without stopping the simulation.\n\n## Key Features\n\n- **⚡ Real-Time Control**: Synchronize SimPy with the system clock using `DynamicRealtimeEnvironment`.\n- **🔗 Dynamic Registry**: Dynamic, path-based updates (e.g., `Line_A.arrival.rate`) that trigger instant logic changes.\n- **🚀 High Throughput**: Optimized to handle high throughput using `orjson` and local batching.\n- **🛡️ Enterprise Ready**: Native `**kwargs` passthrough for SASL, mTLS, OAuth, and AWS IAM Kafka clusters.\n- **📦 Pluggable Serialization**: Stream lightweight JSON by default, or map specific ML topics to lazy-loaded **Avro/Schema Registry** serializers (Confluent \u0026 AWS Glue).\n- **🦆 Pydantic Duck-Typing**: Seamlessly publish strictly-typed Pydantic V2 models straight from your simulation logic.\n- **🌍 Domain Agnostic**: Perfect for factory floors, crypto trading bots, or RPG game state management.\n\n## Installation\n\nInstall the core library:\n\n```bash\npip install dynamic-des\n```\n\nTo include specific backends and enterprise features:\n\n```bash\n# For Kafka support\npip install \"dynamic-des[kafka]\"\n\n# For Confluent Schema Registry (Avro)\npip install \"dynamic-des[kafka,confluent]\"\n\n# For AWS Glue Schema Registry (Avro)\npip install \"dynamic-des[kafka,glue]\"\n\n# For all backends (Kafka, Redis, Postgres, Dashboard, Avro)\npip install \"dynamic-des[all]\"\n```\n\n## Quick Start: Zero-Setup Demos\n\nDynamic DES comes with built-in examples and infrastructure orchestration so you can see it in action immediately.\n\n**Run the local, dependency-free simulation:**\n\n```bash\nddes-local-example\n```\n\n**Run the full Real-Time Digital Twin stack with Kafka and a live UI:**\n\n```bash\n# Start the background Kafka cluster (requires Docker)\nddes-kafka-infra-up\n\n# Open a new terminal and run the simulation\n# Ctrl + C to stop\nddes-kafka-example\n\n# Open a new terminal and start the control dashboard (opens in browser)\n# Visit http://localhost:8080\n# Ctrl + C to stop\nddes-kafka-dashboard\n\n# Clean up the infrastructure when finished\nddes-kafka-infra-down\n```\n\n## Building Your Own Simulation (Local Example)\n\nThe following snippet demonstrates a simple example. It initializes a production line, schedules an external capacity update, and streams telemetry to the console.\n\n```python\nimport logging\nfrom dynamic_des import (\n    CapacityConfig, ConsoleEgress, DistributionConfig,\n    DynamicRealtimeEnvironment, DynamicResource, LocalIngress, SimParameter\n)\n\nlogging.basicConfig(\n    level=logging.INFO, format=\"%(levelname)s [%(asctime)s] %(message)s\"\n)\nlogger = logging.getLogger(\"local_example\")\n\n# 1. Define initial system state\nparams = SimParameter(\n    sim_id=\"Line_A\",\n    arrival={\"standard\": DistributionConfig(dist=\"exponential\", rate=1)},\n    resources={\"lathe\": CapacityConfig(current_cap=1, max_cap=5)},\n)\n\n# 2. Setup Environment with Local Connectors\n# Schedule capacity to jump from 1 to 3 at t=5s\ningress = LocalIngress([(5.0, \"Line_A.resources.lathe.current_cap\", 3)])\negress = ConsoleEgress()\n\nenv = DynamicRealtimeEnvironment(factor=1.0)\nenv.registry.register_sim_parameter(params)\nenv.setup_ingress([ingress])\nenv.setup_egress([egress])\n\n# 3. Create Resource\nres = DynamicResource(env, \"Line_A\", \"lathe\")\n\ndef telemetry_monitor(env: DynamicRealtimeEnvironment, res: DynamicResource):\n    \"\"\"Streams system health metrics every 2 seconds.\"\"\"\n    while True:\n        env.publish_telemetry(\"Line_A.resources.lathe.capacity\", res.capacity)\n        yield env.timeout(2.0)\n\n\nenv.process(telemetry_monitor(env, res))\n\n# 4. Run\nprint(\"Simulation started. Watch capacity change at t=5s...\")\ntry:\n    env.run(until=10.1)\nfinally:\n    env.teardown()\n```\n\n### What this does\n\n1.  **Registry Initialization**: The `SimParameter` defines the initial state. The Registry flattens this into addressable paths (e.g., `Line_A.resources.lathe.current_cap`).\n2.  **Live Ingress**: The `LocalIngress` simulates an external event (like a Kafka message) arriving 5 seconds into the run.\n3.  **Zero-Polling Update**: The `DynamicResource` listens to the Registry. The moment the ingress updates the value, the resource automatically expands its internal token pool without any manual checking.\n4.  **Telemetry Egress**: The `ConsoleEgress` prints system vitals to your terminal, mimicking a live dashboard feed.\n\n### Data Egress JSON Schemas\n\nTo ensure strict data contracts with external consumers (like Kafka, Redis, or PostgreSQL), `dynamic-des` uses Pydantic to validate all outbound payloads. Users can expect two distinct JSON structures depending on the stream type:\n\n#### Telemetry Stream\n\nUsed for scalar metrics like resource utilization, queue lengths, or simulation lag.\n\n```json\n{\n  \"stream_type\": \"telemetry\",\n  \"path_id\": \"Line_A.resources.lathe.utilization\",\n  \"value\": 85.5,\n  \"sim_ts\": 120.5,\n  \"timestamp\": \"2023-10-25T14:30:00.000Z\"\n}\n```\n\n#### Event Stream\n\nUsed for discrete task lifecycle events (e.g., a part arriving, entering a queue, or finishing processing).\n\n```json\n{\n  \"stream_type\": \"event\",\n  \"key\": \"task-001\",\n  \"value\": {\n    \"status\": \"finished\",\n    \"duration\": 45.2,\n    \"path_id\": \"Line_A.service.lathe\"\n  },\n  \"sim_ts\": 125.0,\n  \"timestamp\": \"2023-10-25T14:30:04.500Z\"\n}\n```\n\n### More Examples\n\nFor more examples, including implementations using **Kafka** providers, please explore the [examples](./src/dynamic_des/examples/) folder.\n\n## Core Concepts\n\n**Dynamic DES** is built on the **Switchboard Pattern**, decoupling data sourcing from simulation logic.\n\n### Switchboard Pattern\n\nInstead of resources polling Kafka directly, the architecture is split into three layers:\n\n1.  **Connectors (Ingress/Egress)**: Background threads handle heavy I/O (Kafka, Redis).\n2.  **Registry (Switchboard)**: A centralized state manager that flattens data into dot-notation paths.\n3.  **Resources (SimPy Objects)**: Passive observers that \"wake up\" only when the Registry signals a change.\n\n### Event-Driven Capacity\n\nStandard SimPy resources have static capacities. `DynamicResource` wraps a `Container` and a `PriorityStore`. When the Registry updates:\n\n- **Growing**: Extra tokens are added to the pool immediately.\n- **Shrinking**: The resource requests tokens back. If they are busy, it waits until they are released, ensuring no work-in-progress is lost.\n\n### High-Throughput Events\n\nTo handle high throughput, the `EgressMixIn` uses:\n\n- **Batching**: Pushing lists of events to the I/O thread to reduce lock contention.\n- **orjson**: Rust-powered serialization for maximum speed.\n\n## Documentation\n\nFor full documentation, architecture details, and API reference, visit:\n[https://jaehyeon.me/dynamic-des/](https://jaehyeon.me/dynamic-des/).\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaehyeon-kim%2Fdynamic-des","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjaehyeon-kim%2Fdynamic-des","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaehyeon-kim%2Fdynamic-des/lists"}