{"id":35127134,"url":"https://github.com/qualityunit/pyworkflow","last_synced_at":"2026-04-08T16:02:44.833Z","repository":{"id":326547289,"uuid":"1106052320","full_name":"QualityUnit/pyworkflow","owner":"QualityUnit","description":"Python Framework for Agentic Workflow management","archived":false,"fork":false,"pushed_at":"2026-04-05T18:08:27.000Z","size":2684,"stargazers_count":2,"open_issues_count":9,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-05T20:35:28.228Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/QualityUnit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-28T14:53:24.000Z","updated_at":"2026-04-05T18:08:22.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/QualityUnit/pyworkflow","commit_stats":null,"previous_names":["qualityunit/pyworkflow"],"tags_count":49,"template":false,"template_full_name":null,"purl":"pkg:github/QualityUnit/pyworkflow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QualityUnit%2Fpyworkflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QualityUnit%2Fpyworkflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QualityUnit%2Fpyworkflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QualityUnit%2Fpyworkflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/QualityUnit","download_url":"https://codeload.github.com/QualityUnit/pyworkflow/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QualityUnit%2Fpyworkflow/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31562697,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-08T14:31:17.711Z","status":"ssl_error","status_checked_at":"2026-04-08T14:31:17.202Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-28T03:27:29.780Z","updated_at":"2026-04-08T16:02:44.817Z","avatar_url":"https://github.com/QualityUnit.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PyWorkflow\n\n**Distributed, durable workflow orchestration for Python**\n\nBuild long-running, fault-tolerant workflows with automatic retry, sleep/delay capabilities, and complete observability. PyWorkflow uses event sourcing and Celery for production-grade distributed execution.\n\n---\n\n## What is PyWorkflow?\n\nPyWorkflow is a workflow orchestration framework that enables you to build complex, long-running business processes as simple Python code. It handles the hard parts of distributed systems: fault tolerance, automatic retries, state management, and horizontal scaling.\n\n### Key Features\n\n- **Distributed by Default**: All workflows execute across Celery workers for horizontal scaling\n- **Durable Execution**: Event sourcing ensures workflows can recover from any failure\n- **Auto Recovery**: Automatic workflow resumption after worker crashes with event replay\n- **Time Travel**: Sleep for minutes, hours, or days with automatic resumption\n- **Fault Tolerant**: Automatic retries with configurable backoff strategies\n- **Zero-Resource Suspension**: Workflows suspend without holding resources during sleep\n- **Production Ready**: Built on battle-tested Celery and Redis\n- **Fully Typed**: Complete type hints and Pydantic validation\n- **Observable**: Structured logging with workflow context\n\n---\n\n## Quick Start\n\n### Installation\n\n**Basic installation** (File and Memory storage backends):\n```bash\npip install pyworkflow-engine\n```\n\n**With optional storage backends:**\n```bash\n# Redis backend (includes Redis as Celery broker)\npip install pyworkflow-engine[redis]\n\n# SQLite backend\npip install pyworkflow-engine[sqlite]\n\n# PostgreSQL backend\npip install pyworkflow-engine[postgres]\n\n# All storage backends\npip install pyworkflow-engine[all]\n\n# Development (includes all backends + dev tools)\npip install pyworkflow-engine[dev]\n```\n\n### Prerequisites\n\n**For distributed execution** (recommended for production):\n\nPyWorkflow uses Celery for distributed task execution. You need a message broker:\n\n**Option 1: Redis (recommended)**\n```bash\n# Install Redis support\npip install pyworkflow-engine[redis]\n\n# Start Redis\ndocker run -d -p 6379:6379 redis:7-alpine\n\n# Start Celery worker(s)\ncelery -A pyworkflow.celery.app worker --loglevel=info\n\n# Start Celery Beat (for automatic sleep resumption)\ncelery -A pyworkflow.celery.app beat --loglevel=info\n```\n\nOr use the CLI to set up Docker infrastructure:\n```bash\npyworkflow setup\n```\n\n**Option 2: Other brokers** (RabbitMQ, etc.)\n```bash\n# Celery supports multiple brokers\n# Configure via environment: CELERY_BROKER_URL=amqp://localhost\n```\n\n**For local development/testing:**\n```bash\n# No broker needed - use in-process execution\npyworkflow configure --runtime local\n```\n\nSee [DISTRIBUTED.md](DISTRIBUTED.md) for complete deployment guide.\n\n### Your First Workflow\n\n```python\nfrom pyworkflow import workflow, step, start, sleep\n\n@step()\nasync def send_welcome_email(user_id: str):\n    # This runs on any available Celery worker\n    print(f\"Sending welcome email to user {user_id}\")\n    return f\"Email sent to {user_id}\"\n\n@step()\nasync def send_tips_email(user_id: str):\n    print(f\"Sending tips email to user {user_id}\")\n    return f\"Tips sent to {user_id}\"\n\n@workflow()\nasync def onboarding_workflow(user_id: str):\n    # Send welcome email immediately\n    await send_welcome_email(user_id)\n\n    # Sleep for 1 day - workflow suspends, zero resources used\n    await sleep(\"1d\")\n\n    # Automatically resumes after 1 day!\n    await send_tips_email(user_id)\n\n    return \"Onboarding complete\"\n\n# Start workflow - executes across Celery workers\nrun_id = start(onboarding_workflow, user_id=\"user_123\")\nprint(f\"Workflow started: {run_id}\")\n```\n\n**What happens:**\n1. Workflow starts on a Celery worker\n2. Welcome email is sent\n3. Workflow suspends after calling `sleep(\"1d\")`\n4. Worker is freed to handle other tasks\n5. After 1 day, Celery Beat automatically schedules resumption\n6. Workflow resumes on any available worker\n7. Tips email is sent\n\n---\n\n## Core Concepts\n\n### Workflows\n\nWorkflows are the top-level orchestration functions. They coordinate steps, handle business logic, and can sleep for extended periods.\n\n```python\nfrom pyworkflow import workflow, start\n\n@workflow(name=\"process_order\", max_duration=\"1h\")\nasync def process_order(order_id: str):\n    \"\"\"\n    Process a customer order.\n\n    This workflow:\n    - Validates the order\n    - Processes payment\n    - Creates shipment\n    - Sends confirmation\n    \"\"\"\n    order = await validate_order(order_id)\n    payment = await process_payment(order)\n    shipment = await create_shipment(order)\n    await send_confirmation(order)\n\n    return {\"order_id\": order_id, \"status\": \"completed\"}\n\n# Start the workflow\nrun_id = start(process_order, order_id=\"ORD-123\")\n```\n\n### Steps\n\nSteps are the building blocks of workflows. Each step is an isolated, retryable unit of work that runs on Celery workers.\n\n```python\nfrom pyworkflow import step, RetryableError, FatalError\n\n@step(max_retries=5, retry_delay=\"exponential\")\nasync def call_external_api(url: str):\n    \"\"\"\n    Call external API with automatic retry.\n\n    Retries up to 5 times with exponential backoff if it fails.\n    \"\"\"\n    try:\n        response = await httpx.get(url)\n\n        if response.status_code == 404:\n            # Don't retry - resource doesn't exist\n            raise FatalError(\"Resource not found\")\n\n        if response.status_code \u003e= 500:\n            # Retry - server error\n            raise RetryableError(\"Server error\", retry_after=\"30s\")\n\n        return response.json()\n    except httpx.NetworkError:\n        # Retry with exponential backoff\n        raise RetryableError(\"Network error\")\n```\n\n### Force Local Steps\n\nBy default, steps in a Celery runtime are dispatched to worker processes via the message broker. For lightweight steps where the broker round-trip overhead is undesirable, use `force_local=True` to execute the step inline in the orchestrator process:\n\n```python\nfrom pyworkflow import step\n\n@step(force_local=True)\nasync def quick_transform(data: dict):\n    \"\"\"Runs inline even when runtime is Celery.\"\"\"\n    return {k: v.upper() for k, v in data.items()}\n\n@step()\nasync def heavy_computation(data: dict):\n    \"\"\"Dispatched to a Celery worker as usual.\"\"\"\n    # ... expensive work ...\n    return result\n```\n\nForce-local steps still benefit from full durability: events (`STEP_STARTED`, `STEP_COMPLETED`) are recorded, results are cached for replay, and retry/timeout behavior is preserved. The only difference is that execution happens in the orchestrator process instead of a remote worker.\n\n**When to use `force_local`:**\n- Lightweight data transformations that finish in milliseconds\n- Steps that merely combine results from previous steps\n- Steps where broker serialization overhead exceeds the actual computation time\n\n**When NOT to use `force_local`:**\n- CPU-intensive or I/O-heavy steps (these benefit from worker distribution)\n- Steps that should scale independently from the orchestrator\n\n### Sleep and Delays\n\nWorkflows can sleep for any duration. During sleep, the workflow suspends and consumes zero resources.\n\n```python\nfrom pyworkflow import workflow, sleep\n\n@workflow()\nasync def scheduled_reminder(user_id: str):\n    # Send immediate reminder\n    await send_reminder(user_id, \"immediate\")\n\n    # Sleep for 1 hour\n    await sleep(\"1h\")\n    await send_reminder(user_id, \"1 hour later\")\n\n    # Sleep for 1 day\n    await sleep(\"1d\")\n    await send_reminder(user_id, \"1 day later\")\n\n    # Sleep for 1 week\n    await sleep(\"7d\")\n    await send_reminder(user_id, \"1 week later\")\n\n    return \"All reminders sent\"\n```\n\n**Supported formats:**\n- Duration strings: `\"5s\"`, `\"10m\"`, `\"2h\"`, `\"3d\"`\n- Timedelta: `timedelta(hours=2, minutes=30)`\n- Datetime: `datetime(2025, 12, 25, 9, 0, 0)`\n\n---\n\n## Architecture\n\n### Event-Sourced Execution\n\nPyWorkflow uses event sourcing to achieve durable, fault-tolerant execution:\n\n1. **All state changes are recorded as events** in an append-only log\n2. **Deterministic replay** enables workflow resumption from any point\n3. **Complete audit trail** of everything that happened in the workflow\n\n**Event Types** (16 total):\n- Workflow: `started`, `completed`, `failed`, `suspended`, `resumed`\n- Step: `started`, `completed`, `failed`, `retrying`\n- Sleep: `created`, `completed`\n- Logging: `info`, `warning`, `error`, `debug`\n\n### Distributed Execution\n\n```\n┌─────────────────────────────────────────────────────┐\n│                   Your Application                  │\n│                                                     │\n│  start(my_workflow, args)                          │\n│         │                                           │\n└─────────┼───────────────────────────────────────────┘\n          │\n          ▼\n    ┌─────────┐\n    │  Redis  │  ◄──── Message Broker\n    └─────────┘\n          │\n          ├──────┬──────┬──────┐\n          ▼      ▼      ▼      ▼\n     ┌──────┐ ┌──────┐ ┌──────┐\n     │Worker│ │Worker│ │Worker│  ◄──── Horizontal Scaling\n     └──────┘ └──────┘ └──────┘\n          │      │      │\n          └──────┴──────┘\n                 │\n                 ▼\n          ┌──────────┐\n          │ Storage  │  ◄──── Event Log (File/Redis/PostgreSQL)\n          └──────────┘\n```\n\n### Storage Backends\n\nPyWorkflow supports pluggable storage backends:\n\n| Backend | Status | Installation | Use Case |\n|---------|--------|--------------|----------|\n| **File** | ✅ Complete | Included | Development, single-machine |\n| **Memory** | ✅ Complete | Included | Testing, ephemeral workflows |\n| **SQLite** | ✅ Complete | `pip install pyworkflow-engine[sqlite]` | Embedded, local persistence |\n| **PostgreSQL** | ✅ Complete | `pip install pyworkflow-engine[postgres]` | Production, enterprise |\n| **Redis** | 📋 Planned | `pip install pyworkflow-engine[redis]` | High-performance, distributed |\n\n---\n\n## Advanced Features\n\n### Parallel Execution\n\nUse Python's native `asyncio.gather()` for parallel step execution:\n\n```python\nimport asyncio\nfrom pyworkflow import workflow, step\n\n@step()\nasync def fetch_user(user_id: str):\n    # Fetch user data\n    return {\"id\": user_id, \"name\": \"Alice\"}\n\n@step()\nasync def fetch_orders(user_id: str):\n    # Fetch user orders\n    return [{\"id\": \"ORD-1\"}, {\"id\": \"ORD-2\"}]\n\n@step()\nasync def fetch_recommendations(user_id: str):\n    # Fetch recommendations\n    return [\"Product A\", \"Product B\"]\n\n@workflow()\nasync def dashboard_data(user_id: str):\n    # Fetch all data in parallel\n    user, orders, recommendations = await asyncio.gather(\n        fetch_user(user_id),\n        fetch_orders(user_id),\n        fetch_recommendations(user_id)\n    )\n\n    return {\n        \"user\": user,\n        \"orders\": orders,\n        \"recommendations\": recommendations\n    }\n```\n\n### Error Handling\n\nPyWorkflow distinguishes between retriable and fatal errors:\n\n```python\nfrom pyworkflow import FatalError, RetryableError, step\n\n@step(max_retries=3, retry_delay=\"exponential\")\nasync def process_payment(amount: float):\n    try:\n        # Attempt payment\n        result = await payment_gateway.charge(amount)\n        return result\n    except InsufficientFundsError:\n        # Don't retry - user doesn't have enough money\n        raise FatalError(\"Insufficient funds\")\n    except PaymentGatewayTimeoutError:\n        # Retry - temporary issue\n        raise RetryableError(\"Gateway timeout\", retry_after=\"10s\")\n    except Exception as e:\n        # Unknown error - retry with backoff\n        raise RetryableError(f\"Unknown error: {e}\")\n```\n\n**Retry strategies:**\n- `retry_delay=\"fixed\"` - Fixed delay between retries (default: 60s)\n- `retry_delay=\"exponential\"` - Exponential backoff (1s, 2s, 4s, 8s, ...)\n- `retry_delay=\"5s\"` - Custom fixed delay\n\n### Auto Recovery\n\nWorkflows automatically recover from worker crashes:\n\n```python\nfrom pyworkflow import workflow, step, sleep\n\n@workflow(\n    recover_on_worker_loss=True,    # Enable recovery (default for durable)\n    max_recovery_attempts=5,         # Max recovery attempts\n)\nasync def resilient_workflow(data_id: str):\n    data = await fetch_data(data_id)    # Completed steps are skipped on recovery\n    await sleep(\"10m\")                   # Sleep state is preserved\n    return await process_data(data)      # Continues from here after crash\n```\n\n**What happens on worker crash:**\n1. Celery detects worker loss, requeues task\n2. New worker picks up the task\n3. Events are replayed to restore state\n4. Workflow resumes from last checkpoint\n\nConfigure globally:\n```python\nimport pyworkflow\n\npyworkflow.configure(\n    default_recover_on_worker_loss=True,\n    default_max_recovery_attempts=3,\n)\n```\n\nOr via config file:\n```yaml\n# pyworkflow.config.yaml\nrecovery:\n  recover_on_worker_loss: true\n  max_recovery_attempts: 3\n```\n\n### Idempotency\n\nPrevent duplicate workflow executions with idempotency keys:\n\n```python\nfrom pyworkflow import start\n\n# Same idempotency key = same workflow\nrun_id_1 = start(\n    process_order,\n    order_id=\"ORD-123\",\n    idempotency_key=\"order-ORD-123\"\n)\n\n# This will return the same run_id, not start a new workflow\nrun_id_2 = start(\n    process_order,\n    order_id=\"ORD-123\",\n    idempotency_key=\"order-ORD-123\"\n)\n\nassert run_id_1 == run_id_2  # True!\n```\n\n### Observability\n\nPyWorkflow includes structured logging with automatic context:\n\n```python\nfrom pyworkflow import configure_logging\n\n# Configure logging\nconfigure_logging(\n    level=\"INFO\",\n    log_file=\"workflow.log\",\n    json_logs=True,  # JSON format for production\n    show_context=True  # Include run_id, step_id, etc.\n)\n\n# Logs automatically include:\n# - run_id: Workflow execution ID\n# - workflow_name: Name of the workflow\n# - step_id: Current step ID\n# - step_name: Name of the step\n```\n\n---\n\n## Testing\n\nPyWorkflow uses a unified API for testing with local execution:\n\n```python\nimport pytest\nfrom pyworkflow import workflow, step, start, configure, reset_config\nfrom pyworkflow.storage.memory import InMemoryStorageBackend\n\n@step()\nasync def my_step(x: int):\n    return x * 2\n\n@workflow()\nasync def my_workflow(x: int):\n    result = await my_step(x)\n    return result + 1\n\n@pytest.fixture(autouse=True)\ndef setup_storage():\n    reset_config()\n    storage = InMemoryStorageBackend()\n    configure(storage=storage, default_durable=True)\n    yield storage\n    reset_config()\n\n@pytest.mark.asyncio\nasync def test_my_workflow(setup_storage):\n    storage = setup_storage\n    run_id = await start(my_workflow, 5)\n\n    # Get workflow result\n    run = await storage.get_run(run_id)\n    assert run.status.value == \"completed\"\n```\n\n---\n\n## Production Deployment\n\n### Docker Compose\n\n```yaml\nversion: '3.8'\n\nservices:\n  redis:\n    image: redis:7-alpine\n    ports:\n      - \"6379:6379\"\n\n  worker:\n    build: .\n    command: celery -A pyworkflow.celery.app worker --loglevel=info\n    depends_on:\n      - redis\n    deploy:\n      replicas: 3  # Run 3 workers\n\n  beat:\n    build: .\n    command: celery -A pyworkflow.celery.app beat --loglevel=info\n    depends_on:\n      - redis\n\n  flower:\n    build: .\n    command: celery -A pyworkflow.celery.app flower --port=5555\n    ports:\n      - \"5555:5555\"\n```\n\nStart everything using the CLI:\n```bash\npyworkflow setup\n```\n\nSee [DISTRIBUTED.md](DISTRIBUTED.md) for complete deployment guide with Kubernetes.\n\n---\n\n## Examples\n\nCheck out the [examples/](examples/) directory for complete working examples:\n\n- **[basic_workflow.py](examples/functional/basic_workflow.py)** - Complete example with retries, errors, and sleep\n- **[distributed_example.py](examples/functional/distributed_example.py)** - Multi-worker distributed execution example\n\n---\n\n## Project Status\n\n✅ **Status**: Production Ready (v1.0)\n\n**Completed Features**:\n- ✅ Core workflow and step execution\n- ✅ Event sourcing with 16 event types\n- ✅ Distributed execution via Celery\n- ✅ Sleep primitive with automatic resumption\n- ✅ Error handling and retry strategies\n- ✅ File storage backend\n- ✅ Structured logging\n- ✅ Comprehensive test coverage (68 tests)\n- ✅ Docker Compose deployment\n- ✅ Idempotency support\n\n**Next Milestones**:\n- 📋 Redis storage backend\n- 📋 PostgreSQL storage backend\n- 📋 Webhook integration\n- 📋 Web UI for monitoring\n- 📋 CLI management tools\n\n---\n\n## Contributing\n\nContributions are welcome!\n\n### Development Setup\n\n```bash\n# Clone repository\ngit clone https://github.com/QualityUnit/pyworkflow\ncd pyworkflow\n\n# Install with Poetry\npoetry install\n\n# Run tests\npoetry run pytest\n\n# Format code\npoetry run black pyworkflow tests\npoetry run ruff check pyworkflow tests\n\n# Type checking\npoetry run mypy pyworkflow\n```\n\n---\n\n## Documentation\n\n- **[Distributed Deployment Guide](DISTRIBUTED.md)** - Production deployment with Docker Compose and Kubernetes\n- [Examples](examples/) - Working examples and patterns\n- [API Reference](docs/api-reference.md) (Coming soon)\n- [Architecture Guide](docs/architecture.md) (Coming soon)\n\n---\n\n## License\n\nApache License 2.0 - See [LICENSE](LICENSE) file for details.\n\n---\n\n## Links\n\n- **Documentation**: https://docs.pyworkflow.dev\n- **GitHub**: https://github.com/QualityUnit/pyworkflow\n- **Issues**: https://github.com/QualityUnit/pyworkflow/issues\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqualityunit%2Fpyworkflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqualityunit%2Fpyworkflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqualityunit%2Fpyworkflow/lists"}