{"id":31260000,"url":"https://github.com/mathew005/workflow-engine","last_synced_at":"2025-09-23T08:46:24.274Z","repository":{"id":314903025,"uuid":"1057010231","full_name":"Mathew005/workflow-engine","owner":"Mathew005","description":"A declarative AI workflow engine to build, run, and visualize multi-step pipelines using YAML. Powered by LangGraph with a real-time Streamlit UI.","archived":false,"fork":false,"pushed_at":"2025-09-15T13:39:22.000Z","size":97,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-15T15:21:55.932Z","etag":null,"topics":["ai-workflow","declarative-programming","gemini-api","generative-ai","langgraph","llm","orchestration","pipelines-as-code","python","streamlit","workflow-engine","yaml"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Mathew005.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-15T06:49:00.000Z","updated_at":"2025-09-15T13:39:25.000Z","dependencies_parsed_at":"2025-09-15T15:21:58.223Z","dependency_job_id":"23ba7fa5-7e9a-45ad-a3c2-e06f060257f6","html_url":"https://github.com/Mathew005/workflow-engine","commit_stats":null,"previous_names":["mathew005/workflow-engine"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/Mathew005/workflow-engine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mathew005%2Fworkflow-engine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mathew005%2Fworkflow-engine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mathew005%2Fworkflow-engine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mathew005%2Fworkflow-engine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Mathew005","download_url":"https://codeload.github.com/Mathew005/workflow-engine/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mathew005%2Fworkflow-engine/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":276545728,"owners_count":25661361,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-23T02:00:09.130Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-workflow","declarative-programming","gemini-api","generative-ai","langgraph","llm","orchestration","pipelines-as-code","python","streamlit","workflow-engine","yaml"],"created_at":"2025-09-23T08:46:23.187Z","updated_at":"2025-09-23T08:46:24.262Z","avatar_url":"https://github.com/Mathew005.png","language":"Python","readme":"# Declarative AI Workflow Engine\n\n[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)\n[![LangGraph](https://img.shields.io/badge/LangGraph-Built%20with-green)](https://python.langchain.com/docs/langgraph)\n[![Streamlit](https://img.shields.io/badge/Streamlit-UI-red)](https://streamlit.io/)\n[![Gemini API](https://img.shields.io/badge/Google-Gemini%20API-blue)](https://ai.google.dev/)\n\nThis project is a powerful, declarative AI workflow engine that lets you build and run complex, multi-step tasks using simple YAML files. It uses a Streamlit web interface to provide a live, color-coded graph visualization of your workflow as it executes step-by-step.\n\n## Features\n\n*   **Declarative YAML Workflows:** Define complex logic, dependencies, and data flow in a human-readable format.\n*   **Live Visualizations:** Watch your workflow's graph (DAG) and sub-workflows update in real-time as each step runs, succeeds, or fails.\n*   **Custom Python Nodes:** Inject your own Python code as a step in any workflow. The engine automatically discovers and integrates your code.\n*   **Reusable Workflows (Composition):** Call one workflow from another, allowing you to build complex systems from smaller, reusable parts.\n*   **Conditional Logic (Branching):** Create workflows that make decisions and take different paths based on data, using a `conditional_router`.\n*   **Dynamic Mapping (Fan-Out):** Run a step in parallel for each item in a list, enabling powerful batch processing.\n*   **Multimodal Inputs:** Use files (images, PDFs) as inputs to your workflows for tasks like image analysis or document processing.\n\n## DAG Visualization\n\n![Run](https://github.com/user-attachments/assets/c1908375-d984-4040-8547-26305c8bb30e)\n\n## Project Structure\n\n```text\nworkflow-engine-poc/\n├── .env\n├── .env.example\n├── run.bat                 # Runner for Windows\n├── run.sh                  # Runner for Mac/Linux\n└── src/\n    ├── app/streamlit_app.py\n    ├── custom_code/\n    │   ├── base.py\n    │   ├── __init__.py\n    │   └── steps/            # \u003c-- Add your custom Python files here\n    │       ├── content_processing.py\n    │       ├── image_processing.py\n    │       └── text_analysis.py\n    ├── services/             # Core engine logic\n    ├── shared_prompts/       # \u003c-- Add reusable LLM prompts here\n    └── workflows/            # \u003c-- Add your workflow packages here\n        ├── 1_Basic_Features_Demo/\n        ├── 2_Graph_Structures_Demo/\n        ├── 3_Advanced_Logic_Demo/\n        ├── 4_Master_Orchestrator_Demo/\n        └── 5_Multimodal_Analysis/```\n\n## Setup\n\n#### 1. Prerequisites\n*   Python 3.10 or higher\n\n#### 2. Get the Code\n```bash\ngit clone \u003cyour-repository-url\u003e\ncd workflow-engine-poc\n```\n\n#### 3. Create a Virtual Environment\n*On Mac/Linux:*\n```bash\npython3 -m venv venv\nsource venv/bin/activate\n```\n*On Windows:*\n```bash\npython -m venv venv\n.\\venv\\Scripts\\activate\n```\n\n#### 4. Install Dependencies\n```bash\npip install streamlit google-generativeai pydantic pydantic-settings graphviz langchain-core langgraph nest-asyncio httpx\n```\n\n#### 5. Set Up API Keys\nCopy the example environment file.\n\n```bash\ncp .env.example .env\n```\nOpen the new `.env` file and add your Google Gemini API Key(s).\n\n```ini\n# .env\nGEMINI_API_KEY=\"AIzaSy...YourFirstKey\"\n#GEMINI_API_KEY=\"AIzaSy...YourSecondKey\"\n\nMONGO_URI=\"mongodb://localhost:27017/\"\n```\n\n## How to Run\n\n*On Mac/Linux:*\n```bash\nbash run.sh\n```\n\n*On Windows:*\n```bat\nrun.bat\n```\n\nOpen your web browser and navigate to the local URL shown in your terminal.\n\n---\n\n## Developer Guide: A Tour of the Showcase Workflows\n\nThe best way to understand the engine is to explore the included showcase workflows. Each one is designed to demonstrate a specific set of features, building from the basics to the most advanced capabilities.\n\n### Part 1: The Basics (`1_Basic_Features_Demo`)\n\nThis workflow demonstrates the three fundamental step types in a simple sequence: `llm`, `code`, and `api`.\n\n**`src/workflows/1_Basic_Features_Demo/workflow.yaml`**\n```yaml\nsteps:\n  - name: \"generate_article_idea\"\n    type: \"llm\"       # 1. An LLM call to generate content.\n    dependencies: []\n    params:\n      prompt_template: \"1_generate_idea.txt\"\n      input_mapping: { topic: \"topic\" }\n      output_key: \"article_idea\"\n\n  - name: \"validate_title_length\"\n    type: \"code\"      # 2. A custom Python node to perform business logic.\n    dependencies:\n      - \"article_idea\"\n    params:\n      function_name: \"content_processing.ValidateTitleStep\"\n      input_mapping: { title: \"article_idea.title\" }\n      output_key: \"title_validation\"\n\n  - name: \"fetch_related_post\"\n    type: \"api\"       # 3. An external API call to fetch data.\n    dependencies:\n      - \"title_validation\"\n    params:\n      method: \"GET\"\n      endpoint: \"https://jsonplaceholder.typicode.com/posts/1\"\n      output_key: \"related_post_data\"\n```\n\n### Part 2: Graph Structures (`2_Graph_Structures_Demo`)\n\nThis workflow showcases parallel execution (fan-out) where three steps run simultaneously, followed by a synchronization step (fan-in) that waits for all of them to complete. It also uses a prompt from the `shared_prompts` directory.\n\n**`src/workflows/2_Graph_Structures_Demo/workflow.yaml`**\n```yaml\nsteps:\n  # These three steps have no dependencies, so they run in parallel.\n  - name: \"analyze_sentiment\"\n    type: \"llm\"\n    params:\n      prompt_template: \"analyze_sentiment.txt\" # Uses a shared prompt\n      # ...\n\n  - name: \"extract_hashtags\"\n    type: \"llm\"\n    # ...\n\n  - name: \"get_text_statistics\"\n    type: \"code\"\n    # ...\n\n  # This step will only run after the three steps above have all finished.\n  - name: \"synthesize_engagement_report\"\n    type: \"llm\"\n    dependencies:\n      - \"sentiment_result\"\n      - \"hashtags_result\"\n      - \"stats_result\"\n    # ...\n    ```\n\n### Part 3: Advanced Logic (`3_Advanced_Logic_Demo`)\n\nThis workflow demonstrates the engine's most powerful logic features:\n1.  **Dynamic Mapping:** The `get_length_of_each_title` step runs a custom code node in parallel for each item in the input list.\n2.  **Conditional Routing:** The `content_strategy_router` step makes a decision based on the data and directs the workflow down one of two different branches.\n\n**`src/workflows/3_Advanced_Logic_Demo/workflow.yaml`**\n```yaml\nsteps:\n  - name: \"get_length_of_each_title\"\n    type: \"code\"\n    params:\n      map_input: \"article_titles\" # 1. Fan-out over this list\n      function_name: \"content_processing.ValidateTitleStep\"\n      input_mapping:\n        title: \"item\" # 'item' refers to each element in the mapped list\n      output_key: \"title_lengths\"\n\n  - name: \"calculate_average_length\"\n    type: \"code\"\n    dependencies: [\"title_lengths\"]\n    # ...\n    output_key: \"average_analysis\"\n\n  - name: \"content_strategy_router\"\n    type: \"conditional_router\" # 2. A decision-making step\n    dependencies: [\"average_analysis\"]\n    params:\n      condition_key: \"average_analysis.decision\"\n      routing_map:\n        \"short\": \"suggest_expansions\" # If decision is \"short\", go here.\n        \"long\": \"suggest_summaries\"  # If decision is \"long\", go here.\n  \n  - name: \"suggest_expansions\"\n    type: \"llm\" # This is one branch.\n    # ...\n  \n  - name: \"suggest_summaries\"\n    type: \"llm\" # This is the other branch.\n    # ...\n```\n\n### Part 4: Composition \u0026 Multimodality (`4`, `5`, `6`)\n\nThese workflows showcase the final set of advanced features:\n*   **`4_Master_Orchestrator_Demo`:** Demonstrates composition by using `type: \"workflow\"` to call other workflows as single, reusable steps.\n*   **`5_Multimodal_Analysis`:** Shows how to use `type: \"file\"` to accept image uploads and process them with a multimodal LLM.\n*   **`6_Advanced_Mapping_Demo`:** The ultimate showcase. It uses `map_input` on a `type: \"workflow\"` step to run an entire, complex sub-workflow in parallel for each item in a list, demonstrating the UI's unique ability to track multiple, indexed DAGs in real-time.","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmathew005%2Fworkflow-engine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmathew005%2Fworkflow-engine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmathew005%2Fworkflow-engine/lists"}