{"id":37068103,"url":"https://github.com/dramwig/flowline","last_synced_at":"2026-01-14T08:00:11.294Z","repository":{"id":307782851,"uuid":"965593645","full_name":"Dramwig/FlowLine","owner":"Dramwig","description":"Automated tool for running Python programs in a streamlined manner","archived":false,"fork":false,"pushed_at":"2025-10-22T15:23:06.000Z","size":1000,"stargazers_count":308,"open_issues_count":2,"forks_count":19,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-22T16:31:44.387Z","etag":null,"topics":["automation","experiment-management","gpu-monitoring","machine-learning","python"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Dramwig.png","metadata":{"files":{"readme":"docs/readme_en.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-04-13T14:03:41.000Z","updated_at":"2025-10-22T15:05:32.000Z","dependencies_parsed_at":"2025-08-02T08:26:06.228Z","dependency_job_id":"268f5064-95aa-4ee5-921a-af736e59b5ae","html_url":"https://github.com/Dramwig/FlowLine","commit_stats":null,"previous_names":["dramwig/flowline"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/Dramwig/FlowLine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Dramwig%2FFlowLine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Dramwig%2FFlowLine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Dramwig%2FFlowLine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Dramwig%2FFlowLine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Dramwig","download_url":"https://codeload.github.com/Dramwig/FlowLine/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Dramwig%2FFlowLine/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28413527,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T05:26:33.345Z","status":"ssl_error","status_checked_at":"2026-01-14T05:21:57.251Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","experiment-management","gpu-monitoring","machine-learning","python"],"created_at":"2026-01-14T08:00:10.423Z","updated_at":"2026-01-14T08:00:11.284Z","avatar_url":"https://github.com/Dramwig.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- v0.1.3 --\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./fig/logo.png\" width=\"50%\" alt=\"FlowLine\" /\u003e\n\n  \u003c!-- [![LICENSE](https://badgen.net/static/LICENSE/Apache)](LICENSE) --\u003e\n\n  [中文](../readme.md) | English\n\u003c/div\u003e\n\n\n\nFlowLine is an automated system for **GPU resource management** and **concurrent command stream scheduling**, supporting both **Command Line Interface (CLI)** and **Web Graphical User Interface (GUI)** interaction modes. It is suitable for multi-task experiments, deep learning training, or high-concurrency computing environments.\n\n* 📘 **API Documentation**: See [API Docs](./docs/api.md)\n* 🧩 **System Design Overview**: See [Design Overview](./docs/design.md)\n* 🏗️ **System Architecture Details**: See [Architecture Documentation](./docs/arch.md)\n\nThe system was designed to replace the inefficient manual process of monitoring GPU status and executing commands sequentially. In traditional workflows, users need to continuously monitor GPU VRAM availability and usage to manually launch Python scripts or terminate processes, which is particularly cumbersome in multi-task experimental scenarios. This project addresses these issues through automation, improving experimental efficiency and resource utilization.\n\n## Core Features\n\n* Real-time GPU status monitoring: Automatically detects available GPU count, VRAM usage, process information, and selects the most appropriate GPU.\n* Command scheduling \u0026 resource control: Supports configuring conditions per command (required GPU count, minimum VRAM, max concurrency, etc.).\n* Dynamic control mechanisms: Allows manual termination or restarting of processes for flexible task queue management.\n* Concurrent multi-task execution: Supports task priority queues, failure retry policies, suitable for batch experiments.\n* Dual interaction modes: CLI for scripted control and batch deployment on Linux servers; Web GUI for visual task monitoring, status tracking, and real-time intervention.\n\n## 🚀 Quick Start Guide\n\n\u003e For a concise demonstration, see [example.sh](./example.sh) which provides all usage examples.\n\n### 🖥️ Using Command Line Interface (CLI Mode)\n\n#### 1. Installation\n\nYou can directly reference the `flowline` folder by copying it to your project root, or install it into your Python environment:\n\n- Install via pip:\n```bash\npip install fline\n```\n\n- Or install from source:\n```bash\npip install -e \u003cpath_to_flowline_repository\u003e\n```\n\n\u003e Note: Ensure you have installed basic dependencies from `requirements.txt` (`pandas`, `psutil`, `openpyxl`, etc.).\n\n#### 2. Create Task Control Sheet\n\nThe system uses a list file (`.xlsx`、 `.csv` or `.json` format) to define task parameters. **This is the only input method for all tasks.** Each row represents an independent task, and each column corresponds to a parameter that will be automatically mapped to `--key value` CLI format.\n\n\u003cdetails\u003e\n\u003csummary\u003eExample and Explanation\u003c/summary\u003e\n\nExample files: [`test/example1_todo.xlsx`](./test/example1_todo.xlsx), [`test/example1_todo.csv`](./test/example1_todo.csv),[`test/example1_todo.json`](./test/example1_todo.json), which can be constructed using the example program [`test/task_builder.py`](./test/task_builder.py).\n\n| *name*    | lr    | batch_size | *run_num* | *need_run_num* | *cmd*       |\n| --------- | ----- | ---------- | --------- | -------------- | ----------- |\n| baseline1 | 0.01  | 64         | 0         | 1              | train_main  |\n| baseline2 | 0.001 | 128        | 0         | 2              | train_alt   |\n\nField descriptions:\n* `run_num`: Current execution count (automatically maintained by system, default=0).\n* `need_run_num`: Total desired executions (system controls repeats based on this, default=1).\n* `name`: Task identifier. Auto-generated as `Task:{row_number}` if unspecified.\n* `cmd`: Reserved field (can be empty or specify main command like `train_main`). Can be used with custom `func` logic.\n* Other fields can be freely defined and will be passed to the command constructor.\n\n\u003e Note: If reserved fields are missing, **the system will auto-complete them during loading** to ensure valid structure.\n\nThe flexible task sheet structure supports everything from parameter tuning to complex grid search automation.\n\n\u003c/details\u003e\n\n#### 3. Define Task Constructor `func(dict, gpu_id)`\n\nYou need to define a custom function that constructs the final command string using the task parameters `dict` (from Excel row) and allocated `gpu_id`.\n\n\u003cdetails\u003e\n\u003csummary\u003eExample and Explanation\u003c/summary\u003e\n\nExample:\n```python\nfrom flowline.api import run_cli\n\nif __name__ == \"__main__\":\n    def func(param_dict, gpu_id):\n        cmd = \"CUDA_VISIBLE_DEVICES=\" + str(gpu_id) + \" python -u test/test.py \"\n        args = \" \".join([f\"--{k} {v}\" for k, v in param_dict.items()])\n        return cmd + args\n\n    run_cli(func, \"test/example1_todo.xlsx\")\n```\n\n* `param_dict`: Dictionary built from current Excel row (keys=column names, values=cell content)\n* `gpu_id`: Dynamically allocated GPU ID (ensures no conflicts)\n* Returned command string executes as a subprocess (equivalent to direct CLI execution)\n* Can be adapted for shell scripts, conda environments, or main command variants\n\n\u003cdetails\u003e\n\u003csummary\u003eAbout Output and python -u\u003c/summary\u003e\n\n💡 **About `python -u`:**\nUsing `-u` flag (`python -u ...`) enables **unbuffered mode**:\n* `stdout`/`stderr` flush immediately\n* Essential for real-time log viewing (especially when output is redirected)\n* FlowLine saves each task's output to `log/` directory:\n\n```\nlog/\n├── 0.out    # stdout for task 0\n├── 0.err    # stderr for task 0\n├── 1.out\n├── 1.err\n...\n```\n\nAlways use `-u` to ensure **real-time log writing** to these files.\n\u003c/details\u003e\n\u003c/details\u003e\n\n#### 4. Enter `run` to start the task flow\n\n\u003cdetails\u003e\n\u003csummary\u003eFlowLine CLI Command Reference Table\u003c/summary\u003e\n\n| Command        | Parameter                 | Description                                                                                                |\n| -------------- | ------------------------- | ---------------------------------------------------------------------------------------------------------- |\n| `run`          | None                      | Toggles the task processing loop state (start/stop)                                                        |\n| `gpu \u003cid\u003e`     | `\u003cid\u003e`: GPU ID            | Toggles the availability of the specified GPU (available/unavailable)                                      |\n| `killgpu \u003cid\u003e` | `\u003cid\u003e`: GPU ID            | Terminates all processes running on the specified GPU                                                      |\n| `kill \u003cid\u003e`    | `\u003cid\u003e`: Process ID        | Terminates the process with the specified process ID                                                       |\n| `ls`           | None                      | Lists all running processes, showing process ID, PID, task ID, GPU ID, status, and command                 |\n| `gpus`         | None                      | Displays the status of all GPUs, including utilization, memory usage, temperature, power consumption, etc. |\n| `min \u003cnum\u003e`    | `\u003cnum\u003e`: Memory size (MB) | Sets the minimum required memory for processes                                                             |\n| `max \u003cnum\u003e`    | `\u003cnum\u003e`: Process count    | Sets the maximum number of concurrent processes                                                            |\n| `task`         | None                      | Lists the pending task queue, showing task ID, name, run count, etc.                                       |\n| `exit`         | None                      | Exits the program (equivalent to `Ctrl+D`)                                                                 |\n| `help` or `?`  | None                      | Displays help information                                                                                  |\n\n\u003cdetails\u003e\n\u003csummary\u003eCommand Usage Examples\u003c/summary\u003e\n\n```bash\n# Start the task processing loop\n\u003e run\n\n# Check GPU status\n\u003e gpus\n\n# View running processes\n\u003e ls\n\n# Set the maximum number of concurrent processes to 4\n\u003e max 4\n\n# Set the minimum memory requirement to 2048 MB\n\u003e min 2048\n\n# Disable GPU 1\n\u003e gpu 1\n\n# Terminate all processes on GPU 0\n\u003e killgpu 0\n\n# View pending tasks\n\u003e task\n\n# Exit the program\n\u003e exit\n```\n\n\u003c/details\u003e\n\u003c/details\u003e\n\n\n### 🌐 Using Web Interface (Visual Task Management)\n\n\u003e **No extra configuration needed - Works directly in SSH environments**\n\nBesides CLI, you can use the Web GUI for **real-time monitoring and dynamic intervention**.\n\n#### 1. Start Backend API Service\nRun the Flask backend:\n```bash\npython test/example_server.py\n```\n\n\u003e Note: Web interface uses `from flowline.api.routes import get_app` to import routes.\n\n#### 2. Start Frontend Service\nLaunch static file server:\n```bash\ncd web\npython -m http.server 8000\n```\n\nAccess the frontend at [http://localhost:8000](http://localhost:8000/). The interface communicates with backend via RESTful APIs.\n\n\u003cdiv align=center\u003e\n  \u003cimg src=\"./docs/fig/gpu.png\" alt=\"GPU Monitoring\" height=\"200px\" /\u003e\n  \u003cimg src=\"./docs/fig/task.png\" alt=\"Task Management\" height=\"200px\" /\u003e\n  \u003cimg src=\"./docs/fig/log.png\" alt=\"Log Viewer\" height=\"200px\" /\u003e\n  \u003cimg src=\"./docs/fig/set.png\" alt=\"Settings\" height=\"200px\" /\u003e\n\u003c/div\u003e\n\n## 🛑 Disclaimer\n\nThis project provides **automated detection and utilization of idle GPUs** for resource-constrained environments (e.g., labs), enabling rapid task initiation without manual polling.\n\n### 📌 Important Notes\n- This tool **does NOT forcibly terminate others' tasks** or bypass permission/scheduling systems.\n- Default operation is **limited to devices where user has access permissions**. Comply with institutional policies.\n- **DO NOT misuse to monopolize shared resources or disrupt others' research.**\n\n### 🚨 Risk Statement\n\nPotential risks include but not limited to:\n- Resource conflicts from concurrent scheduling\n- Violation of lab/platform policies if abused\n\nDevelopers **shall not be liable** for any direct/indirect losses including resource conflicts, account restrictions, or data loss resulting from script usage.\n\n\n## 💐 Contributions\n\nWe welcome everyone to contribute code, fix bugs, or improve the documentation for this template!\n\n- If you have any suggestions or questions, please submit an issue.\n- Pull requests are welcome.\n  \n\u003e [!TIP]\n\u003e If this project is helpful to you, please give it a **Star**!\n\n**Thanks to all contributors!**\n\n[![Contributors](https://contrib.rocks/image?repo=dramwig/FlowLine)](https://github.com/dramwig/FlowLine/graphs/contributors)\n\n\u003ca href=\"https://www.star-history.com/#dramwig/FlowLine\u0026Date\"\u003e\n\u003cpicture\u003e\n  \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://api.star-history.com/svg?repos=dramwig/FlowLine\u0026type=Date\u0026theme=dark\" /\u003e\n   \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"https://api.star-history.com/svg?repos=dramwig/FlowLine\u0026type=Date\" /\u003e\n   \n\u003cimg alt=\"Star History Chart\" src=\"https://api.star-history.com/svg?repos=dramwig/FlowLine\u0026type=Date\" /\u003e\n \u003c/picture\u003e\n\u003c/a\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdramwig%2Fflowline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdramwig%2Fflowline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdramwig%2Fflowline/lists"}