{"id":19935987,"url":"https://github.com/imbolc/pg_task","last_synced_at":"2025-07-04T03:05:52.433Z","repository":{"id":179783492,"uuid":"663566325","full_name":"imbolc/pg_task","owner":"imbolc","description":"FSM-based Resumable Postgres tasks","archived":false,"fork":false,"pushed_at":"2024-12-27T08:08:31.000Z","size":67,"stargazers_count":21,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-06-02T09:32:05.554Z","etag":null,"topics":["fsm","postgres","postgresql","state-machine","task-manager","task-runner","task-scheduler"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/imbolc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-07T15:37:30.000Z","updated_at":"2024-12-27T08:08:35.000Z","dependencies_parsed_at":null,"dependency_job_id":"aa01edde-cd49-4e91-a9c6-f7093f763b3f","html_url":"https://github.com/imbolc/pg_task","commit_stats":null,"previous_names":["imbolc/pg-fsm","imbolc/pg_task"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/imbolc/pg_task","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imbolc%2Fpg_task","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imbolc%2Fpg_task/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imbolc%2Fpg_task/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imbolc%2Fpg_task/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/imbolc","download_url":"https://codeload.github.com/imbolc/pg_task/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imbolc%2Fpg_task/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263269769,"owners_count":23440262,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fsm","postgres","postgresql","state-machine","task-manager","task-runner","task-scheduler"],"created_at":"2024-11-12T23:22:55.501Z","updated_at":"2025-07-04T03:05:52.411Z","avatar_url":"https://github.com/imbolc.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pg_task\n\n[![License](https://img.shields.io/crates/l/pg_task.svg)](https://choosealicense.com/licenses/mit/)\n[![Crates.io](https://img.shields.io/crates/v/pg_task.svg)](https://crates.io/crates/pg_task)\n[![Docs.rs](https://docs.rs/pg_task/badge.svg)](https://docs.rs/pg_task)\n\nFSM-based Resumable Postgres tasks\n\n- **FSM-based** - each task is a granular state machine\n- **Resumable** - on error, after you fix the step logic or the external\n  world, the task is able to pick up where it stopped\n- **Postgres** - a single table is enough to handle task scheduling, state\n  transitions, and error processing\n\n## Table of Contents\n\n- [Tutorial](#tutorial)\n  - [Defining Tasks](#defining-tasks)\n  - [Investigating Errors](#investigating-errors)\n  - [Fixing the World](#fixing-the-world)\n- [Scheduling Tasks](#scheduling-tasks)\n- [Running Workers](#running-workers)\n- [Stopping Workers](#stopping-workers)\n- [Delaying Steps](#delaying-steps)\n- [Retrying Steps](#retrying-steps)\n\n## Tutorial\n\n_The full runnable code is in [examples/tutorial.rs][tutorial-example]._\n\n### Defining Tasks\n\nWe create a greeter task consisting of two steps:\n\n```rust,ignore\n#[derive(Debug, Deserialize, Serialize)]\npub struct ReadName {\n    filename: String,\n}\n\n#[async_trait]\nimpl Step\u003cGreeter\u003e for ReadName {\n    const RETRY_LIMIT: i32 = 5;\n\n    async fn step(self, _db: \u0026PgPool) -\u003e StepResult\u003cGreeter\u003e {\n        let name = std::fs::read_to_string(\u0026self.filename)?;\n        NextStep::now(SayHello { name })\n    }\n}\n```\n\nThe first step tries to read a name from a file:\n\n- `filename` - the only state we need in this step\n- `impl Step\u003cGreeter\u003e for ReadName` - our step is a part of a `Greeter` task\n- `RETRY_LIMIT` - the step is fallible, let's retry it a few times\n- `NextStep::now(SayHello { name })` - move our task to the `SayHello` step\n  right now\n\n```rust,ignore\n#[derive(Debug, Deserialize, Serialize)]\npub struct SayHello {\n    name: String,\n}\n#[async_trait]\nimpl Step\u003cGreeter\u003e for SayHello {\n    async fn step(self, _db: \u0026PgPool) -\u003e StepResult\u003cGreeter\u003e {\n        println!(\"Hello, {}\", self.name);\n        NextStep::none()\n    }\n}\n```\n\nThe second step prints the greeting and finishes the task returning\n`NextStep::none()`.\n\nThat's essentially all, except for some boilerplate you can find in the\n[full code][tutorial-example]. Let's run it:\n\n```bash\ncargo run --example hello\n```\n\n### Investigating Errors\n\nYou'll see log messages about the 6 (first try + `RETRY_LIMIT`) attempts and\nthe final error message. Let's look into the DB to find out what happened:\n\n```bash\n~$ psql pg_task -c 'table pg_task'\n-[ RECORD 1 ]------------------------------------------------\nid         | cddf7de1-1194-4bee-90c6-af73d9206ce2\nstep       | {\"Greeter\":{\"ReadName\":{\"filename\":\"name.txt\"}}}\nwakeup_at  | 2024-06-30 09:32:27.703599+06\ntried      | 6\nis_running | f\nerror      | No such file or directory (os error 2)\ncreated_at | 2024-06-30 09:32:22.628563+06\nupdated_at | 2024-06-30 09:32:27.703599+06\n```\n\n- a non-null `error` field indicates that the task has errored and contains\n  the error message\n- the `step` field provides you with the information about a particular step\n  and its state when the error occurred\n\n### Fixing the World\n\nIn this case, the error is due to the external world state. Let's fix it by\ncreating the file:\n\n```bash\necho 'Fixed World' \u003e name.txt\n```\n\nTo rerun the task, we just need to clear its `error`:\n\n```bash\npsql pg_task -c 'update pg_task set error = null'\n```\n\nYou'll see the log messages about rerunning the task and the greeting\nmessage of the final step. That's all 🎉.\n\n## Scheduling Tasks\n\nEssentially scheduling a task is done by inserting a corresponding row into\nthe `pg_task` table. You can do in by hands from `psql` or code in any\nlanguage.\n\nThere's also a few helpers to take care of the first step serialization and\ntime scheduling:\n- [`enqueue`] - to run the task immediately\n- [`delay`] - to run it with a delay\n- [`schedule`] - to schedule it to a particular time\n\n## Running Workers\n\nAfter [defining](#defining-tasks) the steps of each task, we need to\nwrap them into enums representing whole tasks via [`task!`]:\n\n```rust,ignore\npg_task::task!(Task1 { StepA, StepB });\npg_task::task!(Task2 { StepC });\n```\n\nOne more enum is needed to combine all the possible tasks:\n\n```rust,ignore\npg_task::scheduler!(Tasks { Task1, Task2 });\n```\n\nNow we can run the worker:\n\n```rust,ignore\npg_task::Worker::\u003cTasks\u003e::new(db).run().await?;\n```\n\nAll the communication is synchronized by the DB, so it doesn't matter how or\nhow many workers you run. It could be a separate process as well as\nin-process [`tokio::spawn`].\n\n## Stopping Workers\n\nYou can gracefully stop task runners by sending a notification using the\nDB:\n\n```sql\nSELECT pg_notify('pg_task_changed', 'stop_worker');\n```\n\nThe workers would wait until the current step of all the tasks is finished\nand then exit. You can wait for this by checking for the existence of\nrunning tasks:\n\n```sql\nSELECT EXISTS(SELECT 1 FROM pg_task WHERE is_running = true);\n```\n\n## Delaying Steps\n\nSometimes you need to delay the next step. Using [`tokio::time::sleep`]\nbefore returning the next step creates a couple of issues:\n\n- if the process is crashed while sleeping it wont be considered done and\n  will rerun on restart\n- you'd have to wait for the sleeping task to finish on [gracefulshutdown](#stopping-workers)\n\nUse [`NextStep::delay`] instead - it schedules the next step with the delay\nand finishes the current one right away.\n\nYou can find a runnable example in the [examples/delay.rs][delay-example]\n\n## Retrying Steps\n\nUse [`Step::RETRY_LIMIT`] and [`Step::RETRY_DELAY`] when you need to retry a\ntask on errors:\n\n```rust,ignore\nimpl Step\u003cMyTask\u003e for ApiRequest {\n    const RETRY_LIMIT: i32 = 5;\n    const RETRY_DELAY: Duration = Duration::from_secs(5);\n\n    async fn step(self, _db: \u0026PgPool) -\u003e StepResult\u003cMyTask\u003e {\n        let result = api_request().await?;\n        NextStep::now(ProcessResult { result })\n    }\n}\n```\n\n## Contributing\n\n- please run [.pre-commit.sh] before sending a PR, it will check everything\n\n## License\n\nThis project is licensed under the [MIT license](LICENSE).\n\n[.pre-commit.sh]: https://github.com/imbolc/pg_task/blob/main/.pre-commit.sh\n[delay-example]: https://github.com/imbolc/pg_task/blob/main/examples/delay.rs\n[tutorial-example]: https://github.com/imbolc/pg_task/blob/main/examples/tutorial.rs\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimbolc%2Fpg_task","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fimbolc%2Fpg_task","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimbolc%2Fpg_task/lists"}