{"id":15291153,"url":"https://github.com/azure/go-asyncjob","last_synced_at":"2025-10-20T03:31:16.335Z","repository":{"id":42775989,"uuid":"510929029","full_name":"Azure/go-asyncjob","owner":"Azure","description":"asyncjob helps organize code in graph of blocks instead of sequential blocks.","archived":false,"fork":false,"pushed_at":"2025-01-27T22:54:40.000Z","size":276,"stargazers_count":18,"open_issues_count":3,"forks_count":6,"subscribers_count":57,"default_branch":"main","last_synced_at":"2025-01-30T13:51:10.379Z","etag":null,"topics":["async","go","golang","job","promises","workflow"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Azure.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":"SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-05T23:57:53.000Z","updated_at":"2024-11-17T20:03:27.000Z","dependencies_parsed_at":"2024-02-02T01:30:12.093Z","dependency_job_id":"453a376d-f001-421c-b1ec-e78d9a1f3712","html_url":"https://github.com/Azure/go-asyncjob","commit_stats":{"total_commits":109,"total_committers":7,"mean_commits":"15.571428571428571","dds":"0.45871559633027525","last_synced_commit":"1ce6bf11c889068ddffdeb8068b9aaff17e32b39"},"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure%2Fgo-asyncjob","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure%2Fgo-asyncjob/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure%2Fgo-asyncjob/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure%2Fgo-asyncjob/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Azure","download_url":"https://codeload.github.com/Azure/go-asyncjob/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237254184,"owners_count":19279999,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","go","golang","job","promises","workflow"],"created_at":"2024-09-30T16:11:10.704Z","updated_at":"2025-10-20T03:31:10.993Z","avatar_url":"https://github.com/Azure.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AsyncJob\n\nAsyncJob aiming to help you organize code in dependencyGraph(DAG), instead of a sequential chain.\n\n# Concepts\n**JobDefinition** is a graph describe code blocks and their connections.\n- you can use AddStep, StepAfter, StepAfterBoth to organize steps in a JobDefinition.\n- jobDefinition can be and should be build and seal in package init time.\n- jobDefinition have a generic typed input\n- calling Start with the input, will instantiate an jobInstance, and steps will began to execute.\n- jobDefinition can be visualized using graphviz, easier for human to understand.\n\n**JobInstance** is an instance of JobDefinition, after calling .Start() method from JobDefinition\n- all Steps on the definition will be copied to JobInstance.\n- each step will be executed once it's precedent step is done.\n- jobInstance can be visualized as well, instance visualize contains detailed info(startTime, duration) on each step.\n\n**StepDefinition** is a individual code block which can be executed and have inputs, output.\n- StepDefinition describe it's preceding steps.\n- StepDefinition contains generic Params\n- ideally all stepMethod should come from JobInput (generic type on JobDefinition), or static method. To avoid shared state between jobs.\n- output of a step can be feed into next step as input, type is checked by go generics.\n\n**StepInstance** is instance of StepDefinition\n- step is wrapped in [AsyncTask](https://github.com/Azure/go-asynctask)\n- a step would be started once all it's dependency is finished.\n- executionPolicy can be applied {Retry, ContextEnrichment}\n\n# Usage\n\n### Build and run a asyncjob\n```golang\n\n// SqlSummaryAsyncJobDefinition is the job definition for the SqlSummaryJobLib\n//   JobDefinition fit perfectly in init() function\nvar SqlSummaryAsyncJobDefinition *asyncjob.JobDefinitionWithResult[SqlSummaryJobLib, SummarizedResult]\n\nfunc init() {\n\tvar err error\n\tSqlSummaryAsyncJobDefinition, err = BuildJobWithResult(map[string]asyncjob.RetryPolicy{})\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\tSqlSummaryAsyncJobDefinition.Seal()\n}\n\nfunc BuildJob(retryPolicies map[string]asyncjob.RetryPolicy) (*asyncjob.JobDefinition[SqlSummaryJobLib], error) {\n\tjob := asyncjob.NewJobDefinition[SqlSummaryJobLib](\"sqlSummaryJob\")\n\n\tconnTsk, err := asyncjob.AddStep(job, \"GetConnection\", connectionStepFunc, asyncjob.WithRetry(retryPolicies[\"GetConnection\"]), asyncjob.WithContextEnrichment(EnrichContext))\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"error adding step GetConnection: %w\", err)\n\t}\n\n\tcheckAuthTask, err := asyncjob.AddStep(job, \"CheckAuth\", checkAuthStepFunc, asyncjob.WithContextEnrichment(EnrichContext))\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"error adding step CheckAuth: %w\", err)\n\t}\n\n\ttable1ClientTsk, err := asyncjob.StepAfter(job, \"GetTableClient1\", connTsk, tableClient1StepFunc, asyncjob.WithContextEnrichment(EnrichContext))\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"error adding step GetTableClient1: %w\", err)\n\t}\n\n\tqery1ResultTsk, err := asyncjob.StepAfter(job, \"QueryTable1\", table1ClientTsk, queryTable1StepFunc, asyncjob.WithRetry(retryPolicies[\"QueryTable1\"]), asyncjob.ExecuteAfter(checkAuthTask), asyncjob.WithContextEnrichment(EnrichContext))\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"error adding step QueryTable1: %w\", err)\n\t}\n\n\ttable2ClientTsk, err := asyncjob.StepAfter(job, \"GetTableClient2\", connTsk, tableClient2StepFunc, asyncjob.WithContextEnrichment(EnrichContext))\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"error adding step GetTableClient2: %w\", err)\n\t}\n\n\tqery2ResultTsk, err := asyncjob.StepAfter(job, \"QueryTable2\", table2ClientTsk, queryTable2StepFunc, asyncjob.WithRetry(retryPolicies[\"QueryTable2\"]), asyncjob.ExecuteAfter(checkAuthTask), asyncjob.WithContextEnrichment(EnrichContext))\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"error adding step QueryTable2: %w\", err)\n\t}\n\n\tsummaryTsk, err := asyncjob.StepAfterBoth(job, \"Summarize\", qery1ResultTsk, qery2ResultTsk, summarizeQueryResultStepFunc, asyncjob.WithRetry(retryPolicies[\"Summarize\"]), asyncjob.WithContextEnrichment(EnrichContext))\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"error adding step Summarize: %w\", err)\n\t}\n\n\t_, err = asyncjob.AddStep(job, \"EmailNotification\", emailNotificationStepFunc, asyncjob.ExecuteAfter(summaryTsk), asyncjob.WithContextEnrichment(EnrichContext))\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"error adding step EmailNotification: %w\", err)\n\t}\n\treturn job, nil\n}\n\t// execute job\n\tjobInstance1 := SqlSummaryAsyncJobDefinition.Start(ctx, \u0026SqlSummaryJobLib{...})\n\tjobInstance2 := SqlSummaryAsyncJobDefinition.Start(ctx, \u0026SqlSummaryJobLib{...})\n\n    // ...\n\n\tjobInstance1.Wait(context.WithTimeout(context.Background(), 10*time.Second))\n\tjobInstance2.Wait(context.WithTimeout(context.Background(), 10*time.Second))\n```\n\n### visualize of a job\n```\n\t# visualize the job\n\tdotGraph := job.Visualize()\n\tfmt.Println(dotGraph)\n```\n\n![visualize job graph](media/asyncjob.svg)\n\n```\ndigraph {\n\tnewrank = \"true\"\n\t\t\"QueryTable2\" [label=\"QueryTable2\" shape=hexagon style=filled tooltip=\"State: completed\\nStartAt: 2022-12-12T12:00:32.254054-08:00\\nDuration: 13.207µs\" fillcolor=green] \n\t\t\"QueryTable1\" [label=\"QueryTable1\" shape=hexagon style=filled tooltip=\"State: completed\\nStartAt: 2022-12-12T12:00:32.254098-08:00\\nDuration: 11.394µs\" fillcolor=green] \n\t\t\"EmailNotification\" [label=\"EmailNotification\" shape=hexagon style=filled tooltip=\"State: completed\\nStartAt: 2022-12-12T12:00:32.254143-08:00\\nDuration: 11.757µs\" fillcolor=green] \n\t\t\"sqlSummaryJob\" [label=\"sqlSummaryJob\" shape=triangle style=filled tooltip=\"State: completed\\nStartAt: 0001-01-01T00:00:00Z\\nDuration: 0s\" fillcolor=green] \n\t\t\"GetConnection\" [label=\"GetConnection\" shape=hexagon style=filled tooltip=\"State: completed\\nStartAt: 2022-12-12T12:00:32.253844-08:00\\nDuration: 154.825µs\" fillcolor=green] \n\t\t\"GetTableClient2\" [label=\"GetTableClient2\" shape=hexagon style=filled tooltip=\"State: completed\\nStartAt: 2022-12-12T12:00:32.254017-08:00\\nDuration: 25.793µs\" fillcolor=green] \n\t\t\"GetTableClient1\" [label=\"GetTableClient1\" shape=hexagon style=filled tooltip=\"State: completed\\nStartAt: 2022-12-12T12:00:32.254076-08:00\\nDuration: 12.459µs\" fillcolor=green] \n\t\t\"Summarize\" [label=\"Summarize\" shape=hexagon style=filled tooltip=\"State: completed\\nStartAt: 2022-12-12T12:00:32.254121-08:00\\nDuration: 7.88µs\" fillcolor=green] \n\t\t\"CheckAuth\" [label=\"CheckAuth\" shape=hexagon style=filled tooltip=\"State: completed\\nStartAt: 2022-12-12T12:00:32.253818-08:00\\nDuration: 18.52µs\" fillcolor=green] \n\n\t\t\"CheckAuth\" -\u003e \"QueryTable2\" [style=bold tooltip=\"Time: 2022-12-12T12:00:32.254054-08:00\" color=green] \n\t\t\"CheckAuth\" -\u003e \"QueryTable1\" [style=bold tooltip=\"Time: 2022-12-12T12:00:32.254098-08:00\" color=green] \n\t\t\"GetTableClient2\" -\u003e \"QueryTable2\" [style=bold tooltip=\"Time: 2022-12-12T12:00:32.254054-08:00\" color=green] \n\t\t\"GetTableClient1\" -\u003e \"QueryTable1\" [style=bold tooltip=\"Time: 2022-12-12T12:00:32.254098-08:00\" color=green] \n\t\t\"QueryTable1\" -\u003e \"Summarize\" [style=bold tooltip=\"Time: 2022-12-12T12:00:32.254121-08:00\" color=green] \n\t\t\"QueryTable2\" -\u003e \"Summarize\" [style=bold tooltip=\"Time: 2022-12-12T12:00:32.254121-08:00\" color=green] \n\t\t\"Summarize\" -\u003e \"EmailNotification\" [style=bold tooltip=\"Time: 2022-12-12T12:00:32.254143-08:00\" color=green] \n\t\t\"sqlSummaryJob\" -\u003e \"CheckAuth\" [style=bold tooltip=\"Time: 2022-12-12T12:00:32.253818-08:00\" color=green] \n\t\t\"sqlSummaryJob\" -\u003e \"GetConnection\" [style=bold tooltip=\"Time: 2022-12-12T12:00:32.253844-08:00\" color=green] \n\t\t\"GetConnection\" -\u003e \"GetTableClient2\" [style=bold tooltip=\"Time: 2022-12-12T12:00:32.254017-08:00\" color=green] \n\t\t\"GetConnection\" -\u003e \"GetTableClient1\" [style=bold tooltip=\"Time: 2022-12-12T12:00:32.254076-08:00\" color=green] \n}\n```\n\n### collect result from job\nyou can enrich job to aware result from given step, then you can collect result (strongly typed) from that step\n\n```\nvar SqlSummaryAsyncJobDefinition *asyncjob.JobDefinitionWithResult[SqlSummaryJobLib, SummarizedResult]\nSqlSummaryAsyncJobDefinition = asyncjob.JobWithResult(job /*from previous section*/, summaryTsk)\n\njobInstance1 := SqlSummaryAsyncJobDefinition.Start(ctx, \u0026SqlSummaryJobLib{...})\nresult, err := jobInstance1.Result(ctx)\n```\n\n### Overhead?\n- go routine will be created for each step in your jobDefinition, when you call .Start()\n- each step also hold tiny memory as well for state tracking.\n- userFunction is instrumented with state tracking, panic handling.\n\nHere is some simple visualize on how it actual looks like:\n```mermaid\ngantt\n    title       asyncjob.Start()\n    dateFormat  HH:mm\n\n    section GetConnection\n    WaitPrecedingTasks            :des11, 00:00,0ms\n    userFunction                  :des12, after des11, 20ms\n\n    section GetTableClient1\n    WaitPrecedingTasks            :des21, 00:00,20ms\n    userFunction                  :des22, after des21, 15ms\n\n    section GetTableClient2\n    WaitPrecedingTasks            :des31, 00:00,20ms\n    userFunction                  :des32, after des31, 21ms\n\n    section QueryTable1\n    WaitPrecedingTasks            :des41, 00:00,35ms\n    userFunction                  :des42, after des41, 24ms\n\n    section QueryTable2\n    WaitPrecedingTasks            :des51, 00:00,41ms\n    userFunction                  :des52, after des51, 30ms\n\n    section QueryResultSummarize\n    WaitPrecedingTasks            :des61, 00:00, 71ms\n    userFunction                  :des62, after des61, 10ms\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fazure%2Fgo-asyncjob","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fazure%2Fgo-asyncjob","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fazure%2Fgo-asyncjob/lists"}