{"id":21024382,"url":"https://github.com/matroskin13/stepper","last_synced_at":"2026-03-05T06:31:30.743Z","repository":{"id":62973607,"uuid":"563047653","full_name":"matroskin13/stepper","owner":"matroskin13","description":"A simple, efficient, concurrent task runner. Stepper supports MongoDB, Postgresql (beta).","archived":false,"fork":false,"pushed_at":"2023-09-21T09:21:52.000Z","size":93,"stargazers_count":23,"open_issues_count":3,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-27T01:39:47.617Z","etag":null,"topics":["go","golang","job-scheduler","queue","scheduler"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/matroskin13.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-07T19:54:15.000Z","updated_at":"2025-03-15T23:21:35.000Z","dependencies_parsed_at":"2024-06-21T08:40:08.084Z","dependency_job_id":"4ae15f21-f14c-4241-a2a9-763075b82104","html_url":"https://github.com/matroskin13/stepper","commit_stats":{"total_commits":23,"total_committers":2,"mean_commits":11.5,"dds":0.08695652173913049,"last_synced_commit":"c7d91fbb87f92a57e17a0ca9f1e0281571106195"},"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/matroskin13/stepper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matroskin13%2Fstepper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matroskin13%2Fstepper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matroskin13%2Fstepper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matroskin13%2Fstepper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/matroskin13","download_url":"https://codeload.github.com/matroskin13/stepper/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matroskin13%2Fstepper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30112226,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T03:40:26.266Z","status":"ssl_error","status_checked_at":"2026-03-05T03:39:15.902Z","response_time":93,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","job-scheduler","queue","scheduler"],"created_at":"2024-11-19T11:25:44.816Z","updated_at":"2026-03-05T06:31:30.706Z","avatar_url":"https://github.com/matroskin13.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Stepper\n\nA simple, efficient, concurrent task runner.\n\n* **Simple.** Run tasks and schedule jobs with GO.\n* **Database agnostic.** Stepper supports MongoDB, Postgresql (beta).\n* **Concurrent.** Stepper can be used in an unlimited number of instances.\n* **Scalable.** Split one task into small subtasks which will run on different nodes.\n\n## Install\n\n```bash\ngo get github.com/matroskin13/stepper\n```\n\n```bash\ngo get github.com/matroskin13/stepper/engines/mongo\n```\n\nor\n\n```bash\ngo get github.com/matroskin13/stepper/engines/pg\n```\n\n## Getting started\n\n```go\npackage main\n\nimport (\n    \"log\"\n\n    \"github.com/matroskin13/stepper\"\n    \"github.com/matroskin13/stepper/engines/mongo\"\n)\n\nfunc main() {\n    mongoEngine, err := mongo.NewMongo(\"mongodb://localhost:27017\", \"example_database\")\n    if err != nil {\n        log.Fatal(err)\n    }\n\n    ctx := context.Background()\n\n    service := stepper.NewService(mongoEngine)\n\n    // Will publish a task on startup\n    if err := service.Publish(ctx, \"example-task\", []byte(\"Hello world\")); err != nil {\n        log.Fatal(err)\n    }\n\n    service.TaskHandler(\"example-task\", func(ctx stepper.Context, data []byte) error {\n        fmt.Println(string(data))\n\n        return nil\n    })\n\n    if err := service.Listen(ctx); err != nil {\n        log.Fatal(err)\n    }\n}\n```\n\nOr you can use PostgresQL:\n\n```go\nimport \"github.com/matroskin13/stepper/engines/pg\"\n```\n\n```go\nengine, err := pg.NewPG(\"postgres://postgres:test@localhost:5432/postgres\")\nif err != nil {\n    log.Fatal(err)\n}\n```\n\n\n## Table of Contents\n\n* [Stepper](#stepper)\n  * [Publish task](#publish-task)\n    * [Simple way](#simple-way)\n    * [Publish with delay](#publish-with-delay)\n  * [Execute a task](#execute-a-task)\n    * [Simple way](#simple-way-1)\n    * [Error handling](#error-handling)\n    * [Bind a state](#bind-a-state)\n  * [Subtasks](#subtasks)\n    * [Create a subtask](#create-a-subtask)\n  * [Repeated tasks](#repeated-tasks)\n  * [Middlewares](#middlewares)\n    * [Retry](#retry)\n    * [Prometheus](#prometheus)\n\n## Publish task\n\nIf you use the stepper you will use a lot of things but first of all you will publish and execute tasks. Let's discuss how you can publish tasks.\n\n### Simple way\n\n```go\nservice.Publish(context.Background(), \"example-task\", []byte(\"hello\"))\n```\n\nThe example shows the simple way to publish a task. The code will publish a task with a name **example-task** and content **hello**.\n\nBut also the stepper allows you to use additional options.\n\n### Publish with delay\n\nIf you don't want to execute a task immediately you can set up a delay.\n\n```go\nservice.Publish(\n    context.Background(),\n    \"example-task\",\n    []byte(\"hello\"),\n    stepper.SetDelay(time.Minute * 1),\n)\n```\n\nOr you can use particular a date\n\n```go\nservice.Publish(\n    context.Background(),\n    \"example-task\",\n    []byte(\"hello\"),\n    stepper.LaunchAt(time.Now().Add(time.Minute * 10)),\n)\n```\n\n## Execute a task\n\nThe second part of the Stepper is execution of tasks in queue.\n\n### Simple way\n\n```go\ns.TaskHandler(\"example-task\", func(ctx stepper.Context, data []byte) error {\n    fmt.Println(string(data))\n\n    return nil\n})\n```\n\nThe example shows the simple way to execute a task.\n\n### Error handling\n\nIf your handler returns an error, a task will be returned to the queue. And the task will be held in the queue for 10 seconds. But you can set up a delay manually.\n\n```go\ns.TaskHandler(\"example-task\", func(ctx stepper.Context, data []byte) error {\n    ctx.SetRetryAfter(time.Minute) // will be returned in 1 minute\n    return fmt.Errorf(\"some error\")\n})\n```\n\n### Bind a state\n\nIf you have a log running task, you can bind a state of task (cursor for example), and if your task failed you will be able to continue the task with the last state \n\n```go\ns.TaskHandler(\"example-task\", func(ctx stepper.Context, data []byte) error {\n    var lastId int\n\n    if err := ctx.BindState(\u0026lastId); err != nil {\n        return err\n    }\n\n    iter := getSomethingFromId(lastId) // something like a mongodb iterator or anything else\n\n    for iter.Next() {\n        lastId = ... // do something\n\n        if err := ctx.SetState(lastId); err != nil {\n            return err\n        }\n    }\n\n    return nil\n})\n```\n\n## Subtasks\n\nThe most powerful feature of the stepper is creating subtasks. The feature allows you to split a long-running task into separate tasks which will run on different nodes. And when all subtasks will be completed the stepper will call a `onFinish` hook of parent task.\n\n### Create a subtask\n\nThe following example shows how to spawn subtasks within a main task.\n\n```go\ns.TaskHandler(\"task-with-threads\", func(ctx stepper.Context, data []byte) error {\n    fmt.Println(\"have received the word for splitting: \", string(data))\n\n    for _, symbol := range strings.Split(string(data), \"\") {\n        ctx.CreateSubtask(stepper.CreateTask{\n            Data: []byte(symbol),\n        })\n    }\n\n    return nil\n}).Subtask(func(ctx stepper.Context, data []byte) error {\n    fmt.Printf(\"[letter-subtask]: have received symbol: %s\\r\\n\", data)\n    return nil\n}).OnFinish(func(ctx stepper.Context, data []byte) error {\n    fmt.Println(\"subtasks are over\")\n    return nil\n})\n```\n\nOr you can use existing a subtask:\n\n```go\nctx.CreateSubtask(stepper.CreateTask{\n    Name: \"some-task\",\n    Data: []byte(symbol),\n})\n```\n\n## Repeated tasks\n\nIf you want to run repeatead task (cron) you can use jobs\n\n```go\ns.RegisterJob(context.Background(), \u0026stepper.JobConfig{\n    Name:    \"log-job\",\n    Pattern: \"@every 10s\",\n}, func(ctx stepper.Context) error {\n    fmt.Println(\"wake up the log-job\")\n    return nil\n})\n```\n\nRead https://pkg.go.dev/github.com/robfig/cron#hdr-CRON_Expression_Format for more information about a pattern.\n\nAlso you can create subtasks from a job:\n\n```go\ns.RegisterJob(context.Background(), \u0026stepper.JobConfig{\n    Name:    \"log-job\",\n    Pattern: \"@every 10s\",\n}, func(ctx stepper.Context) error {\n    fmt.Println(\"wake up the log-job\")\n\n    ctx.CreateSubtask(stepper.CreateTask{\n        Name: \"log-subtask\",\n        Data: []byte(\"Hello 1 subtask\"),\n    })\n\n    return nil\n}).OnFinish(func(ctx stepper.Context, data []byte) error {\n    fmt.Println(\"success job log-job\")\n\n    return nil\n})\n```\n\n## Middlewares\n\n### Retry\n\nThe retry middleware allows you to limit a number of retries.\n\n```go\nservice := stepper.NewService(db)\n\ns.UseMiddleware(middlewares.Retry(middlewares.RetryOptions{\n    Interval:   time.Second * 5,\n    MaxRetries: 3,\n}))\n```\n\n### Prometheus\n\n\n```go\nservice := stepper.NewService(db)\n\nprometheusMiddleware := middlewares.NewPrometheus()\n\ns.UseMiddleware(prometheusMiddleware.GetMiddleware())\n\ngo func() {\n    http.ListenAndServe(\":3999\", promhttp.HandlerFor(prometheusMiddleware.GetRegistry(), promhttp.HandlerOpts{}))\n}()\n\nif err := s.Listen(context.Background()); err != nil {\n    log.Fatal(err)\n}\n```\n\nThe prometheus middleware provides following metrics:\n\n```go\nprometheus.NewCounterVec(prometheus.CounterOpts{\n    Name: \"stepper_task_execution\",\n    Help: \"Count of all task executions\",\n}, []string{\"task\", \"status\"})\n\nprometheus.NewHistogramVec(prometheus.HistogramOpts{\n    Name:    \"stepper_task_duration_seconds\",\n    Help:    \"Duration of all executions\",\n    Buckets: []float64{.025, .05, .1, .25, .5, 1, 2.5, 5, 10, 20, 30},\n}, []string{\"task\", \"status\"})\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatroskin13%2Fstepper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatroskin13%2Fstepper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatroskin13%2Fstepper/lists"}