{"id":13700945,"url":"https://github.com/myntra/pipeline","last_synced_at":"2025-04-04T06:09:26.742Z","repository":{"id":57488697,"uuid":"74017132","full_name":"myntra/pipeline","owner":"myntra","description":"Pipeline is a package to build multi-staged concurrent workflows with a centralized logging output. ","archived":false,"fork":false,"pushed_at":"2024-03-14T16:49:19.000Z","size":924,"stargazers_count":479,"open_issues_count":7,"forks_count":54,"subscribers_count":21,"default_branch":"master","last_synced_at":"2025-03-28T05:11:48.685Z","etag":null,"topics":["ci","ci-cd","concurrent","jenkins","jenkins-pipeline","pipe","pipeline","workflow"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/myntra.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-11-17T10:39:39.000Z","updated_at":"2025-02-20T12:46:24.000Z","dependencies_parsed_at":"2024-06-18T13:48:10.603Z","dependency_job_id":"31a52438-4f6d-4463-87be-c4f4c1195996","html_url":"https://github.com/myntra/pipeline","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myntra%2Fpipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myntra%2Fpipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myntra%2Fpipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myntra%2Fpipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/myntra","download_url":"https://codeload.github.com/myntra/pipeline/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247128752,"owners_count":20888235,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ci","ci-cd","concurrent","jenkins","jenkins-pipeline","pipe","pipeline","workflow"],"created_at":"2024-08-02T20:01:10.218Z","updated_at":"2025-04-04T06:09:26.716Z","avatar_url":"https://github.com/myntra.png","language":"Go","funding_links":[],"categories":["Go","workflow","Building","Libraries"],"sub_categories":["Workflows"],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://cdn.rawgit.com/myntra/pipeline/master/images/pipes.png\" height=\"150\" width=\"150\" /\u003e\n  \u003ch3 align=\"center\"\u003ePipeline\u003c/h3\u003e\n  \u003cp align=\"center\"\u003eA package to build multi-staged concurrent workflows with a centralized logging output. \u003c/p\u003e\n  \u003cp align=\"center\"\u003e\n    \u003ca href=\"https://travis-ci.org/myntra/pipeline\"\u003e\u003cimg src=\"https://travis-ci.org/myntra/pipeline.svg?branch=master\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://godoc.org/gopkg.in/myntra/pipeline.v1\"\u003e\u003cimg src=\"https://godoc.org/gopkg.in/myntra/pipeline.v1?status.svg\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://goreportcard.com/report/gopkg.in/myntra/pipeline.v1\"\u003e\u003cimg src=\"https://goreportcard.com/badge/gopkg.in/myntra/pipeline.v1\"\u003e\u003c/a\u003e\n  \u003c/p\u003e\n\u003c/p\u003e\n\n---\n\nThe package could be used to define and execute CI/CD tasks(either sequential or concurrent). A tool with similar goals\nwould be [Jenkins Pipeline](https://jenkins.io/doc/book/pipeline/overview/). However, compared to Jenkins Pipeline, this package has\nfewer constructs since the logic is specified in code, as opposed to a [Jenkinsfile](https://jenkins.io/doc/book/pipeline/jenkinsfile/).\n\nIt's tiny by design and is valuable when used as a glue rather than a container.\n\n#### go get\n```\n$ go get gopkg.in/myntra/pipeline.v1\n```\n\n#### Concepts\n\n The package has three building blocks to create workflows : Pipeline, Stage and Step . A pipeline is a collection of stages and a stage is a\n collection of steps. A stage can have either concurrent or sequential steps, while stages are always sequential.\n\n![Pipeline](https://cdn.rawgit.com/myntra/pipeline/master/images/pipeline.png)\n\nThe step block is where the actual work is done. Stage and pipeline act as flow governors.\n\n#### The Step Interface\n\nStep is the unit of work which can be concurrently or sequentially staged with other steps. To do that, we need to implement the\n`Step` interface.\n\n```go\ntype Step interface {\n\tOut\n\tExec(*Request) *Result\n\tCancel() error\n}\n```\n\nTo satisfy the interface we need to embed `pipeline.StepContext` and implement `Exec(*Request)*Result`, `Cancel()error` methods in the\ntarget type. For e.g:\n\n```go\ntype work struct {\n\tpipeline.StepContext\n}\n\nfunc (w work) Exec(request *pipeline.Request) *pipeline.Result {\n\treturn \u0026pipeline.Result{}\n}\n\nfunc (w work) Cancel() error {\n\treturn nil\n}\n```\n\nThe `pipeline.StepContext` type provides a `Status` method which can be used to log to the `out` channel. The current step receives a\n`Request` value passed on by the previous step. Internally data(`Request.Data` and `Request.KeyVal`) is copied from the previous step's\n`Result`.\n\n#### Usage\n\nThe api [NewStage(name string, concurrent bool, disableStrictMode bool)](https://godoc.org/github.com/myntra/pipeline#NewStage) is used to stage work either sequentially or concurrently. In terms of the pipeline package, a unit of work is an interface: [Step](https://godoc.org/github.com/myntra/pipeline#Step). \n\nThe following example shows a sequential stage. For a more complex example, please see: [examples/advanced.go](https://github.com/myntra/pipeline/blob/master/examples/advanced.go)\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"time\"\n\n\t\"github.com/myntra/pipeline\"\n)\n\ntype work struct {\n\tpipeline.StepContext\n\tid int\n}\n\nfunc (w work) Exec(request *pipeline.Request) *pipeline.Result {\n\tw.Status(fmt.Sprintf(\"%+v\", request))\n\n\tduration := time.Duration(1000 * w.id)\n\ttime.Sleep(time.Millisecond * duration)\n\tmsg := fmt.Sprintf(\"work %d\", w.id)\n\n\treturn \u0026pipeline.Result{\n\t\tError:  nil,\n\t\tData:   struct{msg string}{msg:msg},\n\t\tKeyVal: map[string]interface{}{\"msg\": msg},\n\t}\n}\n\nfunc (w work) Cancel() error {\n\tw.Status(\"cancel step\")\n\treturn nil\n}\n\nfunc readPipeline(pipe *pipeline.Pipeline) {\n\tout, err := pipe.Out()\n\tif err != nil {\n\t\treturn\n\t}\n\n\tprogress, err := pipe.GetProgressPercent()\n\tif err != nil {\n\t\treturn\n\t}\n\n\tfor {\n\t\tselect {\n\t\tcase line := \u003c-out:\n\t\t\tfmt.Println(line)\n\t\tcase p := \u003c-progress:\n\t\t\tfmt.Println(\"percent done: \", p)\n\t\t}\n\t}\n}\n\nfunc main() {\n\t// create a new pipeline\n\tworkpipe := pipeline.NewProgress(\"myProgressworkpipe\", 1000, time.Second*3)\n\t// func NewStage(name string, concurrent bool, disableStrictMode bool) *Stage\n\t// To execute steps concurrently, set concurrent=true.\n\tstage := pipeline.NewStage(\"mypworkstage\", false, false)\n\n\t// a unit of work\n\tstep1 := \u0026work{id: 1}\n\t// another unit of work\n\tstep2 := \u0026work{id: 2}\n\n\t// add the steps to the stage. Since concurrent is set false above. The steps will be\n\t// executed one after the other.\n\tstage.AddStep(step1)\n\tstage.AddStep(step2)\n\n\t// add the stage to the pipe.\n\tworkpipe.AddStage(stage)\n\n\tgo readPipeline(workpipe)\n\n\tresult := workpipe.Run()\n\tif result.Error != nil {\n\t\tfmt.Println(result.Error)\n\t}\n\n\tfmt.Println(\"timeTaken:\", workpipe.GetDuration())\n}\n\n```\n\nCheck `examples` directory for more.\n\n\n\n#### Logging and Progress\n\n- `pipeline.Out()` : Get all statuses/logs.\n- `pipeline.Progress` : Get progress in percentage.\n\nOutput of the above example:\n\n![Example Output](https://cdn.rawgit.com/myntra/pipeline/master/images/simple_pipe_out.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmyntra%2Fpipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmyntra%2Fpipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmyntra%2Fpipeline/lists"}