{"id":16633261,"url":"https://github.com/ktomk/pipelines","last_synced_at":"2025-04-12T19:48:22.308Z","repository":{"id":37587745,"uuid":"115522378","full_name":"ktomk/pipelines","owner":"ktomk","description":"Pipelines - Run Bitbucket Pipelines Wherever They Dock","archived":false,"fork":false,"pushed_at":"2024-11-06T19:26:14.000Z","size":4051,"stargazers_count":111,"open_issues_count":13,"forks_count":10,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-12T19:48:12.119Z","etag":null,"topics":["bitbucket-pipelines","docker","local-build","pipeline-runner","pipelines"],"latest_commit_sha":null,"homepage":"https://ktomk.github.io/pipelines/","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ktomk.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-12-27T13:11:31.000Z","updated_at":"2025-03-03T12:06:19.000Z","dependencies_parsed_at":"2024-10-25T18:30:48.325Z","dependency_job_id":"5594903a-4900-4e8f-901f-1f8d2a549e55","html_url":"https://github.com/ktomk/pipelines","commit_stats":{"total_commits":596,"total_committers":1,"mean_commits":596.0,"dds":0.0,"last_synced_commit":"3e67dce08f89e038cbef7088ec1002aab93faf6a"},"previous_names":[],"tags_count":73,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ktomk%2Fpipelines","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ktomk%2Fpipelines/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ktomk%2Fpipelines/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ktomk%2Fpipelines/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ktomk","download_url":"https://codeload.github.com/ktomk/pipelines/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248625498,"owners_count":21135513,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitbucket-pipelines","docker","local-build","pipeline-runner","pipelines"],"created_at":"2024-10-12T05:12:48.481Z","updated_at":"2025-04-12T19:48:22.281Z","avatar_url":"https://github.com/ktomk.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pipelines\n\n## Run Bitbucket Pipelines Wherever They Dock\n\n[![CI Status](https://github.com/ktomk/run-travis-yml/workflows/CI/badge.svg)](https://github.com/ktomk/pipelines/actions)\n[![Build Status](https://api.travis-ci.com/ktomk/pipelines.svg?branch=master)](https://app.travis-ci.com/ktomk/pipelines)\n[![Code Coverage](https://scrutinizer-ci.com/g/ktomk/pipelines/badges/coverage.png?b=master)](https://scrutinizer-ci.com/g/ktomk/pipelines/)\n[![Scrutinizer Code Quality](https://scrutinizer-ci.com/g/ktomk/pipelines/badges/quality-score.png?b=master)](https://scrutinizer-ci.com/g/ktomk/pipelines/?branch=master)\n\nCommand line pipeline runner written in PHP. Available from\nGithub or Packagist.\n\n[Usage](#usage) | [Environment](#environment) |\n[Exit Status](#exit-status) | [Details](#details) |\n[References](#references)\n\n## Usage\n\nFrom anywhere within a project or (Git) repository with a\nBitbucket Pipeline file:\n\n~~~\n$ pipelines\n~~~\n\nRuns pipeline commands from [`bitbucket-pipelines.yml`][BBPL]\n\\[BBPL].\n\nMemory and time limits are ignored. Press \u003ckbd\u003ectrl\u003c/kbd\u003e +\n\u003ckbd\u003ec\u003c/kbd\u003e to quit.\n\nThe Bitbucket limit of 100 (previously 10) steps per pipeline\nis ignored.\n\nExit status is from last pipeline script command, if a command\nfails the following script commands and steps are not executed.\n\nThe default pipeline is run, if there is no default pipeline in\nthe file, pipelines tells it and exists with non-zero status.\n\nTo execute a different pipeline use the `--pipeline \u003cid\u003e` option\nwhere `\u003cid\u003e` is one of the list by the `--list` option. Even more\ninformation about the pipelines is available via `--show`. Both\n`--list` and `--show` output and exit.\n\nUse `--steps \u003csteps\u003e` to specify which step(s) to execute (also in\nwhich order).\n\nIf the next pipeline step has a manual trigger, pipelines stops\nthe execution and outputs a short message on standard error\ngiving info about the fact. Manual triggers can be ignored with\nthe `--no-manual` option.\n\nRun the pipeline as if a tag/branch or bookmark has been pushed\nwith `--trigger \u003cref\u003e` where `\u003cref\u003e` is `tag:\u003cname\u003e`,\n`branch:\u003cname\u003e`, `bookmark:\u003cname\u003e` or\n`pr:\u003cbranch-name\u003e[:\u003cdestination-branch\u003e]`. If there is no tag,\nbranch, bookmark or pull-request pipeline with that name, the\nname is compared against the patterns of the referenced type and\nif found, that pipeline is run.\n\nOtherwise the default pipeline is run, if there is no default\npipeline, no pipeline at all is run and the command exits with\nnon-zero status.\n\n`--pipeline` and `--trigger` can be used together, `--pipeline`\noverrides pipeline from `--trigger` but `--trigger` still\ninfluences the container environment variables.\n\nTo specify a different file use the  `--basename \u003cbasename\u003e`\nor `--file \u003cpath\u003e` option and/or set the working directory\n`--working-dir \u003cpath\u003e` in which the file is looked for unless\nan absolute path is set by `--file \u003cpath\u003e`.\n\nBy default `pipelines` operates on the current working tree which\nis copied into the container to isolate running the pipeline from\nthe working directory (implicit `--deploy copy`).\n\nAlternatively the working directory can be mounted into the\npipelines container by using `--deploy mount`.\n\nUse `--keep` flag to keep containers after the pipeline has\nfinished for further inspection. By default all containers are\ndestroyed. Sometimes for development it is interesting to keep\ncontainers on error only, the `--error-keep` flag is for that.\n\nIn any case, if a pipeline runs again and it finds an existing\ncontainer with the same name (generated by the pipeline name\netc.), the existing container will be re-used. This can be very\nuseful to re-iterate quickly.\n\nManage leftover containers with `--docker-list` showing all\npipeline containers, `--docker-kill` to kill running containers\nand `--docker-clean` to remove stopped pipeline containers. Use\nin combination to fully clean, e.g.:\n\n    $ pipelines --docker-list --docker-kill --docker-clean\n\nOr just run for a more shy clean-up:\n\n    $ pipelines --docker-zap\n\nto kill and remove all pipeline containers (w/o showing a list)\nfirst. \"zap\" is pipelines \"make clean\" equivalent for `--keep`.\n\nAll containers run by `pipelines` are labeled to ease maintaining\nthem.\n\nValidate your `bitbucket-pipelines.yml` file with `--show` which\nhighlights errors found.\n\nFor schema-validation use `--validate [\u003cfile\u003e]`. Schema validation\nmight show errors that are not an issue when executing a pipeline\n(`--show` and/or `--dry-run` is better for that) but validates\nagainst a schema which is aligned with the one that Atlassian/\nBitbucket provides (the schema is more lax compared to upstream\nfor the cases known to offer a better practical experience). E.g.\nuse it for checks in your CI pipeline or linting files before push\nin a pre-commit hook or your local build.\n\nInspect your pipeline with `--dry-run` which will process the\npipeline but not execute anything. Combine with `-v` (, `--verbose`)\nto show the commands which would have run verbatim which allows\nto better understand how `pipelines` actually works. Nothing to\nhide here.\n\nUse `--no-run` to not run the pipeline at all, this can be used\nto test the utilities' options.\n\nPipeline environment variables can be passed/exported to or set\nfor your pipeline by name or file with `-e`, `--env` and\n`--env-file` options.\n\nEnvironment variables are also loaded from dot env files named\n`.env.dist` and `.env` and processed in that order before the\nenvironment options. Use of `--no-dot-env-files` prevents\nautomatic loading, `--no-dot-env-dot-dist` for the `.env.dist`\nfile only.\n\nMore information on pipelines environment variables in the\n[*environment* section](#environment) below.\n\n### Help\n\nA full display of the pipelines utility options and arguments is\navailable via `-h`, `--help`:\n\n\u003c!-- help --\u003e\n```\nusage: pipelines [\u003coptions\u003e] --version | -h | --help\n       pipelines [\u003coptions\u003e] [--working-dir \u003cpath\u003e] [--file \u003cpath\u003e]\n                 [--basename \u003cbasename\u003e] [--prefix \u003cprefix\u003e]\n                 [--verbatim] [--[no-|error-]keep] [--no-run]\n                 [(-e | --env) \u003cvariable\u003e] [--env-file \u003cpath\u003e]\n                 [--no-dot-env-files] [--no-dot-env-dot-dist]\n                 [--docker-client \u003cpackage\u003e] [--ssh]\n                 [--user[=\u003cname|uid\u003e[:\u003cgroup|gid\u003e]]]\n                 [--deploy mount | copy ] [--pipeline \u003cid\u003e]\n                 [(--step | --steps) \u003csteps\u003e] [--no-manual]\n                 [--trigger \u003cref\u003e] [--no-cache]\n       pipelines [\u003coptions\u003e] --service \u003cservice\u003e\n       pipelines [\u003coptions\u003e] --list | --show | --images\n                 | --show-pipelines | --show-services\n                 | --step-script[=(\u003cid\u003e | \u003cstep\u003e[:\u003cid\u003e])]\n                 | --validate[=\u003cpath\u003e]\n       pipelines [\u003coptions\u003e] --docker-client-pkgs\n       pipelines [\u003coptions\u003e] [--docker-list] [--docker-kill]\n                 [--docker-clean] [--docker-zap]\n\nGeneric options\n    -h, --help            show usage and help information\n    --version             show version information\n    -v, --verbose         be more verbose, show more information and\n                          commands to be executed\n    --dry-run             do not execute commands, e.g. invoke docker or\n                          run containers, with --verbose show the\n                          commands that would have run w/o --dry-run\n    -c \u003cname\u003e=\u003cvalue\u003e     pass a configuration parameter to the command\n\nPipeline runner options\n    --basename \u003cbasename\u003e set basename for pipelines file, defaults to\n                          'bitbucket-pipelines.yml'\n    --deploy mount|copy   how files from the working directory are\n                          placed into the pipeline container:\n                          copy     (default) working dir is copied into\n                                 the container. stronger isolation as\n                                 the pipeline scripts can change all\n                                 files without side-effects in the\n                                 working directory\n                          mount    the working directory is mounted.\n                                 fastest, no isolation\n    --file \u003cpath\u003e         path to the pipelines file, overrides looking\n                          up the \u003cbasename\u003e file from the current\n                          working directory, use '-' to read from stdin\n    --trigger \u003cref\u003e       build trigger; \u003cref\u003e can be either of:\n                          tag:\u003cname\u003e, branch:\u003cname\u003e, bookmark:\u003cname\u003e or\n                          pr:\u003cbranch-name\u003e[:\u003cdestination-branch\u003e]\n                          determines the pipeline to run\n    --pipeline \u003cid\u003e       run pipeline with \u003cid\u003e, use --list for a list\n                          of all pipeline ids available. overrides\n                          --trigger for the pipeline while keeping\n                          environment from --trigger.\n    --step, --steps \u003csteps\u003e\n                          execute not all but this/these \u003csteps\u003e. all\n                          duplicates and orderings allowed, \u003csteps\u003e are\n                          a comma/space separated list of step and step\n                          ranges, e.g. 1 2 3; 1-3; 1,2-3; 3-1 or -1,3-\n                          and 1,1,3,3,2,2\n    --no-manual           ignore manual steps, by default manual steps\n                          stop the pipeline execution when not the first\n                          step in invocation of a pipeline\n    --verbatim            only give verbatim output of the pipeline, do\n                          not display other information like which step\n                          currently executes, which image is in use ...\n    --working-dir \u003cpath\u003e  run as if pipelines was started in \u003cpath\u003e\n    --no-run              do not run the pipeline\n    --prefix \u003cprefix\u003e     use a different prefix for container names,\n                          default is 'pipelines'\n    --no-cache            disable step caches; docker always caches\n\nFile information options\n    --images              list all images in file, in order of use, w/o\n                          duplicate names and exit\n    --list                list pipeline \u003cid\u003es in file and exit\n    --show                show information about pipelines in file and\n                          exit\n    --show-pipelines      same as --show but with old --show output\n                          format without services and images / steps are\n                          summarized - one line for each pipeline\n    --show-services       show all defined services in use by pipeline\n                          steps and exit\n    --validate[=\u003cpath\u003e]   schema-validate file, shows errors if any,\n                          exits; can be used more than once, exit status\n                          is non-zero on error\n    --step-script[=(\u003cid\u003e | \u003cstep\u003e[:\u003cid\u003e])]\n                          write the step-script of pipeline \u003cid\u003e and\n                          \u003cstep\u003e to standard output and exit\n\nEnvironment control options\n    -e, --env \u003cvariable\u003e  pass or set an environment \u003cvariable\u003e for the\n                          docker container, just like a docker run,\n                          \u003cvariable\u003e can be the name of a variable which\n                          adds the variable to the container as export\n                          or a variable definition with the name of the\n                          variable, the equal sign \"=\" and the value,\n                          e.g. --env NAME=\u003cvalue\u003e\n    --env-file \u003cpath\u003e     pass variables from environment file to the\n                          docker container\n    --no-dot-env-files    do not pass .env.dist and .env files as\n                          environment files to docker\n    --no-dot-env-dot-dist dot not pass .env.dist as environment file to\n                          docker only\n\nKeep options\n    --keep                always keep docker containers\n    --error-keep          keep docker containers if a step failed;\n                          outputs non-zero exit status and the id of the\n                          container kept and exit w/ container exec exit\n                          status\n    --no-keep             do not keep docker containers; default\n\nContainer runner options\n    --ssh                 ssh agent forwarding: if $SSH_AUTH_SOCK is set\n                          and accessible, mount SSH authentication\n                          socket read only and set SSH_AUTH_SOCK in the\n                          pipeline step container to the mount point.\n    --user[=\u003cname|uid\u003e[:\u003cgroup|gid\u003e]]\n                          run pipeline step container as current or\n                          given \u003cuser\u003e/\u003cgroup\u003e; overrides container\n                          default \u003cuser\u003e - often root, (better) run\n                          rootless by default.\n\nService runner options\n    --service \u003cservice\u003e   runs \u003cservice\u003e attached to the current shell\n                          and waits until the service exits, exit status\n                          is the one of the docker run service\n                          container; for testing services, run in a\n                          shell of its own or background\n\nDocker service options\n    --docker-client \u003cpackage\u003e\n                          which docker client binary to use for the\n                          pipeline service 'docker' defaults to the\n                          'docker-19.03.1-linux-static-x86_64' package\n    --docker-client-pkgs  list all docker client packages that ship with\n                          pipelines and exit\n\nDocker container maintenance options\n      usage might leave containers on the system. either by interrupting\n      a running pipeline step or by keeping the running containers\n      (--keep, --error-keep)\n\n      pipelines uses a \u003cprefix\u003e 'pipelines' by default, followed by '-'\n      and a compound name based on step-number, step-name, pipeline id\n      and image name for container names. the prefix can be set by the\n      --prefix \u003cprefix\u003e option and argument.\n\n      three options are built-in to monitor and interact with leftovers,\n      if one or more of these are given, the following operations are\n      executed in the order from top to down:\n    --docker-list         list prefixed containers\n    --docker-kill         kills prefixed containers\n    --docker-clean        remove (non-running) containers with\n                          pipelines prefix\n\n      for ease of use:\n    --docker-zap          kill and remove all prefixed containers at\n                          once; no show/listing\n\nLess common options\n    --debug               flag for trouble-shooting (fatal) errors,\n                          warnings, notices and strict warnings; useful\n                          for trouble-shooting and bug-reports\n```\n\n### Usage Scenario\n\nGive your project and pipeline changes a quick test run from the\nstaging area. As pipelines are normally executed far away,\nsetting them up becomes cumbersome, the guide given in [Bitbucket\nPipelines documentation][BBPL-LOCAL-RUN] \\[BBPL-LOCAL-RUN] has some\nhints and is of help, but it is not about a bitbucket pipelines runner.\n\nThis is where the `pipelines` command jumps in.\n\nThe `pipelines` command closes the gap between local development\nand remote pipeline execution by executing any pipeline\nconfigured on your local development box. As long as Docker is\naccessible locally, the `bitbucket-pipelines.yml` file is parsed\nand it is taken care of to execute all steps and their commands\nwithin the container of choice.\n\nPipelines YAML file parsing, container creation and script\nexecution is done as closely as possible compared to the\nAtlassian Bitbucket Pipeline service. Environment variables can\nbe passed into each pipeline as needed. You can even switch to a\ndifferent CI/CD service like Github/Travis with little\nintegration work fostering your agility and vendor independence.\n\n### Features\n\nFeatures include:\n\n#### Dev Mode\n\nPipeline from your working tree like never before. Pretend to be\non any branch, tag or bookmark (`--trigger`) even in a different\nrepository or none at all.\n\nCheck if the reference matches a pipeline or just run the default\n(default) or a specific one (`--list`, `--pipeline`). Use a\ndifferent pipelines file (`--file`) or swap the \"repository\" by\nchanging the working directory (`--working-dir \u003cpath\u003e`).\n\nIf a pipeline step fails, the steps container can be kept for\nfurther inspection on error with the `--error-keep` option. The\ncontainer id is shown then which makes it easy to spawn a shell\ninside:\n\n```bash\n$ docker exec -it $ID /bin/sh\n```\n\nContainers can be always kept for debugging and manual testing\nof a pipeline with `--keep` and with the said `--error-keep` on\nerror only. Kept containers are re-used by their name regardless\nof any `--keep` (, `--error-keep`) option.\n\nContinue on a (failed) step with the `--steps \u003csteps\u003e` argument,\nthe `\u003csteps\u003e` option can be any step number or sequence (`1-3`),\nseparate multiple with comma (`3-,1-2`), you can even repeat steps\nor reverse order (`4,3,2,1`).\n\nFor example, if the second step failed, continue with use of\n`--steps 2-` to re-run the second and all following steps\n(`--steps 2` or `--step 2` will run only the next step; to do a\nstep-by-step approach).\n\nAfterwards manage left overs with `--docker-list|kill|clean` or\nclean up with `--docker-zap`.\n\nDebugging options to dream for; benefit from the local build, the\npipeline container.\n\n#### Container Isolation\n\nThere is one container per step, like it is on Bitbucket.\n\nFiles are isolated by being copied into the container before\nthe pipeline step script is executed (implicit `--deploy copy`).\n\nAlternatively files can be mounted into the container instead\nwith `--deploy mount` which normally is faster on Linux, but the\nworking tree might become changed by the container script which\ncauses side-effect that may be unwanted. Docker runs system-wide\nand containers do not isolate users (e.g. root is root).\n\nBetter with `--deploy mount` (and peace of mind) is using Docker\nin rootless mode where files manipulated in the pipeline container\nare accessible to the own user account (like root is your user\nautomatically mapped).\n\n* Further reading: [*How-To Rootless Pipelines*](doc/PIPELINES-HOWTO-ROOTLESS.md)\n\n#### Pipeline Integration\n\nExport files from the pipeline by making use of artifacts, these\nare copied back into the working tree while in (implicit)\n`--deploy copy` mode. Artifacts' files are always created by the\nuser running pipelines. This also (near) perfectly emulates the\nfile format `artifacts` section with the benefit/downside that\nyou might want to prepare a clean build in a pipeline step script\nwhile you can keep artifacts from pipelines locally. This is a\ntrade-off that has turned out to be acceptable over the years.\n\nwrap `pipelines` in a script for clean checkouts or wait for\nfuture options to stage first (*git-deployment* feature). In any\ncase, control your build first of all.\n\n#### Ready for Offline\n\nOn the plane? Riding Deutsche Bahn? Or just a rainy day on a\nremote location with broken net? Coding while abroad? Or just\nBitbucket down again?\n\nBefore going into offline mode, read about [*Working Offline*](doc/PIPELINES-OFFLINE.md)\nyou'll love it.\n\n#### Services? Check!\n\nThe local pipeline runner runs service containers on your local\nbox/system (that is your pipelines' host). This is similar to\n[use services and databases in Bitbucket Pipelines][BBPL-SRV]\n\\[BBPL-SRV].\n\nEven before any pipeline step makes use of a service,\na service definition can already be tested with the `--service`\noption turning setting up services in pipelines into a new\nexperience. A good way to test service definitions and to get\nan impression on additional resources being consumed.\n\n* Further reading: [*Working with Pipeline\n    Services*](doc/PIPELINES-SERVICES.md)\n\n#### Default Image\n\nThe pipelines command uses the default image like Bitbucket\nPipelines does (\"`atlassian/default-image`\"). Get started out\nof the box, but keep in mind it has roughly 1.4 GB.\n\n#### Pipelines inside Pipeline\n\nAs a special feature and by default pipelines mounts the docker\nsocket into each container (on systems where the socket is\navailable).\nThis allows to launch pipelines from a pipeline as long as\n`pipelines` and the Docker client is available in the\npipelines' container.  `pipelines` will take care of the Docker\nclient as `/usr/bin/docker` as long as the pipeline has the\n`docker` service (`services: [docker]`).\n\nThis feature is similar to [run Docker commands in Bitbucket\nPipelines][BBPL-DCK] \\[BBPL-DCK].\n\nThe pipelines inside pipeline feature serves `pipelines` itself\nwell for integration testing the projects build. In combination\nwith `--deploy mount`, the original working-directory is mounted\nfrom the host (again). Additional protection against endless\nloops by recursion is implemented to prevent accidental\npipelines inside pipeline invocations that would be endlessly\non-going.\n\n* Further reading: [*How-To Docker Client Binary Packages for\n    Pipelines*](doc/PIPELINES-HOWTO-DOCKER-CLIENT-BINARY.md)\n\n## Environment\n\nPipelines mimics \"all\" of the [Bitbucket Pipeline in-container\nenvironment variables][BBPL-ENV] \\[BBPL-ENV], also known as\nenvironment parameters:\n\n* `BITBUCKET_BOOKMARK` - conditionally set by `--trigger`\n* `BITBUCKET_BUILD_NUMBER` - always set to \"`0`\"\n* `BITBUCKET_BRANCH` - conditionally set by `--trigger`\n* `BITBUCKET_CLONE_DIR` - always set to deploy point in container\n* `BITBUCKET_COMMIT` - faux as no revision triggers a build;\n    always set to \"`0000000000000000000000000000000000000000`\"\n* `BITBUCKET_REPO_OWNER` - current username from environment or\n    if not available \"`nobody`\"\n* `BITBUCKET_REPO_SLUG` - base name of project directory\n* `BITBUCKET_TAG` - conditionally set by `--trigger`\n* `CI` - always set to \"`true`\"\n\nAll of these (but not `BITBUCKET_CLONE_DIR`) can be set within\nthe environment pipelines runs in and are taken over into container\nenvironment. Example:\n\n    $ BITBUCKET_BUILD_NUMBER=123 pipelines # build no. 123\n\nMore information on (Bitbucket) pipelines environment variables\ncan be found in the [*Pipelines Environment Variable Usage\nReference*](./doc/PIPELINES-VARIABLE-REFERENCE.md).\n\nAdditionally pipelines sets some environment variables for\nintrospection:\n\n* `PIPELINES_CONTAINER_NAME` - name of the container itself\n* `PIPELINES_ID` - `\u003cid\u003e` of the pipeline that currently runs\n* `PIPELINES_IDS` - list of space separated md5 hashes of so\n    far running `\u003cid\u003e`s. used to detect pipelines inside pipeline\n    recursion, preventing execution until system failure.\n* `PIPELINES_PARENT_CONTAINER_NAME` - name of the container name\n    if it was already set when the pipeline started (pipelines\n    inside pipeline \"pip\").\n* `PIPELINES_PIP_CONTAINER_NAME` - name of the first (initial)\n    pipeline container. Used by pipelines inside pipelines (\"pip\").\n* `PIPELINES_PROJECT_PATH` - path of the original project as if\n    it would be used for `--deploy`  with `copy` or `mount` so\n    that it is possible inside a pipeline to do `--deploy mount`\n    when the current container did not mount. A mount always\n    requires the path of the project directory on the system\n    running pipelines. With no existing mount (e.g. `--deploy\n    copy`) it would otherwise be unknown. Manipulating this\n    parameter within a pipeline leads to undefined behaviour and\n    can have system security implications.\n\nThese environment variables are managed by pipelines itself.\nSome of them can be injected which can lead to undefined\nbehaviour and can have system security implications as well.\n\nNext to these special purpose environment variables, any other\nenvironment variable can be imported into or set in the container\nvia the `-e`, `--env` and `--env-file` options. These behave\nexactly as documented for the [`docker run` command][DCK-RN]\n\\[DCK-RN].\n\nInstead of specifying custom environment parameters for each\ninvocation, pipelines by default automatically uses the\n`.env.dist` and `.env` files from each project supporting the\nsame file-format for environment variables as docker.\n\n## Exit Status\n\nExit status on success is 0 (zero).\n\nA non zero exit status denotes an error:\n\n- 1  : An argument supplied (also a missing one) caused the\n       error.\n- 2  : An error is caused by the system not being able to\n       fulfill the command (e.g. a file could not be read).\n- 127: Running pipelines inside pipelines failed detecting an\n       endless loop.\n\n### Example\n\nNot finding a file might cause exit status 2 (two) on error\nbecause a file is not found, however with a switch like `--show`\nthe exit status might still be 1 (one) as there was an error\nshowing that the file does not exists (indirectly) and the error\nis more prominently showing all pipelines of that file.\n\n## Details\n\n[Requirements](#requirements) | [User Tests](#user-tests) |\n[Installation](#installation) | [Known Bugs](#known-bugs) |\n[Todo](#todo)\n\n### Requirements\n\nPipelines works best on a POSIX compatible system having a PHP\nruntime.\n\nDocker needs to be available locally as `docker` command as it is\nused to run the pipelines. Rootless Docker is supported.\n\nA recent PHP version is favored, the `pipelines` command needs\nPHP to run. It should work with PHP 5.3.3+. A development\nenvironment should be PHP 7+, this is especially suggested for\nfuture releases. PHP 8+ is supported as well.\n\nInstalling the [PHP YAML extension][PHP-YAML] \\[PHP-YAML] is\nhighly recommended as it does greatly improve parsing the\npipelines file which is otherwise with a YAML parser on it's\nown as a fall-back and is not bad at all. There are subtle\ndifferences between these parsers, so why not have both at\nhand?\n\n### User Tests\n\nSuccessful use on Ubuntu (16.04 LTS, 18.04 LTS, 20.04 LTS,\n22.04 LTS) and Mac OS X (Sierra and High Sierra) with PHP\nand Docker installed (incl. Rootless).\n\n### Installation\n\n[Phar (Download)](#download-the-phar-php-archive-file) |\n[Composer](#install-with-composer) |\n[Phive](#install-with-phive) |\n[Source (also w/ Phar)](#install-from-source) |\n[Project (Development)](#install-the-project-for-development)\n\nInstallation is available by downloading the phar archive from\nGithub, via Composer/Packagist or with Phive and it should always\nwork from source which includes building the phar file.\n\n#### Download the PHAR (PHP Archive) File\n\nDownloads are available on Github. To obtain the latest released\nversion, use the following URL:\n\n    https://github.com/ktomk/pipelines/releases/latest/download/pipelines.phar\n\nRename the phar file to just \"`pipelines`\", set the executable\nbit and move it into a directory where executables are found.\n\nDownloads from Github are available since version 0.0.4. All\nreleases are listed on the following website:\n\n    https://github.com/ktomk/pipelines/releases\n\n#### Install with Composer\n\nSuggested is to install it globally (and to have the global\ncomposer vendor/bin in $PATH) so that it can be called with ease\nand there are no dependencies in a local project:\n\n    $ composer global require ktomk/pipelines\n\nThis will automatically install the latest available version.\nVerify the installation by invoking pipelines and output the\nversion:\n\n    $ pipelines --version\n    pipelines version 0.0.19\n\nTo uninstall remove the package:\n\n    $ composer global remove ktomk/pipelines\n\nTake a look at [Composer from *`getcomposer.org`*][COMPOSER]\n\\[COMPOSER], a *Dependency Manager for PHP*. Pipelines has\nsupport for composer based installations, which might include\nupstream patches (composer 2 is supported, incl. upstream\npatches).\n\n#### Install with Phive\n\nPerhaps the most easy way to install when *phive* is available:\n\n    $ phive install pipelines\n\nEven if your PHP version does not have the Yaml extension this\nshould work out of the box. If you use *composer* and you're a\nPHP aficionado, dig into *phive* for your systems and workflow.\n\nTake a look at [Phive from *`phar.io`*][PHARIO] \\[PHARIO], the *PHAR\nInstallation and Verification Environment (PHIVE)*. Pipelines has\nfull support for phar.io/phar based installations which includes\nsupport for the *phive* utility including upstream patches.\n\n#### Install from Source\n\nTo install from source, checkout the source repository and\nsymlink the executable file `bin/pipelines` into a segment of\n$PATH, e.g. your $HOME/bin directory or similar. Verify the\ninstallation by invoking pipelines and output the version:\n\n    $ pipelines --version\n    pipelines version 0.0.19 # NOTE: the version is exemplary\n\nTo create a phar archive from sources, invoke from within the\nprojects root directory the build script:\n\n    $ composer build\n    building 0.0.19-1-gbba5a43 ...\n    pipelines version 0.0.19-1-gbba5a43\n    file.....: build/pipelines.phar\n    size.....: 240 191 bytes\n    SHA-1....: 9F118A276FC755C21EA548A77A9DBAF769B93524\n    SHA-256..: 0C38CBBB12E10E80F37ECA5C4C335BF87111AC8E8D0490D38683BB3DA7E82DEF\n    file.....: 1.1.0\n    api......: 1.1.1\n    extension: 2.0.2\n    php......: 7.2.16-[...]\n    uname....: [...]\n    count....: 62 file(s)\n    signature: SHA-1 E638E7B56FAAD7171AE9838DF6074714630BD486\n\nThe phar archive then is (as written in the output of the build):\n\n    build/pipelines.phar\n\nCheck the version by invoking it:\n\n    $ build/pipelines.phar --version\n    pipelines version 0.0.19-1-gbba5a43\n    # NOTE: the version is exemplary\n\n##### Php Compatibility and Undefined Behaviour\n\nThe pipelines project aims to support php 5.3.3 up to php 8.1.\n\nUsing any of its PHP functions or methods with named parameters\nfalls into undefined behaviour.\n\n##### Reproducible Phar Builds\n\nThe pipelines project practices reproducible builds since it's first\nphar build. The build is self-contained, which means that the repository\nships with all required files to build with only little dependencies:\n\n- PHP (for [`build.php`](lib/build/build.php))\n- Composer\n- Git\n\nReproducible builds of the phar file would be incomplete without the\nfine work from the composer projects `phar-utils` (Seldaek/Jordi\nBoggiano) which is forked by the pipelines project in [`Timestamps.php`\n](src/PharBuild/Timestamps.php) by keeping the original license with the\nfile (MIT), providing bug-fixes to upstream under that license (see\n[Phar-Utils #2](https://github.com/Seldaek/phar-utils/pull/2) and\n[Phar-Utils #3](https://github.com/Seldaek/phar-utils/pull/3)).\n\nThis file is used to set the timestamps inside the phar file to that\nof the release as otherwise those would be at the time of build. This\nis the same as the Composer project does (see\n[Composer #3927](https://github.com/composer/composer/issues/3927)).\n\nAdditionally in the pipelines project that file is used to change the\naccess permissions of the files in the phar. That is because across PHP\nversions the behaviour has changed so the build is kept backwards and\nforwards compatible. As this has been noticed later in the projects'\nhistory, the build might show different binaries depending on which PHP\nversion is used (see [PHP #77022](https://bugs.php.net/bug.php?id=77022)\nand [PHP #79082](https://bugs.php.net/bug.php?id=79082)) and the patch\nstate of the timestamps file.\n\n#### Install the Project for Development\n\nWhen working with `git`, clone the repository and then invoke\n`composer install`. The project is setup for development then.\n\nAlternatively it's possible to do the same via composer directly:\n\n~~~\n$ composer create-project --prefer-source --keep-vcs ktomk/pipelines\n...\n$ cd pipelines\n~~~\n\nVerify the installation by invoking the local build:\n\n~~~\n$ composer ci\n~~~\n\nShould exit with status `0` when it went fine, non `0` when there is an\nissue. Composer tells which individual script did fail.\n\nFollow the instructions in [*Install from Source*](#install-from-source)\nto use the development version for `pipelines`.\n\n### Known Bugs\n\n- The command \"`:`\" in pipelines exec layer is never really\n  executed but emulated having exit status 0 and no standard or\n  error output. It is intended for pipelines testing.\n\n- Brace expansion (used for glob patterns with braces) is known\n  to fail in some cases. This *could* affect matching pipelines,\n  collecting asset paths and *did* affect building the phar file.\n\n  For the first two, this has *never* been reported nor experienced,\n  for building the phar file the workaround was to entail the\n  larger parts of the pattern.\n\n- The sf2yaml based parser does not support the backslash at the end\n  of a line to fold without a space with _double quoted strings_.\n\n- The libyaml based parser does not support dots (\"`.`\") in anchor\n  names.\n\n- The libyaml based parser does not support _folded scalar_ (\"`\u003e`\") as\n  _block style indicator_. Suggested workaround is to use _literal\n  style_ (\"`|`\").\n\n- NUL bytes (\"`\\0`\") are not supported verbatim in step-scripts due\n  to defense-in-depth protection on `passthru` in the PHP-runtime to\n  prevent *Null character* injection.\n\n- When the project directory is large (e.g. a couple of GBs) and copying\n  it into the pipeline container, it may appear as if pipelines hangs as\n  the copying operation is ongoing and taking a long time.\n\n  Pressing \u003ckbd\u003ectrl\u003c/kbd\u003e + \u003ckbd\u003ec\u003c/kbd\u003e may stop pipelines but not\n  the copying operation. Kill the process of the copy operation (`tar`\n  pipe to `docker cp`) to stop the operation.\n\n### Todo\n\n- [x] Support for private Docker repositories\n- [x] Inject docker client if docker service is enabled\n- [x] Run specific steps of a pipeline (only) to put the user\n      back into command on errors w/o re-running everything\n- [x] Stop at manual steps (`--no-manual` to override)\n- [x] Support BITBUCKET_PR_DESTINATION_BRANCH with\n      `--trigger pr:\u003csource\u003e:\u003cdestination\u003e`\n- [x] Pipeline services\n- [x] Run as current user with `--user` (`--deploy mount` should not\n      enforce the container default user \\[often \"root\"] for project\n      file operations any longer), however the Docker utility still\n      requires you (the current user) to be root like, so technically\n      there is little win (see [Rootless\n      Pipelines](doc/PIPELINES-HOWTO-ROOTLESS.md) for what works better\n      in this regard)\n- [x] Have caches on a per-project basis\n- [x] Copy local composer cache into container for better\n      (offline) usage in PHP projects (see\n      [Populate Caches](doc/PIPELINES-CACHES.md#populate-caches))\n- [x] Run scripts with `/bin/bash` if available ([#17]) (*bash-runner*\n      feature)\n- [ ] Support for `BITBUCKET_DOCKER_HOST_INTERNAL` environment variable\n      / host.docker.internal hostname within pipelines\n- [ ] Count `BITBUCKET_BUILD_NUMBER` on a per project basis (*build-number*\n      feature)\n- [ ] Option to *not* mount docker.sock\n- [ ] Limit projects' paths below `$HOME`, excluding dot `.` directory\n      children.\n- [ ] More accessible offline preparation (e.g.\n      `--docker-pull-images`, `--go-offline` or similar)\n- [ ] Check Docker existence before running a pipeline\n- [ ] Pipes support (*pipe* feature)\n    - [X] Show scripts with pipe/s\n    - [X] Fake run script with pipe/s showing information\n    - [ ] Create test/demo pipe\n    - [ ] Run script with pipe/s\n- [ ] Write about differences from Bitbucket Pipelines\n- [ ] Write about the file format support/limitations\n- [ ] Pipeline file properties support:\n    - [X] step.after-script (*after-script* feature)\n    - [X] step.trigger (`--steps` / `--no-manual` options)\n    - [X] step.caches (to disable use `--no-cache` option)\n    - [X] definitions\n        - [X] services (*services* feature)\n        - [X] caches (*caches* feature)\n    - [ ] step.condition ([#13])\n    - [ ] clone (*git-deployment* feature)\n    - [ ] max-time (never needed this for local run)\n    - [ ] size (likely neglected for local run, limited support for\n      [Rootless Pipelines](doc/PIPELINES-HOWTO-ROOTLESS.md))\n- [ ] Get VCS revision from working directory (*git-deployment* feature)\n- [ ] Use a different project directory `--project-dir \u003cpath\u003e` to\n  specify the root path to deploy into the container, which\n  currently is the working directory (`--working-dir \u003cpath\u003e` works\n  already)\n- [ ] Run on a specific revision, reference it (`--revision \u003cref\u003e`);\n  needs a clean VCS checkout into a temporary folder which then\n  should be copied into the container (*git-deployment* feature)\n- [ ] Override the default image name (`--default-image \u003cname\u003e`; never\n  needed this for local run)\n\n[#17]: https://github.com/ktomk/pipelines/issues/17\n[#13]: https://github.com/ktomk/pipelines/issues/13\n\n## References\n\n* \\[BBPL]: https://confluence.atlassian.com/bitbucket/configure-bitbucket-pipelines-yml-792298910.html\n* \\[BBPL-ENV]: https://confluence.atlassian.com/bitbucket/environment-variables-794502608.html\n* \\[BBPL-LOCAL-RUN]: https://confluence.atlassian.com/bitbucket/debug-your-pipelines-locally-with-docker-838273569.html\n* \\[BBPL-DCK]: https://confluence.atlassian.com/bitbucket/run-docker-commands-in-bitbucket-pipelines-879254331.html\n* \\[BBPL-SRV]: https://confluence.atlassian.com/bitbucket/use-services-and-databases-in-bitbucket-pipelines-874786688.html\n* \\[COMPOSER]: https://getcomposer.org/\n* \\[DCK-RN]: https://docs.docker.com/engine/reference/commandline/run/\n* \\[PHARIO]: https://phar.io/\n* \\[PHP-YAML]: https://pecl.php.net/package/yaml\n\n[BBPL]: https://confluence.atlassian.com/bitbucket/configure-bitbucket-pipelines-yml-792298910.html\n[BBPL-ENV]: https://confluence.atlassian.com/bitbucket/environment-variables-794502608.html\n[BBPL-LOCAL-RUN]: https://confluence.atlassian.com/bitbucket/debug-your-pipelines-locally-with-docker-838273569.html\n[BBPL-DCK]: https://confluence.atlassian.com/bitbucket/run-docker-commands-in-bitbucket-pipelines-879254331.html\n[BBPL-SRV]: https://confluence.atlassian.com/bitbucket/use-services-and-databases-in-bitbucket-pipelines-874786688.html\n[COMPOSER]: https://getcomposer.org/\n[DCK-RN]: https://docs.docker.com/engine/reference/commandline/run/\n[PHARIO]: https://phar.io/\n[PHP-YAML]: https://pecl.php.net/package/yaml\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fktomk%2Fpipelines","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fktomk%2Fpipelines","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fktomk%2Fpipelines/lists"}