{"id":13722714,"url":"https://github.com/rosineygp/lines","last_synced_at":"2025-08-13T06:18:35.367Z","repository":{"id":54984992,"uuid":"284683404","full_name":"rosineygp/lines","owner":"rosineygp","description":"A pure bash clojureish CI pipeline","archived":false,"fork":false,"pushed_at":"2021-01-17T20:36:02.000Z","size":662,"stargazers_count":71,"open_issues_count":0,"forks_count":0,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-09T17:01:20.776Z","etag":null,"topics":["bash","ci","clojure","docker","edn","fleck","flk","pipeline"],"latest_commit_sha":null,"homepage":"https://rosineygp.github.io/lines/","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rosineygp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-03T11:35:23.000Z","updated_at":"2024-04-03T00:00:20.000Z","dependencies_parsed_at":"2022-08-14T08:10:40.632Z","dependency_job_id":null,"html_url":"https://github.com/rosineygp/lines","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rosineygp%2Flines","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rosineygp%2Flines/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rosineygp%2Flines/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rosineygp%2Flines/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rosineygp","download_url":"https://codeload.github.com/rosineygp/lines/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248225653,"owners_count":21068078,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bash","ci","clojure","docker","edn","fleck","flk","pipeline"],"created_at":"2024-08-03T01:01:32.034Z","updated_at":"2025-04-10T13:14:13.321Z","avatar_url":"https://github.com/rosineygp.png","language":"Clojure","funding_links":[],"categories":["Clojure"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca alt=\"lines\" href=\"https://rosineygp.github.io/lines\"\u003e\n    \u003cimg src=\"docs/assets/logo.png\"/\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n# Lines\n\nA pure bash clojureish CI pipeline.\n\n# Main Features\n\n* Complete CI engine\n* Execute local or remote\n* Pure data syntax `edn`\n* Clojure script syntax\n* Easy command line integration like (docker, kubectl, apt)\n* Concurrency execution\n* Modular and extensible\n\n# Why?\n\n* Alternative method to install clojure, using \"clojure\", but without clojure\n* Tired to write `yaml` every day\n* Create something cool with [Fleck](https://github.com/chr15m/flk)\n\nTable of contents\n-----------------\n\n* [Installation](#installation)\n* [Job keywords](#job-keywords)\n* [Modules](#modules)\n  * [shell](#shell)\n  * [docker](#docker)\n  * [scp](#scp)\n  * [template](#template)\n  * [user module](#user-module)\n* [EDN Pipeline](#edn-pipeline)\n  * [Targets file](#targets-file)\n* [Clojure Pipeline](#clojure-pipeline)\n  * [Lines functions](#lines-functions)\n* [Environment vars](#environment-vars)\n* [Extensions](#extensions)\n* [Development](#development)\n\n## Installation\n\n### requirements\n\n* bash 4\n* coreutils\n* git\n* docker ¹\n* ssh ²\n* scp ²\n\n\u003e ¹ docker module\u003cbr\u003e² remote execution\n\n### install script\n\n```bash\n# download\n\ncurl -L https://github.com/rosineygp/lines-sh/releases/latest/download/lines \u003e lines\nchmod +x lines\nsudo mv lines /usr/bin/lines\n```\n\n### minimal usage\n\nCreate a file `.lines.edn` with followin data.\n\n```edn\n[{:apply [\"echo hello world!\"]}]\n```\n\nJust execute command `lines`\n\n```shell\nname: lines *******************************************************************\ntarget: local\nstart: ter 22 dez 2020 21:38:01 -03\n  cmd: echo hello world!\n    hello world!\n  exit-code: 0\nstatus: true\nfinished: ter 22 dez 2020 21:38:04 -03 ****************************************\n```\n\n## Job keywords\n\nA job is defined as a hashmap of keywords. The commons keywords available for job are:\n\n| keyword                       | type    | description                                                |\n|-------------------------------|---------|------------------------------------------------------------|\n| [apply](#apply)               | array   | the tasks that job will handler                            |\n| [name](#name)                 | string  | optional job name                                          |\n| [module](#module)             | string  | module name                                                |\n| [target](#target)             | hashmap | where the job will run                                     |\n| [vars](#vars)                 | hashmap | set env vars (branch name changes conforme current branch) |\n| [args](#args)                 | hashmap | arguments modules, each module has own args                |\n| [ignore-error](#ignore-error) | boolean | when job fail the execution continue                       |\n| [retries](#retries)           | integer | number of retries if job fail, max 2                       |\n\n### apply\n\nIs the only required keyword that job needs. It's an array of objects that will be executed by a module. Each element of array is a tasks and the job can handler N tasks. If any tasks has a exit code different than 0, the job will stop and throw the error. **ignore-error** and **retries** helps to handler errors.\n\n```edn\n{:apply [\"uname -a\"\n         \"make build\"]}\n```\n\n\u003e using default module **shell**, apply is a array of strings.\n\n```edn\n{:module \"scp\"\n :apply [{:src \"program.tar.gz\" :dest \"/tmp/program.tar.gz\"}\n         {:src \"script.sh\"      :dest \"/tmp/script.sh\"}]}\n```\n\n\u003e **scp** uses a list of hashmaps\n\n### name\n\n**name** is just a label for the job.\n\n```edn\n{:name \"install curl\"\n :module \"shell\"\n :apply [\"apk add curl\"]}\n```\n\n### module\n\n**module** is the method executed by job (default is **shell**)\n\n```edn\n{:module \"shell\"\n :apply [\"whoami\"]}\n```\n\nDocker module\n\n```edn\n{:module \"docker\"\n :args {:image \"node\"}\n :apply [\"npm test\"]}\n```\n\nthe builtin modules are:\n\n| module   | description                                                  |\n|----------|--------------------------------------------------------------|\n| shell    | execute shell commands                                       |\n| docker   | start a docker instance and execute shell commands inside it |\n| template | render **lines template** and copy to it to destiny          |\n| scp      | copy files over scp                                          |\n\n\u003e is possible create custom modules\n\n### target\n\nHost **target** is the location where the job will run. If any target passed the job will run at localhost.\n\n```edn\n{:target {:label \"web-server\"\n          :host \"web.local.net\"\n          :port 22\n          :method \"ssh\"}}\n```\n\u003e Targets can be defined in separated file, during execution is possible to merge data with job and execute the same job in n hosts.\n\n| keywords | description                       |\n|----------|-----------------------------------|\n| label    | host label, just for identify     |\n| host     | ip or fqdn for access host        |\n| user     | login user                        |\n| port     | method port 22 is default         |\n| method   | connection method, ssh is default |\n\n\u003e Is possible set another keywords for filter like **group**, **dc** or any other value you need to organizer targets.\n\nAfter job executed it return themself with result values\n\n### vars\n\nVariables will be inject in environment during tasks execution.\n\n```edn\n{:vars {MY_VAR_0 \"lines\"\n        MY_VAR_1 \"go\"}\n :apply [\"echo $MY_VAR_0\"\n         \"echo $MY_VAR_1\"]}\n```\n\n\u003e **BRANCH_NAME** and **BRANCH_NAME_SLUG** are inject in environment.\n\n### args\n\nArgs is the parameters of modules.\n\n```edn\n{:args {:sudo true}\n :apply [\"apt-get update\"\n         \"apt-get install htop -y]}\n```\n\n\u003e All tasks will run with **sudo**\n\n### ignore-error\n\nIf some task fail, lines will not stop the pipeline, just return the current task failed.\n\n```edn\n{:ignore-error true\n :apply [\"whoami\"\n         \"exit 1\"\n         \"dpkg -l\"]}\n```\n\n\u003e The tasks after error will not be executed.\n\n### retries\n\nIf some task fail, retry will run it again.\n\n```edn\n{:retries 2\n :apply [\"ping -c 1 my-host\"]}\n```\n\n\u003e The max retries are **2**, but it can be increase setting **LINES_JOB_MAX_ATTEMPTS** at environment vars.\n\n## Modules\n\n### shell\n\nIs it the default module, just spawn scripts to shell.\n\n```edn\n{:module \"shell\"\n :apply [\"date\"]}\n```\n\n\u003e Command will be executed inside a `heredoc` block\n\n```shell\n\u003ccommand_header\u003e bash -s \u003c\u003c-'LINES-BLOCK-EOF'\nexport \u003cenv vars\u003e;\ncommands\nLINES-BLOCK-EOF\n```\n\n#### arguments\n\n| keyword    | type    | description                                               |\n|------------|---------|-----------------------------------------------------------|\n| sudo¹      | boolean | apply commands using sudo                                 |\n| user¹      | string  | change current user                                       |\n| entrypoint | array   | change initial entry command (default is `[\"bash\" \"-s\"]`) |\n\n\u003e ¹ needs pre configured sudoers (without password)\n\n### docker\n\nCreate a docker instance and execute commands inside it.\n\n#### single instance\n\n```edn\n{:module \"docker\"\n :apply [\"whoami\"]}\n```\n\n* start docker instance with default image (alpine)\n* run command **whoami** inside container\n\n#### services\n\n```edn\n{:module \"docker\"\n :args {:image \"ubuntu\"\n        :services [{:image \"nginx\"\n                    :alias \"nginx\"}]}\n :apply [\"apt-get update\"\n         \"apt-get install curl -y\"\n         \"curl http://nginx\"]}\n```\n\n* start docker instance with nginx image as a **service** and set network alias as **nginx**\n* start another docker instance with ubuntu image\n* install ubuntu packages\n* execute curl at service from ubuntu instance\n\n### download artifacts\n\nDownload files or directory from a docker instance.\n\n```edn\n{:module \"docker\"\n :args {:artifacts {:paths [\"file\"\n                            \"directory\"]}}\n :apply [\"touch file\"\n         \"mkdir directory\"\n         \"touch directory/file\"]}\n```\n\n#### arguments\n\n| keyword                             | type    | description                                            |\n|-------------------------------------|---------|--------------------------------------------------------|\n| image                               | string  | docker instance path name                              |\n| entrypoint                          | array   | change initial entry command (default is sh)           |\n| privileged                          | boolean | run job with privileged access and mount docker socket |\n| [services](#services-description)   | array   | services description                                   |\n| [artifacts](#artifacts-description) | hasmap  | download artifact from docker instance                 |\n\n#### services description\n\nServices is an array of hashmaps, is possible up N services with docker module, the following keywords can be used to build a service.\n\n| keyword    | type    | description                                       |\n|------------|---------|---------------------------------------------------|\n| image      | string  | docker instance path name                         |\n| vars       | hashmap | like job vars but exclusive from instance service |\n| alias      | string  | network alias name, otherwise slug image name     |\n| entrypoint | string  | service entrypoint string, otherwise ''           |\n\n#### artifacts description\n\nDownload a files or directories from a docker instance.\n\n| keyword | type  | description                          |\n|---------|-------|--------------------------------------|\n| paths   | array | file or folder relative or full path |\n\n### scp\n\nCopy files and folders to remote host over **scp**.\n\n```edn\n{:module \"scp\"\n :apply [{:src \"./dist/command.bin\"\n          :dest \"/usb/bin/command\"}]}\n```\n\n#### apply arguments\n\n| keyword   | type    | description                     |\n|-----------|---------|---------------------------------|\n| src       | string  | file or directory source        |\n| dest      | string  | file or directory destiny       |\n| recursive | boolean | set **true** for directory copy |\n\n### template\n\nSimple template file that only replace values inside double brackets `{{ varname }}`.\n\n```jinja\nHello {{ NAME }}!\n```\n\n\u003e template file\n\n```edn\n{:module \"template\"\n :vars {NAME \"lines\"}\n :apply [{:src \"./hello-world.j2\"\n          :dest \"/tmp/hello-world.txt\"}]}\n```\n\n### user module\n\nLines provides interface with custom user module.\n\nJust put additional clojure scripts at `.lines/modules/\u003cmodule_name\u003e/module.clj`, like following example.\n\n```clj\n; .lines/modules/git/module.clj\n\n; create a boilerplate function for git command\n(str-use [\"git\"])\n\n; custom user function, params: job (receive job definition), i (apply index)\n; the function to return a string eg. `git clone -v git@github.com:rosineygp/mkdkr.git mkdkr`\n(defn str-git-command-line [job i]\n  (git [\"clone\"\n        \"-v\"\n        (get i :repos)\n        (get i :dest)]))\n\n; lines will call this function `lines-module-\u003cmodule_name\u003e`\n(defn lines-module-git [job]\n  (lines-task-loop job str-git-command-line)) ; loop handler\n```\n\nUsing user module.\n\n```edn\n{:module \"git\"\n :apply [{:repos \"git@github.com:rosineygp/lines.git\" :dest \"lines\"}\n         {:repos \"git@github.com:rosineygp/mkdkr.git\" :dest \"mkdkr\"}]}\n```\n\n## EDN Pipeline\n\nPipelines is an array of jobs, and can be described using only edn.\n\n```edn\n; file: node.edn\n\n[{:name \"build\"\n  :apply [\"npm install\"]}\n [{:name \"unit test\"\n   :group [\"test\"]\n   :apply [\"npm unit\"]}\n  {:name \"mocha test\"\n   :group [\"test\"]\n   :apply [\"npm mocha\"]}]\n {:name \"deploy\"\n  :apply [\"npm deploy\"]}]\n```\n\nJobs inside pipeline can be executed in parallel, just group then with `[ array ]`, is possible use custom keywords for better notation like `:groups`.\n\n```shell\n# execute all jobs\nlines -p node.edn\n\n# filter only tests\nlines -p node.edn -j group=test\n```\n\n### Targets file\n\nIs possible describe hosts targets using `edn` files, like pipeline.\n\n```edn\n; file: hosts.edn\n\n[{:label \"vm-0\" :host \"192.168.1.4\" :method \"ssh\" :user \"ubuntu\"}\n {:label \"vm-1\" :host \"192.168.1.5\" :method \"ssh\" :user \"ubuntu\"}]\n```\n\n\u003e ssh only works with `authorized_keys` pre-configured, command `ssh-copy-id` can help configure it.\n\n```shell\n# execute pipeline in all hosts\nlines -p node.edn -i hosts.edn\n\n# filter only wm-0 host\nlines -p node.edn -i hosts.edn -l label=wm-0\n\n# filter jobs deploy and host wm-1\nlines -p node.edn -i hosts.edn -l label=wm-1 -j name=deploy\n```\n\n\u003e EDN files are not safe is possible call functions dynamically.\n\n```edn\n; unsafe pipeline, extensions are based on this behavior\n\n[{:apply [(str \"echo \" (time-ms))]}]\n```\n\n## Clojure Pipeline\n\nPowerful and dynamic pipelines with complex scenarios.\n\n```clojure\n; generate single job for each yml found and test with docker image yamllint\n\n(lines-pp (parallel\n    (map (fn [x] (assoc {}\n                        :name (get x :object)\n                        :module \"docker\"\n                        :args {:image \"my/yamllint\"}\n                        :apply [(str \"yamllint \" (get x :object))]))\n            (filter (fn [i] (= \"yml\" (get i :type))) (list-dir \"src/\")))))\n```\n\n\u003e This project use this case to test itself at `.lines.clj`.\n\n### Lines functions\n\nThe following functions will help to build lines clojure scripts.\n\n| name              | description                                              |\n|-------------------|----------------------------------------------------------|\n| job               | execute edn job                                          |\n| parallel          | execute jobs in parallel                                 |\n| str-use           | mapping command line to function, return str not execute |\n| use               | mapping command line to function, run immediately        |\n| pipeline          | execute end pipeline                                     |\n| lines-pp          | beautiful print pipeline                                 |\n| lines-pp-minimal  | beautiful print pipeline (single line)                   |\n| merge-job-targets | generate a mixed list with edn pipelines and targets     |\n| isremote?         | check if target will run local or in remote machine      |\n\n`src/core.clj` and `src/includes/lang-utils.clj` has more useful functions.\n\nAdditional functions included in Lines.\n\nfilter | pmap | println_stderr | exit! | trap! | unset | str-join | str-subs | range | str-ident | mod | file-exists | file-write | unlink | or | and | hashmap-list | merge | key-name | call | callable? | get-in | even? | odd? | load-once | spit\n\nThe result of function `job` is the following data:\n\n```edn\n({:attempts 1\n  :args {}\n  :module \"shell\"\n  :status true\n  :apply [\"echo hello world!\"]\n  :name \"lines\"\n  :retries 0\n  :target {:label \"local\" :method \"local\"}\n  :pipestatus ((0))\n  :finished 1608684590507\n  :vars {\"BRANCH_NAME_SLUG\" \"master\" \"BRANCH_NAME\" \"master\"}\n  :ignore-error false\n  :start 1608684587657\n  :result (({:exit-code 0\n             :finished 1608684590253\n             :cmd \"echo hello world!\"\n             :stdout \"hello world!\"\n             :stderr \"\"\n             :start 1608684590233\n             :debug \"  bash -s \u003c\u003c-'LINES-BLOCK-EOF'\\n export BRANCH_NAME_SLUG=\\\"master\\\" BRANCH_NAME=\\\"master\\\" ;\\n echo hello world! \\nLINES-BLOCK-EOF\"}))})\n```\n\n## Environment vars\n\nLines variables definition.\n\n| name                   | default           | description                                      |\n|------------------------|-------------------|--------------------------------------------------|\n| LINES_JOB_TTL          | 3600              | max time that a job can run (only docker module) |\n| LINES_JOB_MAX_ATTEMPTS | 2                 | max number of retries                            |\n| LINES_MODULES_DIR      | \".lines/modules/\" | modules default location                         |\n| LINES_EXT_DIR          | \".lines/ext/\"     | extensions default location                      |\n| FLK_MAX_THREADS        | undefined         | max number of threads in parallel execution      |\n\n## Extensions\n\nIs possible create functions and extend usability, all extensions must be keep at `.lines/ext/`, it is just a clojure script with useful functions.\n\n```edn\n; file: .lines/ext/apt-helper.clj\n\n(str-use [\"apt-get\"])\n\n(defn apt-update []\n  (apt-get [\"update\"]))\n\n(defn apt-args [options packages]\n   (apt-get  (concat options packages)))\n\n(defn apt-install [packages]\n  (apt-args [\"install\" \"-y\"] packages))\n\n(defn apt-remove [packages]\n  (apt-args [\"remove\" \"-y\"] packages))\n```\n\nNow edn files or pipeline can call those functions.\n\n```edn\n; edn pipeline\n\n[{:name \"apt with extensions\"\n  :apply [(apt-update)\n          (apt-install [\"python\"\n                        \"mysql-server\"\n                        \"memcached\"])]}]\n```\n\n## Development\n\nBuild this project\n\n```shell\n./build.sh\n\n\u003e result\n  flk lines\n```\n\n`flk` is same of [Fleck](https://github.com/chr15m/flk) with `patches/` applied and `lines` is flk with all `src/` included.\n\nTesting changes in `src/` without build.\n\n```shell\n./flk src/main.clj \u003cparams\u003e\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frosineygp%2Flines","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frosineygp%2Flines","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frosineygp%2Flines/lists"}