{"id":15150378,"url":"https://github.com/bvarnai/respository-installer","last_synced_at":"2026-01-19T14:12:10.011Z","repository":{"id":234733378,"uuid":"773629676","full_name":"bvarnai/respository-installer","owner":"bvarnai","description":"A command-line, configuration based tool to install git repositories","archived":false,"fork":false,"pushed_at":"2024-04-26T09:59:29.000Z","size":6641,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-04T06:29:20.601Z","etag":null,"topics":["bash-script","command-line","curl","devops","git","jq","shell-script","tooling"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bvarnai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-18T05:30:26.000Z","updated_at":"2024-04-26T09:57:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"c3389940-cb72-4a6b-bb8e-33f31b4819a2","html_url":"https://github.com/bvarnai/respository-installer","commit_stats":{"total_commits":27,"total_committers":3,"mean_commits":9.0,"dds":0.2222222222222222,"last_synced_commit":"7d624651c096aff76afab9f6aba692b6a4a4e83a"},"previous_names":["bvarnai/respository-installer"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/bvarnai/respository-installer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bvarnai%2Frespository-installer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bvarnai%2Frespository-installer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bvarnai%2Frespository-installer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bvarnai%2Frespository-installer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bvarnai","download_url":"https://codeload.github.com/bvarnai/respository-installer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bvarnai%2Frespository-installer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28571587,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-19T12:50:50.164Z","status":"ssl_error","status_checked_at":"2026-01-19T12:50:42.704Z","response_time":67,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bash-script","command-line","curl","devops","git","jq","shell-script","tooling"],"created_at":"2024-09-26T14:03:16.118Z","updated_at":"2026-01-19T14:12:09.994Z","avatar_url":"https://github.com/bvarnai.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# installer - a configuration based repository installer\n\n![CI badge](https://github.com/bvarnai/respository-installer/actions/workflows/ci.yml/badge.svg)\n\n**installer** is a tool to help users to work with multiple *Git* repositories from the initial clone to getting updates.\n\n**Where does it fit?**\n\nI worked in a Java development team, we had about 15 repositories. I needed a simple tool which is\n\n- Self-contained and updateable\n- Configuration based\n- Supports development `streams` (for example parallel tooling for java17, java21 etc.)\n- *Git* only (minimal platform specific code)\n\nand **nothing** more.\n\n:bulb: I use the word *project* interchangeably with *repository*\n\n## Table of contents\n\n- [installer - a configuration based repository installer](#installer---a-configuration-based-repository-installer)\n  - [Table of contents](#table-of-contents)\n  - [Demo](#demo)\n  - [Installation](#installation)\n    - [Supported SCM types](#supported-scm-types)\n      - [GitHub](#github)\n        - [Token configuration](#token-configuration)\n      - [Bitbucket Enterprise](#bitbucket-enterprise)\n        - [Token configuration](#token-configuration-1)\n      - [Plain HTTP](#plain-http)\n    - [Getting started for the first time](#getting-started-for-the-first-time)\n    - [Git credentials](#git-credentials)\n    - [Prerequisites](#prerequisites)\n  - [Configuration](#configuration)\n    - [Workspace](#workspace)\n    - [Configuration file](#configuration-file)\n  - [Usage](#usage)\n    - [Options](#options)\n      - [Options for development/testing](#options-for-developmenttesting)\n    - [Link mode](#link-mode)\n    - [Stream explained](#stream-explained)\n    - [Custom environments](#custom-environments)\n      - [Dependencies](#dependencies)\n      - [Link/unlink](#linkunlink)\n    - [Commands](#commands)\n      - [help](#help)\n      - [list](#list)\n      - [install](#install)\n      - [update](#update)\n  - [FAQ](#faq)\n  - [Development notes](#development-notes)\n\n---\n\n## Demo\n\n![installer demo](docs/demo.gif)\n\n---\n\n## Installation\n\nFor easier understanding of the initial steps, the following diagram provides an overview\n\n![Overview](docs/overview.png)\n\nTo get started, you will need the following\n\n- Configuration file aka `projects.json` (see [Configuration file](#configuration-file))\n\nAnd these environment variables pointing to your configuration\n\n- `INSTALLER_CONFIG_URL` - URL of the configuration `projects.json`\n- `INSTALLER_CONFIG_SCM` - type of SCM (GitHub etc.) used for the configuration\n\nIf you are using GitHub, only `INSTALLER_CONFIG_URL` is needed.\n\n:memo: For the best user experience, I recommend forking **installer** and set defaults according to your environment\n\n### Supported SCM types\n\n**installer** needs to know how to get the configuration file form the SCM without actually cloning it. This means assembling a URL used by `curl` to get the configuration. The following SCM types are supported:\n  - github - GitHub *[default]*\n  - bitbucket_server - Bitbucket Enterprise (server/data center)\n  - plain - Plain HTTP\n\n:bulb: This is only used for configuration file discovery, you can use any *Git* platform later for your projects. Authentication for *Git* commands are based on your *Git* configuration.\n\n:warning: Bitbucket Cloud is not yet supported\n\n#### GitHub\n\nTo get the configuration file the following URL format is used:\n\n```\nhttps://#token#@raw.githubusercontent.com/\u003cuser or organization\u003e/\u003crepo name\u003e/#branch#/\u003cpath to file\u003e/\u003cfile name\u003e\n```\nThe following variables are used in the URL:\n\n- `#token#` is replaced with `INSTALLER_CONFIG_TOKEN` env. variable which holds your *Personal access token* or PAT\n- `#branch#` is replaced with the current branch (this done automatically)\n\n##### Token configuration\n\nTo create token or PAT follow the official guide [Creating a personal access token (classic)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic)\n\nMake sure you set `repo` scope (and nothing more) when creating the PAT.\n\n![github-pat](docs/github-pat.png).\n\n:bulb: Token is needed for private repositories only\n\nFor example, using your private repositories would need the following settings:\n```bash\nexport INSTALLER_CONFIG_URL=https://#token#@raw.githubusercontent.com/user/repo/#branch#/projects.json\nexport INSTALLER_CONFIG_TOKEN=1bacnotmyrealtoken123beefbea\n```\n\n#### Bitbucket Enterprise\n\nSince Bitbucket uses the URL's query string to specify the branch, there is no need to use special URL variables. The format is the following:\n\n```\nhttps://\u003cserver url\u003e/projects/\u003cproject name\u003e/repos/\u003crepo name\u003e/raw/\u003cpath to file\u003e/\u003cfile name\u003e?\u003cbranch\u003e\n```\n\n##### Token configuration\n\nTo create token or `HTTP access token` follow the official guide [HTTP access tokens](https://confluence.atlassian.com/bitbucketserver/http-access-tokens-939515499.html)\n\nMake sure you set `Project read` and `Repository read` permissions (and nothing more) when creating the token.\n\n![bitbucket-token](docs/bitbucket-token.png)\n\nToken is inserted in the header using `curl`\n\n```\n-H Authorization: Bearer ${token}\n```\n\n:bulb: Token is needed for private repositories only\n\nFor example, using your private repositories would need the following settings:\n```bash\nexport INSTALLER_CONFIG_URL=https://contoso/projects/project/repos/repo/raw/projects.json\nexport INSTALLER_CONFIG_SCM=bitbucket_server\nexport INSTALLER_CONFIG_TOKEN=1bacnotmyrealtoken123beefbea\n```\n\n#### Plain HTTP\n\nThis type is mainly used for testing and it's very similar to GitHub's format, only that `token` or any other authentication is not supported.\n\nTo get the configuration file the following URL format is used:\n\n```\nhttps://\u003cserver url\u003e/\u003cserver path\u003e/#branch#/\u003cpath to file\u003e/\u003cfile name\u003e\n```\nThe following variables are used in the URL:\n\n- `#branch#` is replaced with the current/working branch (this done by the script)\n\nFor example, using your `localhost` server for configuration:\n```\nexport INSTALLER_CONFIG_URL=https://localhost:8080/folder/#branch#/projects.json\n```\n\nBranches are simply folders like `main`, `master` etc.\n\n:memo: You can set these variables in `~/.profile` or `~/.bashrc` to make them permanent\n\n### Getting started for the first time\n\nNext get the **installer** with `curl` for the first time:\n\n```bash\ncurl -L https://raw.githubusercontent.com/bvarnai/respository-installer/main/src/installer.sh -o installer.sh \u0026\u0026 chmod +x installer.sh\n```\n\nFinally run **installer** in the current working directory:\n\n```bash\n./installer.sh\n```\n:tada: Once downloaded **installer** will upgrade itself, no need to run `curl` again.\n\n### Git credentials\n\n:warning: **installer** doesn't manage your *Git* credentials (ssh keys etc.) in any way. You need setup *Git* credentials to work with the repositories specified in your configuration\n\n### Prerequisites\n\nFollowing tools are required and must be installed:\n  - `git`\n  - `curl`\n  - `jq`\n  - `sed`\n  - `uname`\n  - `awk`\n  - `grep`\n  - `bash` \u003e= 4.0.0\n\n## Configuration\n\n### Workspace\n\nWorkspace is the directory where your repositories/projects are cloned. It's also the current working directory where **installer** runs. Projects in the configuration\nare specified *relative* to this directory.\n\nExample layout with `installer.sh` present:\n```bash\nworkspace-root\n  project1\n  project2\n  subfolder/project3\n  installer.sh\n  projects.json\n```\n\n### Configuration file\n\nThe configuration file is called `projects.json` and it's downloaded using the `INSTALLER_CONFIG_URL` environment variable. It contains information about all your *Git* projects, including setup instructions.\n\n```json\n{\n  \"bootstrap\": \"myproject\",\n  \"projects\": [\n    {\n      \"name\": \"myproject\",\n      \"category\": \"development\",\n      \"default\": \"true\",\n      \"urls\": {\n        \"fetch\": \"https://github.com/octocat/Hello-World.git\",\n        \"push\": \"git@github.com:octocat/Hello-World.git\"\n      },\n      \"options\": {\n        \"clone\": \"--depth 1\"\n      },\n      \"configuration\": [\n        \"core.autocrlf false\",\n        \"core.safecrlf false\"\n      ],\n      \"branch\": \"master\",\n      \"update\": \"true\",\n      \"doLast\": [\n        \"./do_something.sh\"\n      ]\n    }\n  ]\n}\n```\n\n| Elements       |                          |       | Description |\n| -------------- | ------------------------ | ----  | ----------- |\n| bootstrap      |                          |       | Bootstrap project is always added implicitly. Referenced by `name` in `projects`          |\n| projects       |                          |       | Array of projects |\n|                | name                     |       | Project name |\n|                | path                     |       | Project path. Relative to workspace root. If not specified `name` will be used as path *[optional]* |\n|                | category                 |       | Project category. Informal tagging of projects. Displayed during project listing *[optional]* |\n|                | default                  |       | Whether to install the project if no project set is specified |\n|                | urls                     |       | *Git* repository URLs |\n|                |                          | fetch | URL used for `fetch` |\n|                |                          | push  | URL used for `push`. If not specified `fetch` URL will be used *[optional]* |\n|                | options                  |       | *Git* command options |\n|                |                          | clone | Options for `clone` command. For example `--depth 1`\" would result in a shallow clone *[optional]* |\n|                | configuration            |       | Array of *Git* configuration `config` options, repository scope. Add `--global` for global scope *[optional]* |\n|                | branch                   |       | Default branch |\n|                | update                   |       | Whether to force the repository update and reset to latest on the default branch |\n|                | doLast                   |       | Array of shell commands to execute after repository update *[optional]* |\n\n\n:memo: Additional notes\n- A bootstrap project is simply a project that is always installed\n- :warning: A bootstrap project must be set to default `default==true` as well\n- Different `fetch` and `push` URLs can be used to reduce load in *Git* hosting server, for example use `https` for `fetch` and `ssh` for `push`\n- Setting `update==false` means repositories are fetched but not updated. This is desirable for development projects, so working branches are felt unchanged\n- :warning: Setting `update==true` means repositories are fetched, reset and updated. This also means the branch will be switched to the default branch\n\n:bulb: You can use a bootstrap project to host your DevOps scripts etc. for example doLast scripts\n\n## Usage\n\nCommand syntax is the following:\n```bash\n./installer.sh [options] [\u003ccommand\u003e] [arguments]\n```\n\nOptional elements are shown in brackets []. For example, command may take a list of projects as an argument.\n\n### Options\n\n- `-y, --yes` - skip user prompts\n- `--link` - use symlinks to target directory\n- `--branch` - overrides `branch` setting in configuration\n- `--stream` - specifies the `stream` of the configuration\n- `--fetch-all` - fetches all remotes, branches\n- `--prune` - prune during fetch\n- `--git-quiet` - pass quite to git commands (not everything is suppressed)\n- `--skip-dolast` - do not run doLast commends (useful in CI environments where some setup is not wanted)\n\n#### Options for development/testing\n\n- `--skip-self-update` - skip the script update step\n- `--use-local-config` - use a local configuration file\n\n### Link mode\n\nIn some cases, you don't want to have a fresh clone of a project to save some time. For example *Jenkins* multibranch  pipeline would create a new workspace and make a fresh clone using **installer**. This is where `link` mode can help.\n\nLet see an example *Jenkinsfile*\n\n```groovy\npipeline {\n\n    environment {\n\n        // installer configuration\n        INSTALLER_SELF_URL = 'https://raw.githubusercontent.com/bvarnai/respository-installer/#branch#/src/installer.sh'\n        INSTALLER_CONFIG_URL = 'https://raw.githubusercontent.com/bvarnai/respository-installer/#branch#/src/projects.json'\n\n        // use a directory outside of job's workspace\n        SHARED_WORKSPACE = \"${WORKSPACE}/../shared_workspace\"\n    }\n\n    stages {\n        stage('Prepare workspace') {\n            steps {\n                // install dependencies\n                sh '''\n                mkdir -p ${SHARED_WORKSPACE}\n\n                curl -s -o installer.sh -L ${INSTALLER_SELF_URL} \u0026\u0026 chmod +x installer.sh\n                ./installer.sh --yes --link ${SHARED_WORKSPACE} myproject1 myproject2\n                '''\n            }\n        }\n        stage ('Next') {\n            steps {\n              ...\n            }\n        }\n    }\n}\n```\n\nThis will create symlinks `myproject1` and `myproject2` in the job's workspace, pointing to `../shared_workspace/myproject1` and `../shared_workspace/myproject2` directories respectively.\n\n:warning: If you have multiple executors, parallel jobs might be running on the same shared workspace directory. This can be prevented by using the `EXECUTOR_NUMBER` variable\n\n```groovy\nSHARED_WORKSPACE = \"${WORKSPACE}/../shared_workspace/${EXECUTOR_NUMBER}\"\n```\n\n### Stream explained\n\nFor example the team is working on a \"theoretical\" Java update, migrating from Java 8 to Java 17. In the development project repository, they created a branch `java17` and started to work. However `main` development continues on Java 8 until everything is ready. `java17` branch needs the Java 17 JDK, tools etc. This means there are two parallel `stream`s of development. There will be two `projects.json` files on the corresponding branches with default branches set to `main` or `java17`.\n\nIf a developer works on `java17` branch, simply switches tooling to that stream\n```bash\n./installer.sh --stream java17 update\n```\n\nOther developer who remains on `main` just continues as\n```bash\n./installer.sh --stream main update\n```\n\n### Custom environments\n\n#### Dependencies\n\nIn some cases you want/need to manage your dependencies independently from the environment. For example *Git for Windows* does not include `jq` by default.\nYou can add custom code to **installer** to bootstrap `jq` and download it on the fly.\n\n```bash\nfunction user_get_dependencies()\n{\n  # put your code here\n  :\n}\n```\n\nThis function could override the following globals with the location of these executables:\n- `INSTALLER_JQ`\n- `INSTALLER_CURL`\n\nAn example implementation for `jq` for *Linux/Git Bash*\n\n```bash\nfunction user_get_dependencies()\n{\n  if jq --version \u003e /dev/null 2\u003e\u00261; then\n    # jq is available in PATH\n    INSTALLER_JQ='jq'\n  elif .installer/jq --version \u003e /dev/null 2\u003e\u00261; then\n    # jq is available in local temp\n    INSTALLER_JQ='.installer/jq'\n  else\n    # jq will be downloaded\n    local JQSourceURL\n    if [[ \"${system}\" == \"Linux\" ]]; then\n      JQSourceURL='https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-linux-amd64'\n    elif [[ \"${system}\" =~ ^(MINGW64_NT|MSYS_NT) ]]; then\n      JQSourceURL='https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-windows-amd64.exe'\n    else\n      err \"Unsupported system ${system}\"\n      exit 1\n    fi\n    INSTALLER_JQ='.installer/jq'\n  fi\n\n  if [[ -n ${JQSourceURL} ]]; then\n    log \"Getting jq...\"\n    local httpCode\n    httpCode=$(curl_get \"${JQSourceURL}\" '' \"${INSTALLER_JQ}\")\n    if [[ ${httpCode} -ne 200 ]] ; then\n      err \"Failed to download jq binary\"\n      exit 1\n    fi\n\n    # set executable permission\n    chmod +x \"${INSTALLER_JQ}\" \u003e /dev/null 2\u003e\u00261;\n  fi\n}\n```\n\n:memo: I recommend to use `.installer` directory as a \"temp\" directory used to store dependencies\n\n#### Link/unlink\n\nCreating symbolic links might be platform specific. It's definitely the case for *Windows* clients.\n\nThere are user functions `user_link` and `user_unlink` to handle symbolic links.\n\nThe default implementation is tested on the following platforms:\n- Linux amd64 - Ubuntu\n- Windows amd64 - Git for Windows 64 bit\n  - 2.41.0+\n\n### Commands\n\nThe following commands are available. For options see [Options](#options)\n\n---\n#### help\n\n```bash\n./installer.sh help\n```\n\nDisplays the help.\n\n---\n#### list\n\n```bash\n./installer.sh list\n```\n\nLists available projects.\n\n---\n#### install\n\n```bash\n./installer.sh install [project...]\n```\n\nInstalls a project(s). This is the default command, if nothing else is specified.\n\nArguments:\n\n- `project` - the list of projects to install separated by a whitespace\n\n:memo: If you run `install` without any arguments, all projects marked `default==true` will be installed\n\n---\n#### update\n\n```bash\n./installer.sh update\n```\n\nUpdates existing projects in the current directory.\n\n---\n\n## FAQ\n\nTo be added. If you have any questions, just create an issue and I will respond.\n\n## Development notes\n\n- I used Google's [Shell Style Guide](https://google.github.io/styleguide/shellguide.html) with the help of [ShellCheck](https://www.shellcheck.net/)\n- Tests written in [Bats-core: Bash Automated Testing System](https://github.com/bats-core/bats-core)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbvarnai%2Frespository-installer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbvarnai%2Frespository-installer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbvarnai%2Frespository-installer/lists"}