{"id":13827096,"url":"https://github.com/skroutz/mistry","last_synced_at":"2025-04-27T03:31:01.353Z","repository":{"id":30908856,"uuid":"123288096","full_name":"skroutz/mistry","owner":"skroutz","description":"General-purpose build server","archived":true,"fork":false,"pushed_at":"2023-07-05T21:13:31.000Z","size":635,"stargazers_count":104,"open_issues_count":36,"forks_count":8,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-14T14:15:00.174Z","etag":null,"topics":["build","build-server","build-system","builder","incremental-builds"],"latest_commit_sha":null,"homepage":"https://engineering.skroutz.gr/blog/speeding-up-build-pipelines-with-mistry/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/skroutz.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-02-28T13:23:13.000Z","updated_at":"2025-03-13T08:08:49.000Z","dependencies_parsed_at":"2024-01-15T16:13:17.255Z","dependency_job_id":"5fda9123-bddf-4d71-a3eb-c711a007737f","html_url":"https://github.com/skroutz/mistry","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skroutz%2Fmistry","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skroutz%2Fmistry/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skroutz%2Fmistry/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skroutz%2Fmistry/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/skroutz","download_url":"https://codeload.github.com/skroutz/mistry/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251085147,"owners_count":21533821,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["build","build-server","build-system","builder","incremental-builds"],"created_at":"2024-08-04T09:01:49.940Z","updated_at":"2025-04-27T03:31:00.918Z","avatar_url":"https://github.com/skroutz.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"![mistry logo](logo.png)\n\n------------------------------------------------------------------------------\n\n[![Build Status](https://api.travis-ci.org/skroutz/mistry.svg?branch=master)](https://travis-ci.org/skroutz/mistry)\n[![Go report](https://goreportcard.com/badge/github.com/skroutz/mistry)](https://goreportcard.com/report/github.com/skroutz/mistry)\n[![License: GPL v3](https://img.shields.io/badge/License-GPL%20v3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)\n\n*mistry* is a general-purpose build server that enables fast workflows by\nemploying artifact caching and incremental building techniques.\n\nmistry executes user-defined build steps inside isolated environments\nand saves build artifacts for later consumption.\n\nRefer to the introductory blog post *[Speeding Up Our Build Pipelines](https://engineering.skroutz.gr/blog/speeding-up-build-pipelines-with-mistry/)*\nfor more information.\n\nAt Skroutz we use mistry to speed our development and deployment\nprocesses:\n\n- Rails asset compilation (`rails assets:precompile`)\n- Bundler dependency resolution and download (`bundle install`)\n- Yarn dependency resolution and download (`yarn install`)\n\nIn the above use cases, mistry executes these commands once they are needed for\nthe first time and caches the results. Then, when anyone else executes the same\ncommands (i.e.  application servers, developer workstations, CI server etc.)\nthey instantly get the results back.\n\n\n\n\nFeatures\n------------------------------------------------------------------------------\n\n- execute user-defined build steps in pre-defined environments, provided as Docker images\n- build artifact caching\n- incremental building (see [*\"Build cache\"*](https://github.com/skroutz/mistry/wiki/Build-cache))\n- [CLI client](cmd/mistry/README.md) for interacting with the server (scheduling jobs etc.)\n  via a JSON API\n- a web view for inspecting the progress of builds (see [*\"Web view\"*](#web-view))\n- efficient use of disk space due to copy-on-write semantics (using [Btrfs snapshotting](https://en.wikipedia.org/wiki/Btrfs#Subvolumes_and_snapshots))\n\n\n\nFor more information visit the [wiki](https://github.com/skroutz/mistry/wiki).\n\n\n\n\n\n\n\n\n\n\n\nGetting started\n-------------------------------------------------\nYou can get the binaries from the\n[latest releases](https://github.com/skroutz/mistry/releases).\n\nAlternatively, if you have Go 1.10 or later you can get the\nlatest development version.\n\nNOTE: [statik](https://github.com/rakyll/statik) is a build-time dependency,\nso it should be installed in your system and present in your PATH.\n\n```shell\n$ go get github.com/rakyll/statik\n\n# server\n$ go get -u github.com/skroutz/mistry/cmd/mistryd\n\n# client\n$ go get -u github.com/skroutz/mistry/cmd/mistry\n```\n\n\n\n\n\nUsage\n--------------------------------------------------\nTo boot the server a configuration file is needed:\n\n```shell\n$ mistryd --config config.json\n```\n\nYou can use the [sample config](cmd/mistryd/config.sample.json) as a starting\npoint.\n\nUse `mistryd --help` for more info.\n\n\n\n### Adding projects\n\nProjects are essentially directories with at minimum a `Dockerfile` at their\nroot. Each project directory should be placed in the path denoted by\n`projects_path` (see [*Configuration*](#configuration).\n\nRefer to [*File system layout - Projects directory*](https://github.com/skroutz/mistry/wiki/File-system-layout#projects-directory)\nfor more info.\n\n\n\n\n\n### API\n\nInteracting with mistry (scheduling builds etc.) can be done in two ways:\n(1) using the [client](cmd/mistry/README.md) and (2)\nusing the HTTP API directly (see below).\n\nWe recommended using the client whenever possible.\n\n#### Client\n\nSchedule a build for project *foo* and download the artifacts:\n\n```sh\n$ mistry build --project foo --target /tmp/foo\n```\n\nThe above command will block until the build is complete and then download the\nresulting artifacts to `/tmp/foo/`.\n\nSchedule a build without fetching the artifacts:\n\n```sh\n$ mistry build --project foo --no-wait\n```\n\nThe above will just schedule the build and return immediately - it will not\nwait for it to complete and will not fetch the artifacts.\n\nFor more info refer to the client's [README](cmd/mistry/README.md).\n\n#### HTTP Endpoints\n\nSchedule a new build without fetching artifacts (this is equivalent to passing\n`--no-wait` when using the client):\n\n```shell\n$ curl -X POST /jobs \\\n    -H 'Accept: application/json' \\\n    -H 'Content-Type: application/json' \\\n    -d '{\"project\": \"foo\"}'\n{\n    \"Params\": {\"foo\": \"xzv\"},\n    \"Path\": \"\u003cartifact path\u003e\",\n    \"Cached\": true,\n    \"Coalesced\": false,\n    \"ExitCode\": 0,\n    \"Err\": null,\n    \"TransportMethod\": \"rsync\"\n}\n```\n\n\n### Web view\n\nmistry comes with a web view where progress and logs of each build can be\ninspected.\n\nBrowse to http://0.0.0.0:8462 (or whatever address the server listens to).\n\n\n\n\n\n\n\n\n\nConfiguration\n-------------------------------------------------\nConfiguration is provided in JSON format. The following settings are currently\nsupported:\n\n| Setting        | Description           | Default  |\n| ------------- |:-------------:| -----:|\n| `projects_path` (string) | The path where project folders are located | \"\" |\n| `build_path` (string) | The root path where artifacts will be placed       |   \"\" |\n| `mounts` (object{string:string}) | The paths from the host machine that should be mounted inside the execution containers     |    {} |\n| `job_concurrency` (int) | Maximum number of builds that may run in parallel | (logical-cpu-count) |\n| `job_backlog` (int) | Used for back-pressure - maximum number of outstanding build requests. If exceeded subsequent build requests will fail | (job_concurrency * 2) |\n\nThe paths denoted by `projects_path` and `build_path` should be\npresent and writable by the user running the server.\n\nFor an example refer to the [sample config](cmd/mistryd/config.sample.json).\n\n\n\n\n\nDevelopment\n---------------------------------------------------\n\nBefore anything, make sure you install the dependencies:\n```shell\nmake deps\n```\n\nThe tests will attempt to ssh to localhost. You will need to add your\npublic key to the authorized keys as if you were setting this up to a remote\nhost.\n\n```shell\ncat ~/.ssh/id_rsa.pub \u003e\u003e ~/.ssh/authorized_keys\n```\n\nTo run the tests, the [Docker daemon](https://docs.docker.com/install/) should\nbe running and SSH access to localhost should be configured.\n\n```shell\n$ make test\n```\n\nNote: the command above may take more time the first time it's run,\nsince some Docker images will have to be fetched from the internet.\n\n\nLicense\n-------------------------------------------------\nmistry is released under the GNU General Public License version 3. See [COPYING](COPYING).\n\nmistry [logo](logo.png) contributed by @cyfugr\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskroutz%2Fmistry","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fskroutz%2Fmistry","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskroutz%2Fmistry/lists"}