{"id":16936566,"url":"https://github.com/jonhoo/experiment","last_synced_at":"2025-04-11T19:07:55.414Z","repository":{"id":17553963,"uuid":"20356844","full_name":"jonhoo/experiment","owner":"jonhoo","description":"A tool for running concurrent multi-configuration experiments.","archived":false,"fork":false,"pushed_at":"2017-07-10T00:05:35.000Z","size":92,"stargazers_count":12,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-11T01:48:28.656Z","etag":null,"topics":["experimentation","measurement","orchestration"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jonhoo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-05-31T14:04:37.000Z","updated_at":"2025-02-06T02:48:27.000Z","dependencies_parsed_at":"2022-08-25T18:30:22.586Z","dependency_job_id":null,"html_url":"https://github.com/jonhoo/experiment","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonhoo%2Fexperiment","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonhoo%2Fexperiment/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonhoo%2Fexperiment/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonhoo%2Fexperiment/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jonhoo","download_url":"https://codeload.github.com/jonhoo/experiment/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248465316,"owners_count":21108244,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["experimentation","measurement","orchestration"],"created_at":"2024-10-13T20:57:18.632Z","updated_at":"2025-04-11T19:07:55.392Z","avatar_url":"https://github.com/jonhoo.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# experiment\n[![Build Status](https://travis-ci.org/jonhoo/experiment.svg?branch=master)](https://travis-ci.org/jonhoo/experiment)\n\nExperiment is a tool for running concurrent multi-configuration experiments.\n\nQuite often, we need to run different versions of an application to\ndetermine the effect of a change, or we need to run it multiple times\nwith different parameters for benchmarking. Furthermore, to get\nstatistically relevant results, we need to execute each experiment\nmultiple times. For posteriority, we would also like to keep track of\nexactly how the application was run, how many times each version has\nbeen run, and what the exact changes were made to the application for\neach experiment.\n\nexperiment tries to solve this by integrating closely with version\ncontrol systems, allowing developers to specify exactly which versions\nof the application to build, and what changes to apply (if any). It will\nexecute each version multiple times, possibly concurrently, and report\nback when it finishes, leaving you to do other things than wait for one\nexperiment to finish before starting the next.\n\n## usage\n\nExperiment gets information about what versions it should run from an\n`experiment.json` file. This file has a number of root-level fields:\n\nField       | Type   | Purpose\n------------|--------|--------\nexperiment  | string | A description of the experiment being run.\nrepository  | string | Specifies the repository to draw source code from (optional; can use `-r` instead).\ncheckout    | string | Default commit, branch or tag to check out from the repository.\niterations  | number | The number of times to run each version.\nparallelism | number | The number of versions to run in parallel.\nbuild       | string | Default command used to build the application under test -- passed to system shell.\nkeep-stdout | bool   | If false or unset, application standard output is discarded. If true, it is kept in stdout.log.\narguments   | array  | Default command and arguments to run the application with -- **not** passed to system shell. The special string `$SRC` will be replaced with the source code directory the current version was built from.\ninto        | string | Copy the source tree into a subdirectory of the build directory (useful for e.g. Go packages)\npreserve    | array  | Preserves dirty or unstaged changes to files or directories from the working directory\nversions    | hash   | Described below.\n\nThe `versions` hash is where all the versions you want experiment to\nexecute are defined. For each version, experiment will clone the source\nrepository, check out the appropriate commit, build the application, and\nthen run it. In the output directory (specified with `-o`; defaults to\n`.`), a directory is created for each version (the version's key is used\nas the directory name). Each such directory contains a `source`\ndirectory holding the sources the version was built from, a `build.log`\nfile giving information about the version's build, as well as\n`iterations` directories called `run-1`, `run-2`, etc. These all hold at\nleast two files: `experiment.log` and `stderr.log`. `experiment.log`\ncontains information about how the process was run, and how long it took\nto execute. `stderr.log` contains the error output of the application.\nIf `keep-stdout` is set to `true`, a file called `stdout.log` holding\nthe application's regular output will also be present.\n\nEach version may override the default `checkout`, `build`, and\n`arguments` if the wish. In addition, they may specify a list of\n`diffs`. Each `diff` is a patch file that will be applied to the\nversion's source directory before it is built.\n\nVersions are run in random order within each iteration, but each\niteration waits for all versions in the previous iteration to be started\nbefore any version in the next iteration is started.\n\nTo start the experiment, simply run\n\n    $ experiment\n\nand experiment will read the `experiment.json` file and start running\nyour jobs. It will show a progress report, and notify you when the job\nhas finished.\n\n## templated versions\n\nExperiment also supports *templated versions*. A templated version is\nexpanded to multiple versions, each with some set of parameters. To give\nan example:\n\n```javascript\n{\n\t// ...\n\t\"versions\": {\n\t\t// ...\n\t\t\"$animal-$number\": {\n\t\t\t\"vary\": {\n\t\t\t\t\"animal\": \"set(rabbit, turtle)\",\n\t\t\t\t\"number\": \"range(1, 3, 1)\"\n\t\t\t},\n\t\t\t\"arguments\": [\n\t\t\t\t\"$SRC/run\",\n\t\t\t\t\"$animal\",\n\t\t\t\t\"-n\",\n\t\t\t\t\"$number\"\n\t\t\t]\n\t\t}\n\t\t// ...\n\t}\n\t// ...\n}\n```\n\nthis will produce four different versions: `rabbit-1`, `rabbit-2`,\n`turtle-1`, and `turtle-2`, run with `$SRC/run rabbit -n 1`, `$SRC/run\nrabbit -n 2`, etc. Currently, the only vary functions that are supported\nare `set`, `range`, and `cmd`. `set` produces all given values, which\nare comma-separated and may be quoted. `range(a, b, c)` produces every\nnumber less than `b`, starting at `a`, in increments of `c`.\n\n`cmd` is special in that it executes a shell command (using Ruby's\n[`system`](http://ruby-doc.org/core-2.2.0/Kernel.html#method-i-system)\ncommand), and expands into the values output by the command when run.\nThe command is run in the same directory as `experiment` was called from\n(this is because templates are expanded before the repository has even been\ncloned, and so cannot refer to anything else). There are two variants of\nthis function: `cmd` and `cmd_l`. These differ only in what delimiter\nthey use to distinguish different values in the output; `cmd` uses `\\0`,\nwhereas `cmd_l` uses newlines. **Authors should be careful about using\n`cmd_l`, as values containing newlines will be misinterpreted as\nmultiple values.**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonhoo%2Fexperiment","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjonhoo%2Fexperiment","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonhoo%2Fexperiment/lists"}