Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://github.com/JuliaCI/Nanosoldier.jl

A package for running JuliaCI services on MIT's Nanosoldier cluster
https://github.com/JuliaCI/Nanosoldier.jl

Last synced: about 1 month ago
JSON representation

A package for running JuliaCI services on MIT's Nanosoldier cluster

Lists

README 

        
# Nanosoldier.jl



[![Build Status](https://travis-ci.org/JuliaCI/Nanosoldier.jl.svg?branch=master)](https://travis-ci.org/JuliaCI/Nanosoldier.jl)



This package contains the infrastructure powering the @nanosoldier CI bot used by the Julia language.



## Quick start



If you're a collaborator in the JuliaLang/julia repository, you can submit CI jobs by

commenting on commits or pull requests. The @nanosoldier bot looks for a special "trigger

phrase" in your comment, and if the trigger phrase is found, it is parsed by the bot to

configure and submit a CI job.



The trigger phrase syntax is:



```

@nanosoldier `command(args..., kwargs...)`

```



Backticks are mandatory. If the job is accepted, a status update will be pushed to the

commit you commented on (look for a yellow dot, green check or red cross). Once the job

finishes, @nanosoldier will reply with a comment, and upload results to the

[NanosoldierReports](https://github.com/JuliaCI/NanosoldierReports) repository.



There are two kinds of jobs you can invoke: **benchmark jobs**, which run the

[BaseBenchmarks.jl](https://github.com/JuliaCI/BaseBenchmarks.jl) suite, and **package test

jobs** which rely on [PkgEval.jl](https://github.com/JuliaCI/PkgEval.jl) to run the test

suite of all registered packages.



**Note that only one job can be triggered per comment.**



One of the most common invocations runs all benchmarks on your PR, comparing against the

current Julia master branch:



```

@nanosoldier `runbenchmarks()`

```



Similarly, you can run all package tests, e.g. if you suspect your PR might be breaking:



```

@nanosoldier `runtests()`

```



Both operations take a long time, so it might be wise to restrict which benchmarks you want

to run, or which packages you want to test:



```

@nanosoldier `runbenchmarks("linalg")`



@nanosoldier `runtests(["JSON", "Crayons"])`

```



When a job is completed, @nanosoldier will reply to your comment to tell you how the job

went and link you to any relevant results.



## Available job types



CI jobs are implemented in this package as subtypes of `Nanosoldier.AbstractJob`. See

[here](https://github.com/JuliaCI/Nanosoldier.jl/blob/master/src/jobs/jobs.jl) for a

description of the interface new job types need to implement.



### `BenchmarkJob`



#### Execution Cycle



A `BenchmarkJob` has the following execution cycle:



1. Pull in the JuliaLang/julia repository and build the commit specified by the context of

   the trigger phrase.

2. Using the new Julia build, fetch the `nanosoldier` branch of the

   [BaseBenchmarks](https://github.com/JuliaCI/BaseBenchmarks.jl) repository and run the

   benchmarks specified by the trigger phrase.

3. If the trigger phrase specifies a commit to compare against, build that version of Julia

   and perform step 2 using the comparison build.

4. Upload a markdown report to the

   [NanosoldierReports](https://github.com/JuliaCI/NanosoldierReports) repository.



#### Trigger Syntax



A `BenchmarkJob` is triggered with the following syntax:



```

@nanosoldier `runbenchmarks(tag_predicate, vs = "ref")`

```



The `vs` keyword argument is optional; if invoked from a pull request, it will be derived

automatically from the merge base. In other cases, the comparison step (step 3 above) will

be skipped.



The tag predicate is used to decide which benchmarks to run, and supports the syntax defined

by the [tagging

system](https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/doc/manual.md#indexing-into-a-benchmarkgroup-using-tagged)

implemented in the [BenchmarkTools](https://github.com/JuliaCI/BenchmarkTools.jl) package.

Additionally, you can run all benchmarks by using the keyword `ALL`, e.g.

`runbenchmarks(ALL)`, which is the same as specifying no predicate at all.



The `vs` keyword argument takes a reference string which can points to a Julia commit to

compare against. The following syntax is supported for reference string:



- `":branch"`: the head commit of the branch named `branch` in the current repository (`JuliaLang/julia`)

- `"@sha"`: the commit specified by `sha` in the current repository (`JuliaLang/julia`)

- `"#tag"`: the commit pointed to by the tag named `tag` in the current repository (`JuliaLang/julia`)

- `"%self"`: to use the same commit for both parts of the comparison

- `"owner/repo:branch"`: the head commit of the branch named `branch` in the repository `owner/repo`

- `"owner/repo@sha"`: the commit specified by `sha` in the repository `owner/repo`

- `"owner/repo#tag"`: the commit pointed to by the tag named `tag` in the repository `owner/repo`



#### Benchmark Results



Once a `BenchmarkJob` is complete, the results are uploaded to the

[NanosoldierReports](https://github.com/JuliaCI/NanosoldierReports) repository. Each job

has its own directory for results. This directory contains the following items:



- `report.md` is a markdown report that summarizes the job results

- `data.tar.gz` contains raw timing data in JSON format. To untar this file, run

`tar -xzvf data.tar.gz`. You can analyze this data using the

[BenchmarkTools](https://github.com/JuliaCI/NanosoldierReports) package.

- `logs` is a directory containing the build logs and benchmark execution logs for the job.



#### Comment Examples



Here are some examples of comments that trigger a `BenchmarkJob` in various contexts:



```

I want to run benchmarks tagged "array" on the current commit.



@nanosoldier `runbenchmarks("array")`



If this comment is on a specific commit, benchmarks will run on that commit. If

it's in a PR, they will run on the head/merge commit of the PR. If it's on a diff,

they will run on the commit associated with the diff.

```



```

I want to run benchmarks tagged "array" on the current commit, and compare the results

with the results of running benchmarks on commit 858dee2b09d6a01cb5a2e4fb2444dd6bed469b7f.



@nanosoldier `runbenchmarks("array", vs = "@858dee2b09d6a01cb5a2e4fb2444dd6bed469b7f")`

```



```

I want to run benchmarks tagged "array", but not "simd" or "linalg", on the

current commit. I want to compare the results against those of the release-0.4

branch.



@nanosoldier `runbenchmarks("array" && !("simd" || "linalg"), vs = ":release-0.4")`

```



```

I want to run all benchmarks on the current commit. I want to compare the results

against a commit on my fork.



@nanosoldier `runbenchmarks(vs = "christopher-dG/julia@c70ab26bb677c92f0d8e0ae41c3035217a4b111f")`

```



```

I want to run all benchmarks on the current commit. I want to compare the results

against the head commit of my fork's branch.



@nanosoldier `runbenchmarks(vs = "christopher-dG/julia:mybranch")`

```



### `PkgEvalJob`



#### Execution Cycle



A `PkgEvalJob` has the following execution cycle:



1. Pull in the JuliaLang/julia repository and build the commit specified by the context of

   the trigger phrase.

2. Using the new Julia build, test the packages from the

   [General](https://github.com/JuliaRegistries/General) registry as specified by the

   trigger phrase.

3. If the trigger phrase specifies a commit to compare against, build that version of Julia

   and perform step 2 using the comparison build.

4. Upload a markdown report to the

   [NanosoldierReports](https://github.com/JuliaCI/NanosoldierReports) repository.



#### Trigger Syntax



A `PkgEvalJob` is triggered with the following syntax:



```

@nanosoldier `runtests(package_selection, vs = "ref")`

```



The package selection argument is used to decide which packages to test. It should be a list

of package names, e.g. `["Example"]`, that will be looked up in the registry. Additionally,

you can test all packages in the registry by using the keyword `ALL`, e.g. `runtests(ALL)`,

which is the same as not providing a package selection argument at all.



The `vs` keyword argument is again optional. Its syntax and behavior is identical to the

`BenchmarkJob` `vs` keyword argument.



Both sides of the comparison can be further configured by using respectively the

`configuration` and `vs_configuration` arguments. These options expect a named tuple where

the elements correspond to fields of the `PkgEval.Configuration` type.



For example, a common configuration is to include buildflags that enable assertions:



```

@nanosoldier `runtests(vs = "%self", configuration = (buildflags=["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"],))`

```



Another useful example makes PkgEval run under rr and use a Julia debug build for a better debugging experience:



```

@nanosoldier `runtests(configuration = (buildflags=["JULIA_BUILD_MODE=debug"], julia_binary="julia-debug", rr=true))`

```



If no configuration arguments are specified, the defaults as specified by the

`PkgEval.Configuration` constructor are used.



#### Reverse-CI for packages



Nanosoldier.jl also supports testing for regression introduced by *package changes*. This

feature is currently only enabled on select repositories (contact @maleadt if you think this

is valuable for your package).



The interface for testing package changes is identical to testing Julia changes: just invoke

Nanosoldier by commenting with an appropriate trigger phrase on a commit, issue or pull

request on a package repository. The execution cycle is slightly different:



- The Julia version will be the same for both sides of the comparison, defaulting to

  `stable` (which can be customized by setting the `julia` argument of the respective

  `configuration`, e.g., to `"1.8"`)

- If no package selection is made, or the set of `ALL` packages is requested, Nanosoldier

  will look up the direct dependents of the package and test those.

- Tests will be run after registering the current state of the package in a temporary

  registry (implying that your `Project.toml` should contain a version bump). The `vs` side

  of the comparison will use an unmodified version of the registry.



## Acknowledgements



The development of the Nanosoldier benchmarking platform was supported in part by the US

Army Research Office through the Institute for Soldier Nanotechnologies under Contract

No. W911NF-07-D0004.