Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/spinnaker/igor

Integration with Jenkins and Git for Spinnaker
https://github.com/spinnaker/igor

hacktoberfest

Last synced: about 24 hours ago
JSON representation

Integration with Jenkins and Git for Spinnaker

Awesome Lists containing this project

README 

        
[![Build Status](https://api.travis-ci.org/spinnaker/igor.svg?branch=master)](https://travis-ci.org/spinnaker/igor)



[![](https://github.com/spinnaker/igor/workflows/Igor%20CI/badge.svg)](https://github.com/spinnaker/igor/actions?query=workflow%3A%22Igor+CI%22+branch%3Amaster)



Igor is a service that provides a single point of integration with Continuous Integration (CI) and Source Control Management (SCM) services for Spinnaker.



# Common Polling Architecture



Igor runs a number of pollers that all share the same common architecture. At a high level, they all:



- periodically get a list of items from an external resource (e.g. builds on a Jenkins master)

- compare that list against their own persisted cache of items (the difference is called _delta size_)

- send an echo event for each new item

- cache the new list of items



Features:



- *health*: igor has a `HealthIndicator` that reports `Down` if no pollers are running or if they have not had a successful polling cycle in a long time

- *locking*: pollers can optionally acquire a distributed lock in the storage system before attempting to complete a polling cycle. This makes it possible to run igor in a high-availability configuration and scale it horizontally.

- *safeguards*: abnormally large delta sizes can indicate a problem (e.g. lost or corrupt cache data) and cause downstream issues. If a polling cycle results in a delta size above the threshold, the new items will not be cached and events will not be submitted to echo to prevent a trigger storm. Manual action will be needed to resolve this, such as using the fast-forward admin endpoint: `/admin/pollers/fastforward/{monitorName}[?partition={partition}]`. Fast-forwarding means that all pending cache state will be polled and saved, but will not send echo notifications.



Relevant properties:



| *Property* | *Default value* | *Description* |

| --- | --- | --- |

| `spinnaker.build.pollingEnabled` | true | Defines whether or not the build system polling mechanism is enabled. Disabling this will effectively disable any integration with a build system that depends on Igor polling it. |

| `spinnaker.build.pollInterval` | `60` | Interval in seconds between polling cycles |

| `spinnaker.pollingSafeguard.itemUpperThreshold` | `1000` | Defines the upper threshold for number of new items before a cache update cycle will be rejected |

| `locking.enabled` | `false` | Enables distributed locking so that igor can run on multiple nodes without interference |



Relevant metrics:



| *Metric* | *Type* | *Description* |

| --- | --- | --- |

| `pollingMonitor.newItems` | gauge | represents the number of new items cached by a given monitor during a polling cycle |

| `pollingMonitor.itemsOverThreshold` | gauge | 0 if deltaSize < threshold, deltaSize otherwise |

| `pollingMonitor.pollTiming` | timer | published for every polling cycle with the duration it took to complete |

| `pollingMonitor.failed` | counter | an error counter indicating a failed polling cycle |



All these metrics can be grouped by a `monitor` tag (e.g. `DockerMonitor`, `JenkinsMonitor`...) to track down issues.



# Storage



The following storage backends are supported:



- Redis



Relevant properties:

```

redis:

  enabled: true

  connection: redis://host:port

```



# Integration with SCM services



The following SCM services are supported:



- Bitbucket

- Github

- Gitlab

- Stash



`Commit` controller classes expose APIs to retrieve lists of commits, such as `/github/{{projectKey}}/{{repositorySlug}}/compareCommits?from={{fromHash}}&to={{toHash}}`



At the moment, igor only exposes read APIs, there are no pollers and no triggers involving SCM services directly.



Relevant properties:



```

github:

  baseUrl: "https://api.github.com"

  accessToken: ''

  commitDisplayLength: 8



stash:

  baseUrl: ""

  username: ''

  password: ''



bitbucket:

  baseUrl: "https://api.bitbucket.org"

  username: ''

  password: ''

  commitDisplayLength: 7



gitlab:

  baseUrl: "https://gitlab.com"

  privateToken: ''

  commitDisplayLength: 8

```



# Integration with CI services



The following CI services are supported:



- Artifactory

- Nexus

- Concourse

- Gitlab CI

- Google Cloud Build (GCB)

- Jenkins

- Travis

- Wercker



For each of these services, a poller can be enabled (e.g. with `jenkins.enabled`) that will start monitoring new builds/pipelines/artifacts, caching them and submitting events to echo, thus supporting pipeline triggers. GCB is a bit different in that it doesn't poll and requires setting up [pubsub subscriptions](https://www.spinnaker.io/setup/ci/gcb/).



The `BuildController` class also exposes APIs for services that support them such as:



- getting build status

- listing builds/jobs on a master

- listing queued builds

- starting and stopping builds/jobs



These APIs are used to provide artifact information for bake stages.



## Configuring Jenkins Masters



In your configuration block (either in igor.yml, igor-local.yml, spinnaker.yml or spinnaker-local.yml), you can define multiple masters blocks by using the list format.



You can obtain a Jenkins API token by navigating to `http://your.jenkins.server/me/configure` (where `me` is your username).



```

jenkins:

  enabled: true

  masters:

    -

      address: "https://spinnaker.cloudbees.com/"

      name: cloudbees

      password: f5e182594586b86687319aa5780ebcc5

      username: spinnakeruser

    -

      address: "http://hostedjenkins.amazon.com"

      name: bluespar

      password: de4f277c81fb2b7033065509ddf31cd3

      username: spindoctor

```



## Configuring Travis Masters



In your configuration block (either in igor.yml, igor-local.yml, spinnaker.yml or spinnaker-local.yml), you can define multiple masters blocks by using the list format.



To authenticate with Travis you use a "Personal access token" on a git user with permissions `read:org, repo, user`. This is added in `settings -> Personal access tokens` on github/github-enterprise.



```

travis:

  enabled: true

  # Travis names are prefixed with travis- inside igor.

  masters:

  - name: ci # This will show as travis-ci inside spinnaker.

    baseUrl: https://travis-ci.com

    address: https://api.travis-ci.com

    githubToken: 6a7729bdba8c4f9abc58b175213d83f072d1d832

  regexes:

  - /Upload https?:\/\/.+\/(.+\.(deb|rpm))/

```



When parsing artifact information from Travis builds, igor uses a default regex

that will match on output from the `jfrog rt`/`art` CLI tool.  Different regexes than the

default may be configured using the `regexes` list.



## Configuring Gitlab CI Masters



In your configuration block (either in igor.yml, igor-local.yml, spinnaker.yml or spinnaker-local.yml), you can define multiple masters blocks by using the list format.



To authenticate with Gitlab CI use a [Personal Access Token](https://docs.gitlab.com/ee/security/token_overview.html#personal-access-tokens) with permissions `read_api`.



```

gitlab-ci:

  enabled: true

  itemUpperThreshold: 1000 # Optional, default 1000.  Determines max new pipeline count before a cache cycle is rejected

  masters:

    - address: "https://git.mycompany.com"

      name: mygitlab

      privateToken: kjsdf023ofku209823

      # Optional:

      defaultHttpPageLength: 100 # defaults 100, page length when querying paginated Gitlab API endpoints (100 is max per Gitlab docs)

      limitByOwnership: false # defaults false, limits API results to projects/groups owned by the token creator

      limitByMembership: true # defaults true, limits API results to projects/groups the token creator is a member in

      httpRetryMaxAttempts: 5 # defaults 5, # default max number of retries when hitting Gitlab APIs and errors occur

      httpRetryWaitSeconds: 2 # defaults 2, # of seconds to wait between retries

      httpRetryExponentialBackoff: false # deafults false, if true retries to Gitlab will increase exponentially using the httpRetryWaitSeconds option's value

```



Build properties are automatically read from successful Gitlab CI Pipelines using the pattern `SPINNAKER_PROPERTY_*=value`.  For example a log containing a line

`SPINNAKER_PROPERTY_HELLO=world` will create a build property item `hello=world`.  Gitlab CI artifacts are not yet supported.



## Integration with Docker Registry



Clouddriver can be [configured to poll your registries](http://www.spinnaker.io/v1.0/docs/target-deployment-configuration#section-docker-registry). When that is the case, igor can then create a poller that will list the registries indexed by clouddriver, check each one for new images and submit events to echo (hence allowing Docker triggers)



Relevant properties:



- `dockerRegistry.enabled`

- requires `services.clouddriver.baseUrl` to be configured



# Running igor



Igor requires redis server to be up and running.



Start igor via `./gradlew bootRun`. Or by following the instructions using the [Spinnaker installation scripts](https://www.github.com/spinnaker/spinnaker).



## Debugging



To start the JVM in debug mode, set the Java system property `DEBUG=true`:

```

./gradlew -DDEBUG=true

```



The JVM will then listen for a debugger to be attached on port 8188.  The JVM will _not_ wait for the debugger to be attached before starting igor; the relevant JVM arguments can be seen and modified as needed in `build.gradle`.