{"id":13990534,"url":"https://github.com/MeltwaterArchive/lighter","last_synced_at":"2025-07-22T12:33:03.091Z","repository":{"id":145437201,"uuid":"44743426","full_name":"MeltwaterArchive/lighter","owner":"MeltwaterArchive","description":"DEPRECATED Marathon deployment automation tool","archived":true,"fork":false,"pushed_at":"2019-06-11T07:55:17.000Z","size":314,"stargazers_count":42,"open_issues_count":0,"forks_count":5,"subscribers_count":177,"default_branch":"master","last_synced_at":"2025-06-10T08:52:30.076Z","etag":null,"topics":["archived","automatic-deployment","continuous-deployment","deployment","deprecated","infrastructure-as-code","marathon","mesos","obsolete"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MeltwaterArchive.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2015-10-22T12:24:08.000Z","updated_at":"2024-06-18T17:08:39.000Z","dependencies_parsed_at":null,"dependency_job_id":"b3bf7fe4-0a69-427f-bb09-febea9b0da2b","html_url":"https://github.com/MeltwaterArchive/lighter","commit_stats":null,"previous_names":["meltwaterarchive/lighter","meltwater/lighter"],"tags_count":43,"template":false,"template_full_name":null,"purl":"pkg:github/MeltwaterArchive/lighter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeltwaterArchive%2Flighter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeltwaterArchive%2Flighter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeltwaterArchive%2Flighter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeltwaterArchive%2Flighter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MeltwaterArchive","download_url":"https://codeload.github.com/MeltwaterArchive/lighter/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeltwaterArchive%2Flighter/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266496336,"owners_count":23938710,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archived","automatic-deployment","continuous-deployment","deployment","deprecated","infrastructure-as-code","marathon","mesos","obsolete"],"created_at":"2024-08-09T13:02:53.476Z","updated_at":"2025-07-22T12:33:02.788Z","avatar_url":"https://github.com/MeltwaterArchive.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# DEPRECATED\nThis repository is no longer activiely maintained.\n\n---\n\n# ![Lighter](docs/logo.png)\n[![Travis CI](https://img.shields.io/travis/meltwater/lighter/master.svg)](https://travis-ci.org/meltwater/lighter)\n[![Coverage Status](https://codecov.io/github/meltwater/lighter/coverage.svg?branch=master)](https://codecov.io/github/meltwater/lighter?branch=master\u0026view=all)\n\n[Lighter](https://en.wikipedia.org/wiki/Lighter_(barge)) solves the problem of automating\ndeployments to [Marathon](https://github.com/mesosphere/marathon) and handling of differences\nbetween multiple environments. Given a hierachy of yaml files and environments, Lighter can\nexpand service config files and deploy them to Marathon.\n\nFor even tighter integration into the development process, Lighter can resolve Marathon config files\nfrom Maven and merge these with environment specific overrides. This enables continuous deployment\nwhenever new releases or snapshots appear in the Maven repository. Optional version range constraints\nallows patches/minor versions to be rolled out continuously, while requiring a config change to roll\nout major versions.\n\n## Usage\n```\nusage: lighter COMMAND [OPTIONS]...\n\nMarathon deployment tool\n\npositional arguments:\n  {deploy,verify}       Available commands\n    deploy              Deploy services to Marathon\n    verify              Verify and generate Marathon configuration files\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -n, --noop            Execute dry-run without modifying Marathon [default:\n                        False]\n  -v, --verbose         Increase logging verbosity [default: False]\n  -t TARGETDIR, --targetdir TARGETDIR\n                        Directory to output rendered config files\n  -p PROFILES, --profile PROFILES\n                        Extra profile file(s) to be merged with service\n                        definitions.\n```\n\n### Deploy Command\n```\nusage: lighter deploy [OPTIONS]... YMLFILE...\n\nDeploy services to Marathon\n\npositional arguments:\n  YMLFILE               Service files to expand and deploy\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -m MARATHON, --marathon MARATHON\n                        Marathon URL like \"http://marathon-host:8080/\".\n                        Overrides default Marathon URL's provided in config\n                        files\n  -f, --force           Force deployment even if the service is already\n                        affected by a running deployment [default: False]\n  --canary-group CANARYGROUP\n                        Unique name for this group of canaries [default: None]\n  --canary-cleanup      Destroy canaries that are no longer present [default:\n                        False]\n```\n\n### Verify Command\n```\nusage: lighter verify YMLFILE...\n\nVerify and generate Marathon configuration files\n\npositional arguments:\n  YMLFILE           Service files to expand and deploy\n\noptional arguments:\n  -h, --help        show this help message and exit\n  --verify-secrets  Fail verification if unencrypted secrets are found\n                    [default: False]\n```\n\n## Configuration\nGiven a directory structure like\n\n```\nconfig-repo/\n|   globals.yml\n|   myprofile.yml\n└─ production/\n|   |   globals.yml\n|   |   myfrontend.yml\n|   └─ mysubsystem/\n|          globals.yml\n|          myservice-api.yml\n|          myservice-database.yml\n└─ staging/\n    |   globals.yml\n    |   myfrontend.yml\n```\n\nRunning `lighter deploy -p myprofile1.yml -p myprofile2.yml staging/myfrontend.yml` will\n\n* Merge *myfrontend.yml* with environment defaults from *config-repo/staging/globals.yml*, *config-repo/globals.yml*, *myprofile1.yml* and *myprofile2.yml*\n* Fetch the *json* template for this service and version from the Maven repository\n* Expand the *json* template with variables and overrides from the *yml* files\n* Post the resulting *json* configuration into Marathon\n\n## Marathon\nYaml files may contain a `marathon:` section with a default URL to reach Marathon at. The `-m/--marathon`\nparameter will override this setting when given on the command-line.\n\n*globals.yml*\n```\nmarathon:\n  url: 'http://marathon-host:8080/'\n```\n\n## Maven\nThe `maven:` section specifies where to fetch *json* templates from which are\nmerged into the configuration. For example\n\n*globals.yml*\n```\nmaven:\n  repository: \"http://username:password@maven.example.com/nexus/content/groups/public\"\n```\n\n*myservice.yml*\n```\nmaven:\n  groupid: 'com.example'\n  artifactid: 'myservice'\n  version: '1.0.0'\n  classifier: 'marathon'\n```\n\nThe Maven 'classifier' tag is optional.\n\n### Dynamic Versions\nVersions can be dynamically resolved from Maven using a range syntax.\n\n```\nmaven:\n  groupid: 'com.example'\n  artifactid: 'myservice'\n  version: '[1.0.0,2.0.0)'\n```\n\nFor example\n\nExpression | Resolve To\n:----------|:-----------\n[1.0.0,2.0.0) | 1.0.0 up to but not including 2.0.0\n[1.0.0,1.2.0] | 1.0.0 up to and including 1.2.0\n[1.0.0,2.0.0)-featurebranch | 1.0.0 up to and including 1.2.0, only matches *featurebranch* releases\n[1.0.0,1.2.0]-SNAPSHOT | 1.0.0 up to and including 1.2.0, only matches *SNAPSHOT* versions\n[1.0.0,2.0.0]-featurebranch-SNAPSHOT | 1.0.0 up to and including 1.2.0, only matches *featurebranch-SNAPSHOT* versions\n[1.0.0,] | 1.0.0 or greater\n(1.0.0,] | Greater than 1.0.0\n[,] | Latest release version\n[,]-SNAPSHOT | Latest *SNAPSHOT* version\n\n## Freestyle Services\nYaml files may contain a `service:` tag which specifies a Marathon *json* fragment\nto use as the service configuration base for further merging. This allows for\nservices which aren't based on a *json* template but rather defined exclusively\nin *yaml*.\n\n*myservice.yml*\n```\nservice:\n  id: '/myproduct/myservice'\n  container:\n    docker:\n      image: 'meltwater/myservice:latest'\n  env:\n    DATABASE: 'database:3306'\n  cpus: 1.0\n  mem: 1200\n  instances: 1\n```\n\n## Overrides\nYaml files may contain an `override:` section that will be merged directly into the Marathon json. The\nstructure contained in the `override:` section must correspond to the [Marathon REST API](https://mesosphere.github.io/marathon/docs/rest-api.html#post-v2-apps). For example\n\n```\noverride:\n  instances: 4\n  cpus: 2.0\n  env:\n    LOGLEVEL: 'info'\n    NEW_RELIC_APP_NAME: 'MyService Staging'\n    NEW_RELIC_LICENSE_KEY: '123abc'\n```\n\n### Deep Merge\nAn YAML and JSON file upwards-recursive deep merge is performed when parsing service definitions. Precedence is defined by the directory structure\n\n* myservice.yml has the highest precedence\n* globals.yml files are merged with decreasing precedency upwards in the directory structure\n* myservice-1.0.0-marathon.json if fetched from Maven has the lowest precedence\n\nLists, dicts and scalar values are deep merged\n\n * Dicts are deep merged, the result containing the union of all keys\n * Lists are appended together\n * Scalar values coalesce to the not-null value with highest precedence\n\nThe default behaviour is to append lists together, however specific list items can be overriden and deep merged using a dict with integer keys. For example\n\n*myservice-1.0.0-marathon.json*\n```\n{\n    \"container\": {\n        \"docker\": {\n            \"portMappings\": [\n                {\"containerPort\": 8080, \"servicePort\": 1234},\n                {\"containerPort\": 8081, \"servicePort\": 1235}\n            ]\n        }\n    }\n}\n```\n\n*myservice-override-serviceport.yml*\n```\noverride:\n  container:\n    docker:\n      portMappings:\n        # Override service ports 1234,1235 with port 4000,4001\n        0:\n          servicePort: 4000\n        1:\n          servicePort: 4001\n```\n\n#### Non-string Environment Variables\nBooleans, integers and floats in the `env` section are converted to strings before being posted\nto Marathon. Non-scalar environment variables like dicts and lists are deep merged and automatically\nserialized to json strings.\n\n*myservice.yml*\n```\noverride:\n  env:\n    intvar: 123\n    boolvar: TRUE\n    dictvar:\n      mykey:\n       - 1\n       - 'abc'\n```\n\nWould result in a rendered json like\n```\n{\n  \"env\": {\n    \"intvar\": \"123\",\n    \"boolvar\": \"true\",\n    \"dictvar\": \"{\\\"mykey\\\": [1, \\\"abc\\\"]}\"\n  }\n}\n```\n\n## Variables\nYaml files may contain an `variables:` section containing key/value pairs that will be substituted into the *json* template. All\nvariables in a templates must be resolved or it's considered an error. This can be used to ensure that some parameters are\nguaranteed to be provided to a service. For example\n```\nvariables:\n  docker.registry: 'docker.example.com'\n  rabbitmq.host: 'rabbitmq-hostname'\n```\n\nAnd used from the *json* template like\n```\n{\n    \"id\": \"/myproduct/myservice\",\n    \"container\": {\n        \"docker\": {\n            \"image\": \"%{docker.registry}/myservice:1.2.3\"\n        }\n    },\n    \"env\": {\n        \"rabbitmq.url\": \"amqp://guest:guest@%{rabbitmq.host}:5672\"\n    }\n}\n```\n\nLighter also allows specifying environment variables as values in the configuration yaml files.\n\nWith the following configuration:\n*myservice.yml*\n```\nservice:\n  id: '/myproduct/myservice'\n  container:\n    docker:\n      image: 'meltwater/myservice:%{env.VERSION}'\n  env:\n    DATABASE: 'database:3306'\n  cpus: 1.0\n  mem: 1200\n  instances: 1\n```\n\nAnd Running `VERSION=1.1.1 lighter deploy myservice.yml`, lighter will deploy the docker image ``meltwater/myservice:1.1.1`` to marathon.\n\n\nTo avoid interpolating some string like ``%{id}`` when you really want it, use ``%%{id}``\n\n### Predefined Variables\n\nVariable | Contains\n:--------|:--------\n%{lighter.version} | Maven artifact version or Docker image version\n%{lighter.uniqueVersion} | Unique build version resolved from Maven/Docker metadata\n\n## Snapshot Builds\nIf an image is rebuilt with the same Docker tag, Marathon won't detect a change and hence won't roll out the new\nimage. To ensure that new snapshot/latest versions are deployed use `%{lighter.uniqueVersion}` and `forcePullImage`\nlike this\n\n*myservice.yml*\n```\noverride:\n  container:\n    docker:\n      forcePullImage: true\n  env:\n    SERVICE_BUILD: '%{lighter.uniqueVersion}'\n```\n\n### Docker Registry\nLighter calls the Docker Registry API to resolve `%{lighter.uniqueVersion}` when it's used\nin a non-Maven based service. This is only enabled if the `%{lighter.uniqueVersion}` variable\nis actually referenced from the service config.\n\nFor authenticated reprositories you must supply read-access credentials to be used when calling\nthe registry API. You can find the base64 encoded credentials in your *~/.docker/config.json* or\n*~/.dockercfg* files. Note that Docker Hub is not supported at this time.\n\n*globals.yml*\n```\ndocker:\n  registries:\n    'registry.example.com':\n      auth: 'dXNlcm5hbWU6cGFzc3dvcmQ='\n```\n\n## Facts\nYaml files may contain a `facts:` section with information about the service surroundings\n\n*staging/globals.yml*\n```\nfacts:\n  environment: 'staging'\n```\n\n## Secrets Management\nLighter has support for [Secretary](https://github.com/meltwater/secretary) which can\nsecurely distribute secrets to containers.\n\n*someenv/globals.yml *\n```\nsecretary:\n  url: 'https://secretary-daemon-loadbalancer:5070'\n  master:\n    publickey: 'someenv/keys/master-public-key.pem'\n```\n\n*someenv/myservice.yml*\n```\noverride:\n  env:\n    DATABASE_PASSWORD: \"ENC[NACL,NVnSkhxA010D2yOWKRFog0jpUvHQzmkmKKHmqAbHAnz8oGbPEFkDfyKHQHGO7w==]\"\n```\n\n**Note**\n\nMake sure the environment variable name is a valid shell script identifier and supported by [Secretary](https://github.com/meltwater/secretary), only alphanumeric characters and underscores are supported, starting with an alphabetic or underscore character. e.g `DATABASE.PASSWORD` is invalid but `DATABASE_PASSWORD` is valid.\n\n## Canary Deployments\nLighter together with [Proxymatic](http://github.com/meltwater/proxymatic) supports [canary deployments](http://martinfowler.com/bliki/CanaryRelease.html) using\nthe `--canary-group` parameter. This parameter makes Lighter rewrite the app id and servicePort to avoid conflicts and automatically add\nthe metadata labels that Proxymatic use for canaries. The `--canary-cleanup` parameter destroys canary instances when they are removed\nfrom configuration.\n\nTake care that the `--canary-group` parameter is unique to the deployment job and branch that executes the canary deployment. Lighter will clean out canaries with the same group name if they aren't being generated anymore, and if multiple deployment jobs share a group name they'd conflict and destroy each others canaries.\n\n### Canaries From Files\nThis example use a `*-canary-*` filename convention to separate canaries from normal services. In this workflow\nyou would copy the regular service file `myservice.yml`, and make any tentative changes in this new\n`myservice-canary-somechange.yml`. When the canary has served its purpose you'd `git mv` back or `git rm` the\ncanary file. You can run multiple independent canaries in parallel, e.g. `myservice-canary-somechange.yml`, `myservice-canary-foo.yml` and `myservice-canary-bar.yml` can co-exist at the same time\n\n```\n# Deploy regular services\nlighter deploy -f -m \"http://marathon-host:8080/\" $(find . -name \\*.yml -not -name globals.yml -not -name \\*-canary-\\*)\n\n# Deploy and prune canaries\nlighter deploy -f -m \"http://marathon-host:8080/\" --canary-group=generic --canary-cleanup $(find . -name \\*-canary-\\*.yml)\n```\n\n### Canaries From Pull Requests\nThis usage would run `lighter -t /some/output/dir verify ...` on a PR and again on its base revision. Then `diff -r` the\nrendered json files to figure out what services were modifed in the PR. The modified services would be deployed as canaries\nwith `lighter deploy --canary-group=mybranchname --canary-cleanup ...` whenever the PR branch is changed. When the PR is closed\nor merged the canaries would be destroyed using `lighter deploy --canary-group=mybranchname --canary-cleanup`\n\n### Canary Metrics\nLighter adds a [Docker label](https://docs.docker.com/engine/userguide/labels-custom-metadata/) `com.meltwater.lighter.canary.group`\nwhich can be used to separate out container metrics from the canaries.\n\n## Installation\nPlace a `lighter` script in the root of your configuration repo. Replace the LIGHTER_VERSION with\na version from the [releases page](https://github.com/meltwater/lighter/releases).\n\n```\n#!/bin/bash\nset -e\n\nLIGHTER_VERSION=\"x.y.z\"\n\nBASEDIR=\"$( cd \"$( dirname \"${BASH_SOURCE[0]}\" )\" \u0026\u0026 pwd )\"\nLIGHTER=\"$BASEDIR/target/lighter-`uname -s`-`uname -m`-${LIGHTER_VERSION}\"\n\nif [ ! -x \"$LIGHTER\" ]; then\n    mkdir -p $(dirname \"$LIGHTER\")\n    curl -sfLo \"$LIGHTER\" https://github.com/meltwater/lighter/releases/download/${LIGHTER_VERSION}/lighter-`uname -s`-`uname -m`\n    chmod +x \"$LIGHTER\"\nfi\n\n# Ligher will write the expanded json files to /tmp/output\nexec \"$LIGHTER\" -t \"`dirname $0`/target\" $@\n```\n\nUse the script like\n```\ncd my-config-repo\n\n# Deploy/sync all services (from Jenkins or other CI/CD server)\n./lighter deploy $(find staging -name \\*.yml -not -name globals.yml)\n\n# Deploy single services\n./lighter deploy staging/myservice.yml staging/myservice2.yml\n```\n\n## Integrations\nLighter can push deployment notifications to a number of services.\n\n### HipChat\nYaml files may contain an `hipchat:` section that specifies where to announce deployments. Create a [HipChat V2 token](https://www.hipchat.com/docs/apiv2) that is allowed to post to rooms. The numeric room ID can be found in the room preferences in the HipChat web interface.\n\n```\nhipchat:\n  token: '123abc'\n  rooms:\n    - '123456'\n```\n\nThe default message of the hipchat notification is :\n\n```\nDeployed /my-image-id with image example-registry.io/image:1.0.0 to staging (marathon-url.example.com).\n```\nIf you would like to post releases notes in addition to the above message, you have 2 options :\n\n1. You can add the following block to your service config \n\n```\nhipchat:\n  releaseNotes: \"my amazing release notes\"\n```\n\n2. You can add the release notes with a label on the docker image itself  and then indicate which label lighter should use to get the release notes :\n\ndockerfile\n```\nFROM registry.io/image:tag\nLABEL com.example.component.release-notes=\"my amazing release notes\"\n```\n\nservice config \n```\nhipchat:\n  message.image.label: \"com.example.component.release-notes\" # image label to use for hipchat message\n\n```\n\n### Slack\nYaml files may contain an `slack:` section that specifies where to announce deployments. Create a [Slack App Oauth token](https://api.slack.com/apps) that is allowed to post to channels with `chat:write:user`, `chat:write:bot` scopes.\n\n```\nslack:\n  token: 'xoxb-1234-56789abcdefghijklmnop'\n  channels:\n    - '#channel'\n```\n\nThe default message of the slack notification is :\n\n```\nDeployed /my-image-id with image example-registry.io/image:1.0.0 to staging (marathon-url.example.com).\n```\nIf you would like to post releases notes in addition to the default message, you have 2 options :\n\n1. You can add the following block to your service config \n\n```\nslack:\n  releaseNotes: \"my amazing release notes\"\n```\n\n2. You can add the release notes with a label on the docker image itself  and then indicate which label lighter should use to get the release notes :\n\ndockerfile\n```\nFROM registry.io/image:tag\nLABEL com.example.component.release-notes=\"my amazing release notes\"\n```\n\nservice config \n```\nslack:\n  message.image.label: \"com.example.component.release-notes\" # image label to use for slack message\n\n```\n\n### New Relic\nTo send [New Relic deployment notifications](https://docs.newrelic.com/docs/apm/new-relic-apm/maintenance/deployment-notifications) supply your [New Relic REST API key](https://docs.newrelic.com/docs/apis/rest-api-v2/requirements/api-keys) (different from the license key given to the agent).\n\n*globals.yml*\n```\nnewrelic:\n  token: '123abc'\n```\n\n*myservice.yml*\n```\noverride:\n  env:\n    NEW_RELIC_LICENSE_KEY: 'abc123'\n    NEW_RELIC_APP_NAME: 'MyService'\n```\n\n### Datadog\nTo send [Datadog deployment events](http://docs.datadoghq.com/guides/overview/#events) supply\nyour [Datadog API key](https://app.datadoghq.com/account/settings#api). Lighter will add Marathon\nappid and canary group as Docker container labels in order for Datadog to tag collected metrics,\n[see: collect_labels_as_tags](https://github.com/DataDog/dd-agent/blob/master/conf.d/docker_daemon.yaml.example).\n\n\n*globals.yml*\n```\ndatadog:\n  token: '123abc'\n  tags:\n    - subsystem:example\n```\n\n*Datadog Puppet Config*\n```\ndatadog::docker:\n  docker_daemon:\n    instances:\n      - url: \"unix://var/run/docker.sock\"\n        new_tag_names: true\n        collect_labels_as_tags: [\"com.meltwater.lighter.appid\", \"com.meltwater.lighter.canary.group\"]\n```\n\n### Graphite\nTo send [Graphite deployment events](http://docs.grafana.org/reference/annotations/) supply your Graphite plaintext and HTTP endpoints.\n\n*globals.yml*\n```\ngraphite:\n  address: 'graphite-host:2003'\n  url: 'http://graphite-host:80/'\n  prefix: 'lighter'\n  tags:\n    - subsystem:example\n```\n\n## Contributors\n\n* **[Giuliano Manno](https://github.com/xyden)**\n\n  * Lighter logo\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMeltwaterArchive%2Flighter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMeltwaterArchive%2Flighter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMeltwaterArchive%2Flighter/lists"}