{"id":18045326,"url":"https://github.com/dpb587/cfsummit-bosh-jq","last_synced_at":"2026-01-18T00:20:40.838Z","repository":{"id":138804063,"uuid":"59781122","full_name":"dpb587/cfsummit-bosh-jq","owner":"dpb587","description":null,"archived":false,"fork":false,"pushed_at":"2016-05-30T05:53:27.000Z","size":21,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-10T11:48:33.764Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JSONiq","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dpb587.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-05-26T20:26:20.000Z","updated_at":"2018-08-10T01:16:41.000Z","dependencies_parsed_at":null,"dependency_job_id":"7111001d-bf26-4d24-96e8-369439b9d84e","html_url":"https://github.com/dpb587/cfsummit-bosh-jq","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpb587%2Fcfsummit-bosh-jq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpb587%2Fcfsummit-bosh-jq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpb587%2Fcfsummit-bosh-jq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpb587%2Fcfsummit-bosh-jq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dpb587","download_url":"https://codeload.github.com/dpb587/cfsummit-bosh-jq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247284914,"owners_count":20913691,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-30T18:12:55.476Z","updated_at":"2026-01-18T00:20:40.800Z","avatar_url":"https://github.com/dpb587.png","language":"JSONiq","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cfsummit-bosh-jq\n\nAn example of using [`jq`](https://stedolan.github.io/jq/) to help abstract and compose manifests for BOSH and letting Concourse deploy changes. This was presented with [slides](https://docs.google.com/presentation/d/1PiiO6ckMY5xZDJHcpu4JjpOMZu_F04D8Sd-krVv1rG0/view) as a Lightning Talk at CF Summit on May 26, 2016. It was based on my earlier blog post, [Composing Configurations with JQ](https://dpb587.me/blog/2016/04/26/composing-configurations-with-jq.html). See the slides, read the post, or ask me for a walkthrough.\n\nCaveat emptor...\n\n * this repo is for demo purposes only; the scripts here are the ones based off my actual environments using jq\n * it spins up actual instances you will be charged for\n * it's not well secured; deployments are only limited to your IPs\n * I don't know CF release; it deploys and you can deploy an app, but something's still misconfigured; based on [minimal-aws.yml](https://github.com/cloudfoundry/cf-release/blob/master/example_manifests/minimal-aws.yml)\n * it's a demo repository... proof of concept... spike\n * it doesn't have a one-click tear down; deploy it to a test region, then you can just select all EC2 instances and terminate them, then delete the 3 CloudFormation stacks this makes\n\nPoints of interest...\n\n * [`ci/bin/render`](ci/bin/render) - takes your JQ-manifest, loads the *-stack files and ./config/* files as arguments, renders and outputs your manifest\n * [`ci/tasks/bosh/bosh-init-render.yml`](ci/tasks/bosh/bosh-init-render.yml) - tasks for rendering your manifests; you're forced to explicitly list what the manifests depend on so bots know what to do\n * [`ci/pipeline.jq`](ci/pipeline.jq) - the pipeline; kind of messy due to odd dependencies, but in theory this could be generated from task configs and conventions\n * [`bosh/cloudformation.jq`](bosh/cloudformation.jq) - a JQ-friendly AWS CloudFormation which references the original infrastructure stack (VPC ID)\n * [`bosh/bosh-init.jq`](bosh/bosh-init.jq) - example of referencing certificates, file-based credentials, and CloudFormation stack results\n * [`bosh/generate-config`](bosh/generate-config) - a deployment knowing how to generate its own \"secure\" default settings\n * jq can reference environment variables - rely on default AWS environment variables rather than user needing to hard-code credentials to files\n \n\n## Demo\n\n**Prerequisites**:\n\n * `git`, `fly`, `jq`, `openssl`, `cf`, `go`\n * Concourse server which deploys everything for you (try [concourse-lite](https://github.com/concourse/concourse-lite))\n * A private git repository/branch and private key for your CI to access it - bosh-init will update its state file and commit/push it to the repo\n * AWS Route 53 domain for testing CF\n * AWS environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_DEFAULT_REGION`) - these are only used by concourse for bosh-init and stack management; bosh-init provisions with these credentials, but BOSH director will use an IAM role instead of keys\n * AWS SSH key pair for you and your CI to access VMs - bosh-init uses this key while provisioning BOSH director\n\nClone this repo and create your own git repository/branch from it...\n\n    $ mkdir cfsummit-bosh-jq\n    $ wget -qO- https://github.com/dpb587/cfsummit-bosh-jq-demo/archive/master.tgz | tar -xzf- --strip-components 1\n    $ git init . \u0026\u0026 git add -A . \u0026\u0026 git commit -m 'new environment'\n\nSymlink to your git repository and AWS keys and configure the AWS keypair name...\n\n    $ ln -s ~/.ssh/github_rsa ci/config/repo.pem\n    $ ln -s ~/.ssh/aws_rsa ci/config/aws.pem\n    $ ci/bin/jsonconfig set env.json ssh_key_name \"{{example = default}}\"\n\nConfigure which repository/branch you're using...\n\n    $ ci/bin/jsonconfig set env.json repo_uri \"{{example = git@github.com:whoami/myenv.git}}\"\n    $ git remote add origin \"$( ci/bin/jsonconfig get env.json repo_uri )\"\n\nConfigure the Route 53 domain and hostname to use for the CF deployment...\n\n    $ ci/bin/jsonconfig set cf/config/settings.json fqdn_zone \"{{example = example.com}}\"\n    $ ci/bin/jsonconfig set cf/config/settings.json system_domain \"{{example = cfsummit-bosh-jq.example.com}}\"\n\nRegenerate random passwords and keys for your environment, then commit and push...\n\n    $ ./ci/bin/generate-config\n    $ git add -A . \u0026\u0026 git commit -m 'customize environment' \u0026\u0026 git push\n\nNow deploy the pipeline to Concourse (this will actually start creating resources and AWS will charge you money!)...\n\n    $ ./ci/bin/deploy $FLY_TARGET\n\nOnce the pipeline has finished... you can deploy something to it...\n\n    # target, login, prepare\n    $ cf target --skip-ssl-validation api.$( ci/bin/jsonconfig get cf/config/settings.json system_domain )\n    $ cf login -u admin -p PASSWORD\n    $ cf create-space test\n    $ cf target -s test\n    \n    # use the sample dora app\n    $ git clone git clone https://github.com/cloudfoundry/cf-acceptance-tests.git\n    $ cd cf-acceptance-tests/assets/dora\n    $ cf push doratest\n    \n    # see it\n    $ open http://doratest.$( ci/bin/jsonconfig get cf/config/settings.json system_domain )\n\n\n## Links\n\n * [jq](https://stedolan.github.io/jq/)\n * [aws-cloudformation-stack-resource](https://github.com/dpb587/aws-cloudformation-stack-resource) (soon to be merged back to [pivotal-cf-experimental/cloudformation-resource](https://github.com/pivotal-cf-experimental/cloudformation-resource))\n * [Concourse](http://concourse.ci/)\n \n\n## License\n\n[MIT License](./LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpb587%2Fcfsummit-bosh-jq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdpb587%2Fcfsummit-bosh-jq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpb587%2Fcfsummit-bosh-jq/lists"}