{"id":19959831,"url":"https://github.com/joaohf/buildbot-ci","last_synced_at":"2026-05-01T23:32:41.255Z","repository":{"id":137546493,"uuid":"242559718","full_name":"joaohf/buildbot-ci","owner":"joaohf","description":"How to build a CI using buildbot","archived":false,"fork":false,"pushed_at":"2020-05-03T14:20:49.000Z","size":17,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-01T16:24:06.597Z","etag":null,"topics":["ansible","aws","buildbot","ci","packer","terraform"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joaohf.png","metadata":{"files":{"readme":"README.adoc","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-23T17:25:30.000Z","updated_at":"2020-05-30T16:10:16.000Z","dependencies_parsed_at":"2024-04-03T20:00:22.680Z","dependency_job_id":null,"html_url":"https://github.com/joaohf/buildbot-ci","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/joaohf/buildbot-ci","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joaohf%2Fbuildbot-ci","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joaohf%2Fbuildbot-ci/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joaohf%2Fbuildbot-ci/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joaohf%2Fbuildbot-ci/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joaohf","download_url":"https://codeload.github.com/joaohf/buildbot-ci/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joaohf%2Fbuildbot-ci/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32517210,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"online","status_checked_at":"2026-05-01T02:00:05.856Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","aws","buildbot","ci","packer","terraform"],"created_at":"2024-11-13T01:51:43.043Z","updated_at":"2026-05-01T23:32:41.239Z","avatar_url":"https://github.com/joaohf.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"= buildbot-ci\nifdef::env-github[]\n//Admonitions\n:tip-caption: :bulb:\n:note-caption: :information_source:\n:important-caption: :heavy_exclamation_mark:\n:caution-caption: :fire:\n:warning-caption: :warning:\n:linkattrs:\n:imagesdir: https://github.com/joaohf/buildbot-ci/images\nendif::[]\nifndef::env-github[]\n:imagesdir: ./images\nendif::[]\n:userhome: ~/\n:repohome: buildbot-ci\n:ansible: Ansible\n:ansible-home: https://www.ansible.com/\n:tf: Terraform\n:tf-home: https://www.terraform.io/\n:packer: Packer\n:packer-home: https://packer.io/\n:awscli: AWS Command Line Interface\n:awscli-home: https://aws.amazon.com/cli/\n:aws: AWS\n:bbm: buildbot master\n:bbw: buildbot worker\n:github: Github\n// Asciidoctor Front Matter\n:doctype: book\n:sectlinks:\n:icons: font\n:experimental:\n:sourcedir: .\n:imagesdir: images\n:toc:\n:toc-placement!:\n\nSteps needed in order to build a Continuous Integration on cloud as well all the configuration specific configuration.\n\n[WARNING]\n====\nThe code and instructions fits to my purposes and requirements.\n\nHowever this repository could be useful if you are looking for how to do things using the tools that I used.\n====\n\ntoc::[]\n\n== Motivation\n\nA long time ago I heard about https://buildbot.net/index.html#basics[buildbot project] as a framework to build https://en.wikipedia.org/wiki/Continuous_integration[Continuous Integration]. In that time I thought that it could be a waste of time because there are many powerful and friendly tools (like Jenkins, Bamboo, Gitlab) as well SaaS solutions like travisCI, circleCI. So why anyone would like to spend time configuring a buildbot project ?\n\nThe quick answer is: flexibility and resource control when you need to build projects which use high CPU and disk resources.\n\nAfter a while trying to use https://en.wikipedia.org/wiki/Software_as_a_service[SaaS] tools to get work with a huge time consuming build, I gave up and change my mind to give a try and test buildbot.\n\n[NOTE]\n====\nCI SaaS tools are great. I use to use in my other projects. But for this particular case those tools don't fit.\n====\n\nTo sum up, my aim is to provide a guide about how to use buildbot and tools that helped me when I was building the environment. There are many details captured by {packer}, {tf} and {ansible} configuration files and worth reading them in case of trying this out.\n\n== Tools\n\n{ansible-home}[{ansible}] :: is the tool to configure buildbot master server and any additional service needed. Like applying github webhook integration\n{tf-home}[{tf}] :: creates and destroys cloud infrastructure. The tool allows describing the cloud environment and versioning it using any source code control system\n{packer-home}[{packer}] :: creates machine images installing and configuring based on a template that can be easily reused\n{awscli-home}[{awscli}] :: AWS default command line tool\n\nAlso three additional playbooks do all the buildbot configuration steps:\n\nhttps://github.com/joaohf/ansible-role-buildbot[ansible-role-buildbot] :: a multi OS role that configures and installs buildbot master\nhttps://github.com/joaohf/ansible-role-buildbot_worker[ansible-role-worker] :: also a multi OS role that does all the steps need to configure a buildbot worker\nhttps://github.com/joaohf/ansible-awscli[ansible-role-awscli] :: a basic role install and setup {awscli} \n\n== Use case\n\nThe main use case that this project solves is as follow:\n\n. A _developer_ pushes code or open a Pull Request on {github} repository\n. {github} calls the {bbm} webhook, previously configured\n.. {bbm} receives all the context to initiate a new worker\n.. a new latent {bbw} is spawned to handle the specific build context\n.. after finish the worker, the {bbm} sends the status and pull request back to {github}\n.. the latent {bbw} is finished after 10 minutes of inactivity\n. the _developer_ can see the build results on {github} or buildbot web view.\n\n[NOTE]\n====\nMaybe you are thinking that it is hard to buy the idea. Any CI SaaS does the same tasks.\n\nSure and I agree. However, think about the possibilites to enhance your build CI using AWS services for example. E.g.: cache layers, increasing machine power, coordinate large scale tests.\n====\n\n== Configuration\n\nThis repository has some steps in order to prepare the tools to work together.\n\nAnsible external roles need to be installed ::\n[source,bash]\n----\nansible-galaxy install -r ansible/requirements.yml\n----\n\nCreates ssh key pair ::\n[source,bash]\n----\nssh-keygen -t rsa -b 4096 -C \"buildbot\" -f ~/.ssh/buildbot.key\n----\n\nAdd buildbot key to keychain ::\n[source,bash]\n----\nssh-add ~/.ssh/buildboot.key\n----\n\nSetup AWS CLI :: the https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration[Quickly Configuring the AWS CLI] is a good guide about how to get start with AWS CLI.\n\nCreate the {ansible} vault file password ::\nThe file _{repohome}/ansible/.vault_ should be create with a valid password to decrypt sensitive variables from ansible\n+\n[source,bash]\n----\necho \"MY PASS\" \u003e ansible/.vault\n----\n\n\n== Steps\n\n[NOTE]\n====\nAll commands below are suppose to be running in {repohome} directory.\n====\n\n### Use ansible-vault to manage sensitive files\n\nThe file _group_vars/tag_Type_master/vault.yml_ has sensitive data and should be protect using https://docs.ansible.com/ansible/latest/user_guide/vault.html[ansible-vault]. \n\n[source,bash]\n----\nansible-vault [edit | view] \\\n  --vault-password-file .vault \\\n  group_vars/tag_Type_master/vault.yml\n----\n\nThis file should be updated when changed any AWS or Github access parameters.\n\n### Create the bootstrap AWS infrastructure\n\nThe directory _networkTerraform_ has a {tf} code that creates a basic networking in AWS, just the necessary elements to run :packer: later.\n\n[source,bash]\n----\ncd networkTerraform\nterraform apply\n----\n\n### Create AMI images using {packer}\n\nAfter running the initial {tf} environment, {tf} will output a variable called *public_subnets*. {packer} will use that subnet to create the initial images there.\n\nCalling {packer} for each image template to create the images. When creating a _buildbot worker_ AMI, {packer} will need some extra variables from previous {tf} execution.\n\n[source,bash]\n----\ncd packer\npacker build -var 'subnet_id=\u003csubnet output by terraform\u003e' bb-master.json\npacker build -var 'subnet_id=\u003csubnet output by terraform\u003e' \\\n    -var 'aws_profile=\u003caws buildbot profile\u003e' \\\n    -var 'aws_access_key_id=\u003caws buildpot access key\u003e' \\\n    -var 'aws_secret_access_key=\u003caws buildbot secret key\u003e' \\\n    bb-worker.json\n----\n\n{packer} creates two private AMI with all the software installed and configured by {ansible}.\n\n### Destroy the bootstrap AWS infrastructure\n\nAfter running the {packer} templates, The follow procedure destroys the bootstrap environment.\n\n[source,bash]\n----\ncd networkTerraform\nterraform destroy\n----\n\n### Create buildbot AWS infrastructure\n\nRunning {tf} to create the buildbot infrastructure.\n\n[source,bash]\n----\ncd terraform\nterraform apply\n----\n\n### Configure {bbm}\n\nIn the previously step, {tf} output some important variables that {ansible} need when running the playbook to setup the correct buildbot master configuration values.\n\nThe follow variables should get from {tf}:\n\n* url-bb-master\n* public-subnets\n* security-groups\n\nAnd update the follow variables in the file _ansible/group_vars/tag_Type_master/vars.yml_:\n\n* buildbot_aws_subnet_id, subnet where the buildmaster worker will live\n* buildbot_aws_security_group_id: security group to attach each worker\n* buildbot_aws_url: public https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html[Elastic IP Address] where buildbot master is receiving requests\n\nAlso, any other buildbot parameter should be updated before running the _ansible/bb-master-configure.yml_ playbook.\n\n[source,bash]\n----\ncd ansible\nansible-playbook \\\n  --vault-password-file .vault \\\n  -u ubuntu \\\n  -i config_aws_ec2.yml \\\n  -T 300 \\\n  bb-master-configure.yml\n----\n\n### Teardown {bbm}\n\nThe playbook _ansible/bb-master-teardown.yml_ deal with all the operations when removing any configuration done in external services, like {github}.\n\n[source,bash]\n----\ncd ansible\nansible-playbook \\\n  --vault-password-file .vault \\\n  -u ubuntu \\\n  -i config_aws_ec2.yml \\\n  -T 300 \\\n  bb-master-teardown.yml\n----\n\n### Destroy AWS infrastructure\n\nThe infrastructure can be destroyed using {tf}\n\n[source,bash]\n----\ncd terraform\nterraform destroy\n----\n\n[bibliography]\n## References\n\n- [[[buildbot-tale]]] Buildbot: a tale with examples of one more continuous integration system,   https://sudonull.com/post/977-Buildbot-a-tale-with-examples-of-one-more-continuous-integration-system\n\n- [[[buildbot]]] Buildbot Manual, http://docs.buildbot.net/current/index.html\n\n- [[autobuilder]] Autobuilder2 presentation at the Yocto Project summit 2019, https://koansoftware.com/autobuilder2-talk-yocto-project/\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoaohf%2Fbuildbot-ci","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoaohf%2Fbuildbot-ci","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoaohf%2Fbuildbot-ci/lists"}