{"id":13415977,"url":"https://gitlab.com/blackfish/blackfish","last_synced_at":"2025-03-14T23:31:17.753Z","repository":{"id":64589753,"uuid":"1177932","full_name":"blackfish/blackfish","owner":"blackfish","description":"a Swarm Cluster Maker for Dev \u0026 Production","archived":false,"fork":false,"pushed_at":null,"size":null,"stargazers_count":20,"open_issues_count":2,"forks_count":12,"subscribers_count":null,"default_branch":"master","last_synced_at":"2024-07-31T21:54:50.960Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://gitlab.com/uploads/-/system/project/avatar/1177932/blackfish_logo.png","metadata":{"files":{"readme":"README.org","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-05-12T14:24:36.309Z","updated_at":"2017-04-26T16:34:36.602Z","dependencies_parsed_at":"2022-12-13T08:32:38.900Z","dependency_job_id":null,"html_url":"https://gitlab.com/blackfish/blackfish","commit_stats":null,"previous_names":[],"tags_count":5,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/repositories/blackfish%2Fblackfish","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/repositories/blackfish%2Fblackfish/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/repositories/blackfish%2Fblackfish/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/repositories/blackfish%2Fblackfish/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/owners/blackfish","download_url":"https://gitlab.com/blackfish/blackfish/-/archive/master/blackfish-master.zip","host":{"name":"gitlab.com","url":"https://gitlab.com","kind":"gitlab","repositories_count":4518057,"owners_count":6822,"icon_url":"https://github.com/gitlab.png","version":null,"created_at":"2022-05-30T11:31:42.605Z","updated_at":"2024-07-18T11:24:13.055Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/gitlab.com/owners"}},"keywords":[],"created_at":"2024-07-30T21:00:53.411Z","updated_at":"2025-03-14T23:31:13.530Z","avatar_url":"https://gitlab.com/uploads/-/system/project/avatar/1177932/blackfish_logo.png","language":null,"funding_links":[],"categories":["Container Operations","Running Containers"],"sub_categories":["Deployment and Infrastructure","Deployment \u0026 Platforms"],"readme":"#+TITLE: Blackfish: a CoreOS VM to build swarm clusters for Dev \u0026 Production\n#+OPTIONS: toc:1\n#+SETUPFILE: theme.setup\n\nNote: you may prefer reading the [[https://blackfish.gitlab.io/blackfish/][README]] on the gitlab website.\n\n* Description\n\nBlackfish is a pre-built CoreOS VM that ease the bootstrap of swarm clusters.\nIt contains all basic services required to ease the deployment of your apps and manage basic production operations.\nAll the services are pre configured for production with HA and TLS communication enabled.\n\n- a Consul cluster with a registrator\n- an internal HA Docker registry\n- an HAProxy load balancer with auto discovery\n- centralized logs with an HA graylog2 setup\n- centralized monitoring with telegraf+influx+grafana\n\nThe project also comes with all the materials to boot a cluster on your local machine with a Vagrant setup, or on AWS EC2 with terraform scripts.\n\nWe hope it will help you fill the gap between \"Docker in Dev\" and \"Docker in Production\"\n\n#+CAPTION: Docker in Dev VS Prod\n #+NAME:   fig:docker-prod.jpg\n #+ATTR_HTML: width=\"100px\"\n [[https://blackfish.gitlab.io/blackfish/images/docker-prod.jpg]]\n\n\nYou'll find other and maybe simpler tutorials or Github projects to deploy Swarm on AWS, but if you don't want your cluster to be exposed on public facing IPs, you'll then have to get your hands dirty on a lot of other things.\n\nBlackfish is built on top of the following components:\n- [[http://packer.io/][Packer]] for building the boxes for various providers (Virtualbox, AWS, Kvm, ...)\n- [[https://www.terraform.io/][Terraform]] for provisioning the infrastructure on AWS\n- [[http://vagrantup.com][Vagrant]] for running the Swarm cluster in Virtualbox\n\n\n* Pre-Requisites\n\nTo use this project you will need at least this list of tools properly installed on your box:\n\n- Docker 1.10\n- Vagrant 1.8\n- Virtualbox 5.0\n\n* Quickstart\n\nTo quickly bootstrap a Swarm cluster with Vagrant on your local machine, configure the [[https://gitlab.com/blackfish/blackfish/raw/master/nodes.yml][nodes.yml]] file and type the following commands.\n\n #+BEGIN_SRC bash\n $ git clone https://gitlab.com/blackfish/blackfish.git\n $ cd blackfish\n $ vi nodes.yml\n ...\n #+END_SRC\n\n #+CAPTION: nodes.yml\n #+NAME:   fig:nodes.yml\n #+BEGIN_SRC yaml\n---\nprovider            : \"virtualbox\"\nadmin-network       : 192.168.101.0/24\nbox                 : yanndegat/blackfish-coreos\nid                  : myid\nstack               : vagrant\ndc                  : dc1\nconsul-joinip       : 192.168.101.101\nconsul-key          : hMJ0CFCrpFRQfVeSQBUUdQ==\nregistry-secret     : A2jMOuYSS+3swzjAR6g1x3iu2vY/cROWYZrJLqYffxA=\nssh-key : ./vagrant.pub\nprivate-ssh-key : ~/.vagrant.d/insecure_private_key\nnodes:\n  - ip      : 192.168.101.101\n    docker-registry : true\n    memory  : \"3072\"\n    cpus    : \"2\"\n  - memory  : \"3072\"\n    cpus    : \"2\"\n    ip      : 192.168.101.102\n  - memory  : \"3072\"\n    cpus    : \"2\"\n    ip      : 192.168.101.103\n #+END_SRC\n\n #+BEGIN_SRC bash\n $ vagrant up\n ==\u003e box: Loading metadata for box 'yanndegat/blackfish'\n     box: URL: https://atlas.hashicorp.com/yanndegat/blackfish\n ==\u003e box: Adding box 'yanndegat/blackfish' (v0.1.0) for provider: virtualbox\n     box: Downloading: https://atlas.hashicorp.com/yanndegat/boxes/blackfish/versions/0.0.5/providers/virtualbox.box\n     box: Progress: 26% (Rate: 1981k/s, Estimated time remaining: 0:02:24)\n ...\n $\n #+END_SRC\n\nTLS certificates have been generated in your ~$HOME/.blackfish/vagrant~ directory. You have to declare the CA Cert on your host, according to your system.\n\nOnce you have registered your [[tls][tls certificates]] setup your [[dns][dns configuration]], you can go to https://consul-agent.service.vagrant:4443/ui\n\nNow refer to the [[play][Play with swarm]] section to go futher.\n\n* Getting Started on AWS\n\n#+BEGIN_NOTES Warning\nThis section is obsolete. To use blackfish on AWS, several terraform scripts are available but they're toooooo complex.\nInstead we want to be able to use the same \"nodes*yml\" configuration files to bootstrap swarm on AWS. To do so, we will in a near future\nprovide some kind of go binary that reads \"nodes.yml\" \u0026 terraform templates and outputs a complete terraform directory.\n\nMeanwhile, you can use the ~terraform/aws/vpc/~ templates or refer to the old scripts to bootstrap the vpc, and use vagrant --provider=aws to bootstrap the aws nodes.\n#+END_NOTES\n\nRefer to the [[https://blackfish.gitlab.io/blackfish/terraform/aws/README.html][Aws README]] file.\n\n\nHere's an example of what a nodes.yml for AWS could look like\n\n#+BEGIN_SRC yaml\n---\nprovider            : aws\nadmin-network       : 10.233.1.0/24\nstack               : demo\ndc                  : one\nid                  : prod1\nconsul-joinip       : 10.233.1.201\nconsul-key          : dlnA+EWVSiQyfd0vkApmTUu4lDvMlmJcjMy+8dMEVkw=\nregistry-secret     : rJ/9vXube9iujCjFiniJODQX60Q/XJytUJyOKQfPaLo=\n\njournald-sink       : journald.service.demo:42201\ninfluxdb-url        : http://influxdb.service.demo:48086\n\nlabels              :\n  - type=control\n\nnodes:\n  - ip              : 10.233.1.201\n    docker-registry : true\n  - ip              : 10.233.1.202\n  - ip              : 10.233.1.203\n\naws:\n  region            : eu-west-1\n  access-key-id     : XXXXXXXXXXXXXXX\n  secret-access-key : YYYYYYYYYYYYYYYYYYYYYYYYYYYYY\n  availability-zone : eu-west-1a\n  instance-type     : m3.xlarge\n  ebs-optimized     : false\n  keypair-name      : my-keypair\n  private-ssh-key   : ~/.ssh/my-keypair.key\n  subnet-id         : subnet-d779afb3\n  s3bucket          : bucket-742587092752\n  security-groups   :\n    - sg-2676b741\n\n#+END_SRC\n\n\n* \u003c\u003cplay\u003e\u003ePlay with your Swarm cluster\n\nNow we can play with Swarm.\n\n** Using the Swarm cluster\n\nYou can now use your Swarm cluster to run Docker containers as simply as you would do to run a container on your local Docker engine. All you have to do is\ntarget the IP of one of your Swarm node.\n\n#+BEGIN_SRC bash\n$ export PATH=$PATH:$(pwd)/bin\n$ blackfish vagrant run --rm -it alpine /bin/sh\n/ # ...\n#+END_SRC\n\n\n** Using the Blackfish internal registry\n\nThe Blackfish internal registry which is automatically started on the Swarm cluster is registered on the \"registry.service.dev.vagrant:5000\" name. So you have to tag \u0026 push Docker images with this name if you want the nodes to be able to download your images.\n\nAs the registry has an auto signed TLS certificate, you have to declare its CA Cert on your Docker engine and on your system (again according to your OS)\n\n#+BEGIN_SRC bash\n$ export PATH=$PATH:$(pwd)/bin\n$ sudo mkdir -p /etc/docker/certs.d/registry.service.vagrant\n$ sudo cp ~/.blackfish/vagrant/ca.pem /etc/docker/certs.d/registry.service.vagrant/\n$ sudo systemctl restart docker\n...\n$ docker tag alpine registry.service.vagrant/alpine\n$ docker push registry.service.vagrant/alpine\n...\n$ blackfish vagrant pull registry.service.vagrant/alpine\n...\n$ blackfish vagrant run --rm -it registry.service.vagrant/alpine /bin/sh\n/ # ...\n#+END_SRC\n\n\n** Run the examples\n\nExamples are available in the [[https://gitlab.com/blackfish/blackfish/raw/master/examples][examples]] directory. You can play with them to discover how to work with Docker Swarm.\n\n\n* Blackfish Components\n\n** Architecture guidelines\n\nThe Blackfish VM tries to follow the Immutable Infrastructure precepts:\n\n- Every component of the system must be able to boot/reboot without having to be provisionned with configuration elements other than via cloud init.\n- Every component of the system must be able to discover its pairs and join them\n- If a component can't boot properly, it must be considered as dead. Don't try to fix it.\n- To update the system, we must rebuild new components and replace existing ones with the new ones.\n\n\n** Blackfish is architectured with the following components :\n\n- a Consul cluster setup with full encryption setup, which consists of a set of Consul agents running in \"server\" mode, and additionnal nodes running in \"agent\" mode.\n  The Consul cluster could be used:\n  - as a distributed key/value store\n  - as a service discovery\n  - as a DNS server\n  - as a backend for Swarm master election\n\n- a Swarm cluster setup with full encryption setup, which consists of a set of Swarm agents running in \"server\" mode, and additionnal nodes running in agent mode.\n  Every Swarm node will also run a Consul agent and a Registrator service to declare every running container in Consul.\n\n- a HA private Docker registry with TLS encryption. It's registered under the DNS address registry.service.vagrant.\n  HA is made possible by the use of a shared filesystem storage.\n  On AWS, it is possible to configure the registry's backend to target a S3 bucket.\n\n- a load balancer service built on top of HAProxy and consul-template with auto discovery\n\nSome nodes could play both \"consul server\" and \"swarm server\" roles to avoid booting too many servers for small cluster setups.\n\n\n** The Nodes.yml\n\nThe philosophy behind the \"nodes.yml\" files is to configure several \"nodes.yml\" files, each defining small clusters with special infrastructure features,\nand make them join together to form a complete and robust Swarm cluster, multi az, multi dc, with several types of storage, ...\n\nYou can select the nodes.yml file you want to target by simply setting the ~BLACKFISH_NODES_YML~ env variable\n\n#+BEGIN_SRC bash\n$ BLACKFISH_NODES_YML=nodes_mycluster_control.yml vagrant up --provider=aws\n...\n$ BLACKFISH_NODES_YML=nodes_mycluster_ebs-storage.yml vagrant up --provider=aws\n...\n$ BLACKFISH_NODES_YML=nodes_mycluster_ssd.yml vagrant up --provider=aws\n...\n\n#+END_SRC\n\n\n* Considerations \u0026 Roadmap\n\n\n** Volume driver Plugins\n\nFlocker is available on Blackfish but it has been disabled due to a critical bug when running with a Docker engine \u003c 1.11.\nRex Ray and Convoy suffer the same bug. Plus Flocker's control service is not HA ready.\n\n** CoreOS Alpha channel\nToo many critical bugs are fixed on every Docker release and the alpha channel is the one that sticks to the Docker engine.\nWhen a good configuration of the different components will be stabilized, we will move to more stable channels.\n\n\n** Consul + Etcd on the same hosts ?\n\nIt sounds crazy. Yet it is recommanded to use 2 separate Consul clusters to run a Swarm cluster : One for master election and service discovery, one for the Docker overlay network.\nAs we run on CoreOS, there's a feature we'd like to enable: the automatic upgrade of CoreOS nodes based on their channel. To avoid that a whole cluster reboots all its node at the same time, CoreOS can use etcd to provide coordination.\n\n** Use docker-machine\n\nWe didn't use docker-machine as it forces us to enter the \"post provisionning\" way of doing things.\n\n** Run Consul and Swarm services as rocket containers\n\nThere are some caveats running the system services as Docker containers, even on CoreOS. The main problem is the process supervision with systemd, as full described in this [[https://lwn.net/Articles/676831/][article]].\nPlus, we don't want system services to be visible and \"killable\" by a simple Docker remote command.\n\n That's why every system component is run with the rocket CoreOS container engine.\n\n** Monitoring\n\nMonitoring and Log centralization is provided in a simple yet powerful manner:\n\nEach node can be configured to report metrics and output its log to a remote machine.\nThat way, you can bootstrap a cluster and then run monitoring tools such as graylog2 and influxdb on it, the nodes will automatically start sending to it.\n\n\n** Running on GCE\n\nTBDone\n\n** Running on Azure\n\n\nTBDone\n\n** Running on premise\n\nThats why we'e chosen CoreOs. Coreos comes with powerful tools such as Ignition and coreos-baremetal that allow us to boot our solution on premise infrastructures.\n\n** How to do rolling upgrades of the infrastructure with terraform...?\n\nwell. that has to be defined.\n\n* \u003c\u003cdns\u003e\u003eConfigure DNS resolution\n\nBefore using Swarm, you have to declare the Blackfish VMs internal DNS in your system. To do so, you have multiple options:\n\n- add one of the hosts in your /etc/resolv.conf (quick but dirty)\n- configure your network manager to add the hosts permanently\n  #+BEGIN_SRC bash\n  echo 'server=/vagrant/192.168.101.101' | sudo tee -a /etc/NetworkManager/dnsmasq.d/blackfish-vagrant\n  sudo systemctl restart NetworkManager.service\n  #+END_SRC\n- configure a local dnsmasq service which forwards dns lookups to the Blackfish Dns service according to domain names.\n\nFor the latter solution, you can refer to the [[https://gitlab.com/blackfish/blackfish/raw/master/bin/dns][./bin/dns]] file which runs a dnsmasq forwarder service.\n\n#+BEGIN_SRC bash\n$ # check your /etc/resolv.conf file\n$ cat /etc/resolv.conf\nnameserver 127.0.0.1\n...\n$ # eventually run the following command for your next boot (according to your OS)\n$ sudo su\nroot $ echo \"nameserver 127.0.0.1\" \u003e /etc/resolv.conf.head\nroot $ exit\n$ ./bin/dns vagrant 192.168.101.101\n$ dig registry.service.vagrant\n\n$ ...\n#+END_SRC\n\n** \u003c\u003cdns_macos\u003e\u003e On macOS\n\nOn MacOS, registering another DNS server for a given zone is very easy.\n#+BEGIN_SRC bash\necho \"nameserver 192.168.101.101\" | sudo tee /etc/resolver/vagrant\n#+END_SRC\n\n* \u003c\u003ctls\u003e\u003e Registering TLS Certificates\n\n** Registering the CA Root on your OS\nAccording to your OS and your distro, there are several ways to register the ca.pem on your OS\n\n#+CAPTION: Example on ubuntu\n#+NAME:   Example on ubuntu\n#+BEGIN_SRC bash\n$ sudo cp ~/.blackfish/vagrant/ca.pem /usr/local/share/ca-certificates/blackfish-vagrant-ca.crt\n$ sudo update-ca-certificates\n#+END_SRC\n\n#+CAPTION: Example on archlinux\n#+NAME:   Example on archlinux\n#+BEGIN_SRC bash\n$ sudo cp ~/.blackfish/vagrant/ca.pem /etc/ca-certificates/trust-source/blackfish-vagrant-ca.crt\n$ sudo update-ca-trust\n#+END_SRC\n\n#+CAPTION: Example on macOS\n#+NAME: Example on macOS\n#+BEGIN_SRC bash\nsudo security add-trusted-cert -d -r trustRoot -k \"/Library/Keychains/System.keychain\" ~/.blackfish/vagrant/ca.pem\n#+END_SRC\n\nIf you still have issues about pushing docker images in the registry:\n\n#+CAPTION: Example on ubuntu\n#+NAME:   Example on ubuntu\n#+BEGIN_SRC bash\n$ sudo cp ~/.blackfish/vagrant/ca.pem /etc/docker/certs.d/registry.service.[stack]\n$ systemctl restart docker\n#+END_SRC\n\n** Registering the certificates on your browser\n\nActually, there's a problem with the generated certificates under Chrome/Chromium browsers. We have to spend time on it to understand why Chrome rejects it.\nYou can still use Firefox by following these steps:\n - go to the ~Preferences/Advanced/Certificates/View Certificates/Authorities~ menu\n - import the ~$HOME/.blackfish/vagrant/ca.pem~ file\n - go to the ~Preferences/Advanced/Certificates/View Certificates/Your Certificates~ menu\n - import the ~$HOME/.blackfish/vagrant/client.pfx~\n\nNow you should be able to access the https://consul-agent.service.vagrant:4443/ui url.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/gitlab.com%2Fblackfish%2Fblackfish","html_url":"https://awesome.ecosyste.ms/projects/gitlab.com%2Fblackfish%2Fblackfish","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/gitlab.com%2Fblackfish%2Fblackfish/lists"}