Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/redhat-openstack/openshift-on-openstack

A place to write templates, docs etc. for deploying OpenShift on OpenStack.
https://github.com/redhat-openstack/openshift-on-openstack

Last synced: 2 months ago
JSON representation

A place to write templates, docs etc. for deploying OpenShift on OpenStack.

Host: GitHub
URL: https://github.com/redhat-openstack/openshift-on-openstack
Owner: redhat-openstack
License: apache-2.0
Created: 2015-04-27T11:46:34.000Z (over 9 years ago)
Default Branch: master
Last Pushed: 2020-09-10T08:49:56.000Z (over 4 years ago)
Last Synced: 2024-08-03T01:38:13.960Z (6 months ago)
Language: Shell
Size: 1.32 MB
Stars: 136
Watchers: 41
Forks: 87
Open Issues: 67
Metadata Files:
- Readme: README.adoc
- License: LICENSE.txt

Awesome Lists containing this project

awesome-repositories - redhat-openstack/openshift-on-openstack - A place to write templates, docs etc. for deploying OpenShift on OpenStack. (Shell)

README

= OpenShift on OpenStack

== Maintenance Status

This project is no longer being developed or maintained by its original
authors.

The https://github.com/openshift/openshift-ansible[official OpenShift installer]
now supports various cloud providers including OpenStack so a lot of the development
effort has moved there:

https://github.com/openshift/openshift-ansible/tree/master/playbooks/openstack

We recommend you take a look at it.

== About

A collection of documentation, https://wiki.openstack.org/wiki/Heat[Heat] templates, configuration and everything
else that's necessary to deploy http://www.openshift.org/[OpenShift]
on http://www.openstack.org/[OpenStack].

This template uses Heat to create the OpenStack infrastructure
components, then calls the https://github.com/openshift/openshift-ansible[OpenShift Ansible] installer playbooks to
install and configure OpenShift on the VMs.

== Architecture

All of the OpenShift VMs will share a private network. This network is
connected to the public network by a router.

The deployed OpenShift environment is composed of a replicated
set of _OpenShift master_ VMs fronted by a _load_balancer_. This provides
both a single point of access and some HA capabilities. The
applications run on one or more _OpenShift node_ VMs. These are
connected by a private _software defined network_ (SDN) which can be
implemented either with http://openvswitch.org/[OpenVSwitch] or https://github.com/coreos/flannel[Flannel].

A _bastion server_ is used to control the host and service
configuration. The host and service configuration is run using
https://www.ansible.com/[Ansible] playbooks executed from the bastion
host.

_Bastion server_, _master nodes_ and _infra nodes_ is also given a _floating IP_
address on the public network. This provides direct access to the
_bastion server_ from where you can access all nodes by SSH.
_Master nodes_ and _infra nodes_ have _floating IP_ assigned to make sure
these nodes are accessible when an _external loadbalancer_ is used for
accessing OpenShift services.

All of the OpenShift hosts (master, infra and node) have block storage for
Docker images and containers provided by Cinder. OpenShift will run a
local Docker registry, also backed by Cinder block storage. Finally
all nodes will have access to Cinder volumes which can be created by
OpenStack users and mounted into containers by http://kubernetes.io/[Kubernetes].

image:graphics/architecture.png[caption="VM and Network Layout",
title="OpenShift Architecture"]

== Prerequisites

. OpenStack version Juno or later with the Heat, Neutron, Ceilometer, Aodh (Mitaka or later) services
running:
* heat-api-cfn service - used for passing heat metadata to nova instances
* Neutron LBaaS service (optional) - used for loadbalancing requests in HA
mode, if this service is not available, you can deploy dedicated
loadbalancer node, see <>
* Ceilometer services (optional) - used when autoscaling is enabled
. `ServerGroupAntiAffinityFilter` enabled in Nova service
(optionally `ServerGroupAffinityFilter` when using all-in-one OpenStack
environment)
. http://www.centos.org/[CentOS] 7.2 cloud image (we leverage cloud-init)
loaded in Glance for OpenShift Origin Deployments.
https://access.redhat.com/downloads[RHEL]_ 7.2 cloud image if doing Atomic
Enterprise or OpenShift Container Platform. Make sure to use official images to avoid
unexpected issues during deployment (e.g. a custom firewall may block OpenShift
inter-node communication).
. An SSH keypair loaded into Nova
. A (Neutron) network with a pool of floating IP addresses available

CentOS and RHEL are the only tested distros for now.

=== DNS Server

The OpenShift installer requires that all nodes be reachable via their
hostnames. Since OpenStack does not currently provide an internal name
resolution, this needs to be done with an external DNS service that
all nodes use via the `dns_nameserver` parameter.

In a production deployment this would be your existing DNS, but if you
don't have the ability to update it to add new name records, you will
have to deploy one yourself.

We have provided a separate repository that can deploy a DNS server
suitable for OpenShift:

https://github.com/openshift/openshift-ansible-contrib/tree/master/reference-architecture/osp-dns

NOTE: If your DNS supports dynamic updates via RFC 2136, you can pass
the update key to the Heat stack and all nodes will register
themselves as they come up. Otherwise, you will have to update your
DNS records manually.

=== Red Hat Software Repositories

When installing OpenShift Container Platform on RHEL the OpenShift and
OpenStack repositories must be enabled, along with several common
repositories. These repositories must be available under the
subscription account used for installation.

.Required Repositories for RHEL installation
|===
|Repo Name |Purpose

== Creating an All-In-One Demo Environment

Following steps can be used to setup all-in-one testing/developer environment:

```bash
# OpenStack does not run with NetworkManager
systemctl stop NetworkManager
systemctl disable NetworkManager

# The Packstack Installer is not supported for production but will work
# for demonstrations
yum -y install openstack-packstack libvirt git

# Add room for images if /varlib is too small
mv /var/lib/libvirt/images /home
ln -s /home/images /var/lib/libvirt/images

# Install OpenStack demonstrator with no real security
# This produces the keystonerc_admin file used below
packstack --allinone --provision-all-in-one-ovs-bridge=y \
--os-heat-install=y --os-heat-cfn-install=y \
--os-neutron-lbaas-install=y \
--keystone-admin-passwd=password --keystone-demo-passwd=password

# Retrieve the Heat templates for OpenShift
git clone https://github.com/redhat-openstack/openshift-on-openstack.git

# Retrieve a compatible image for the OpenShift VMs
curl -O http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2

# Set access environment parameters for the new OpenStack service
source keystonerc_admin

# Load the VM image into the store and make it available for creating VMs
glance image-create --name centos72 --is-public True \
--disk-format qcow2 --container-format bare \
--file CentOS-7-x86_64-GenericCloud.qcow2
# For newer versions of glance clients, substitute "--is-public True" with "--visibility public"

# Install the current user's SSH key for access to VMs
nova keypair-add --pub-key ~/.ssh/id_rsa.pub default
```

== Deployment

You can pass all environment variables to heat on command line. However,
two environment files are provided as examples.

* ``env_origin.yaml`` is an example of the variables to deploy an OpenShift
Origin 3 environment.
* ``env_aop.yaml`` is an example of the variables to deploy an Atomic
Enterprise or OpenShift Container Platform 3 environment. Note deployment type
should be *openshift-enterprise* for OpenShift or *atomic-enterprise*
for Atomic Enterprise. Also, a valid RHN subscription is required
for deployment.

Here is a sample of environment file which uses a subset of parameters which
can be set by the user to configure OpenShift deployment. All configurable
parameters including description can be found in the `parameters` section in
the link:openshift.yaml[main template]. Assuming your external
network is called ``public``, your SSH key is `default`, your
CentOS 7.2 image is ``centos72`` and your domain name is ``example.com``,
this is how you deploy OpenShift Origin:

```yaml
cat << EOF > openshift_parameters.yaml
parameters:
# Use OpenShift Origin (vs OpenShift Container Platform)
deployment_type: origin

# set SSH access to VMs
ssh_user: centos
ssh_key_name: default

# Set the image type and size for the VMs
bastion_image: centos72
bastion_flavor: m1.medium
master_image: centos72
master_flavor: m1.medium
infra_image: centos72
infra_flavor: m1.medium
node_image: centos72
node_flavor: m1.medium
loadbalancer_image: centos72
loadbalancer_flavor: m1.medium

# Set an existing network for inbound and outbound traffic
external_network: public
dns_nameserver: 8.8.4.4,8.8.8.8

# Define the host name templates for master and nodes
domain_name: "example.com"
master_hostname: "origin-master"
node_hostname: "origin-node"

# Allocate additional space for Docker images
master_docker_volume_size_gb: 25
infra_docker_volume_size_gb: 25
node_docker_volume_size_gb: 25

# Specify the (initial) number of nodes to deploy
node_count: 2

# Add auxiliary services: OpenStack router and internal Docker registry
deploy_router: False
deploy_registry: False

# If using RHEL image, add RHN credentials for RPM installation on VMs
rhn_username: ""
rhn_password: ""
rhn_pool: '' # OPTIONAL

# Currently Ansible 2.1 is not supported so add these parameters as a workaround
openshift_ansible_git_url: https://github.com/openshift/openshift-ansible.git
openshift_ansible_git_rev: master

resource_registry:
# use neutron LBaaS
OOShift::LoadBalancer: openshift-on-openstack/loadbalancer_neutron.yaml
# use openshift SDN
OOShift::ContainerPort: openshift-on-openstack/sdn_openshift_sdn.yaml
# enable ipfailover for router setup
OOShift::IPFailover: openshift-on-openstack/ipfailover_keepalived.yaml
# create dedicated volume for docker storage
OOShift::DockerVolume: openshift-on-openstack/volume_docker.yaml
OOShift::DockerVolumeAttachment: openshift-on-openstack/volume_attachment_docker.yaml
# use ephemeral cinder volume for openshift registry
OOShift::RegistryVolume: openshift-on-openstack/registry_ephemeral.yaml
EOF
```

```bash
# retrieve the Heat template (if you haven't yet)
git clone https://github.com/redhat-openstack/openshift-on-openstack.git
```

After this you can deploy using the heat command

```bash
# create a stack named 'my-openshift'
heat stack-create my-openshift -t 180 \
-e openshift_parameters.yaml \
-f openshift-on-openstack/openshift.yaml
```

or using the generic OpenStack client

```
# create a stack named 'my-openshift'
openstack stack create --timeout 180 \
-e openshift_parameters.yaml \
-t openshift-on-openstack/openshift.yaml my-openshift
```

The ``node_count`` parameter specifies how many compute nodes you
want to deploy. In the example above, we will deploy one master, one infra node
and two compute nodes.

The templates will report stack completion back to Heat only when the whole
OpenShift setup is finished.

== Debugging

Sometimes it's necessary to find out why a stack was not deployed as expected.
link:README_debugging.adoc[Debugging] helps you find the root cause of the
issue.

== OpenStack Integration

OpenShift on OpenStack takes advantage of the cloud provider to offer
features such as dynamic storage to the OpenShift users. Auto scaling
also requires communication with the OpenStack service. You must
provide a set of OpenStack credentials so that OpenShift and the heat
scaling mechanism can work correctly.

These are the same values used to create the Heat stack.

.Sample OSP Credentials - `osp_credentials.yaml`
----
---
parameters:
os_auth_url: http://10.0.x.x:5000/v2.0
os_username:
os_password:
os_region_name: regionOne
os_tenant_name:
os_domain_name:
----

When invoking the stack creation, include this by adding `-e
osp_credentials.yaml` to the command.

== [[ca-certificates]]OpenStack with SSL/TLS

If your OpenStack service is encrypted with SSL/TLS, you will need to
provide the CA certificate so that the communication channel can be
validated.

The CA certificate is provided as a literal string copy of contents of
the CA certificate file, and can be included in an additional
environment file:

.CA Certificate Parameter File `ca_certificates.yaml`
----
---
parameters:
ca_cert: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
----

When invoking the stack creation, includ this by adding `-e
ca_certificates.yaml`.

You can include multiple CA certificate strings and all will be imported
into the CA list on all instances.

== Multiple Master Nodes

You can deploy OpenShift with multiple master hosts using the 'native'
HA method (see
https://docs.openshift.org/latest/install_config/install/advanced_install.html#multiple-masters
for details) by increasing number of master nodes. This can be done by setting
``master_count`` heat parameter:

```bash
heat stack-create my-openshift \
-e openshift_parameters.yaml \
-P master_count=3 \
-f openshift-on-openstack/openshift.yaml
```

Three master nodes will be deployed. Console and API URLs
point to the loadbalancer server which distributes requests across all
three nodes. You can get the URLs from Heat by running
`heat output-show my-openshift console_url` and
`heat output-show my-openshift api_url`.

== Multiple Infra Nodes

You can deploy OpenShift with multiple infra hosts. Then OpenShift router
is deployed on each of infra node (only if `-P deploy_router=true` is used)
and router requests are load balanced by either dedicated or neutron
loadbalancer. This can be done by setting ``infra_count`` heat parameter:

```bash
heat stack-create my-openshift \
-e openshift_parameters.yaml \
-P infra_count=2 \
-P deploy_router=true \
-f openshift-on-openstack/openshift.yaml
```

Two infra nodes will be deployed. Loadbalancer server distributes requests on
ports 80 and 443 across both nodes.

[[LoadBalancing]]
== Select Loadbalancer Type

When deploying multiple master nodes, both access to the nodes and OpenShift
router pods (which run on infra nodes) have to be loadbalanced.
openshift-on-openstack provides multiple options for setting up loadbalancing:

* Neutron LBaaS - this loadbalancer is used by default. Neutron loadbalancer
serviceis used for loadbalancing console/api requests to master nodes. At the
moment OpenShift router requests are not loadbalanced and an external
loadbalancer has to be used for it.
This is default option, but can be set
explicitly by including `-e openshift-on-openstack/env_loadbalancer_neutron.yaml`
when creating the stack. By default, this mode uses <>.

* External loadbalancer - a user is expected to set its own loadbalancer both
for master nodes and OpenShift routers.
This is suggested type for production.
To select this type include `-e openshift-on-openstack/env_loadbalancer_external.yaml`
when creating the stack and also set `lb_hostname` parameter to point to the
loadbalancer's fully qualified domain name. Once stack creation is finished
you can set your external loadbalancer with the list of created master nodes.

* Dedicated loadbalancer node - a dedicated node is created during stack
creation and HAProxy loadbalancer is configured on it. Both console/API and
OpenSHift router requests are load balanced by this dedicated node.
This type is useful for demo/testing purposes only because HA is not assured for
the single loadbalancer. To select this type include
`-e openshift-on-openstack/env_loadbalancer_dedicated.yaml` when creating the stack.
node.

* None - if only single master node is deployed, it's possible to skip
loadbalancer creation, then all master node requests and OpenShift router requests
point to the single master node.
To select this type include `-e openshift-on-openstack/env_loadbalancer_none.yaml`
when creating the stack. By default, this mode uses <>.

== Select SDN Type ==

By default, OpenShift is deployed with https://docs.openshift.com/enterprise/3.2/architecture/additional_concepts/sdn.html[OpenShift-SDN].
When used with OpenStack Neutron with GRE or VXLAN tunnels, packets are encapsulated twice
which can have an impact on performances. Those Heat templates allow using https://github.com/coreos/flannel[Flannel]
instead of openshift-sdn, with the `host-gw` backend to avoid the double encapsulation.
To do so, you need to include the `env_flannel.yaml` environment file when you create the stack:

```bash
heat stack-create my_openshift \
-e openshift_parameters.yaml \
-f openshift-on-openstack/openshift.yaml \
-e openshift-on-openstack/env_flannel.yaml
```

To use this feature, the Neutron `port_security` extension driver needs to be enabled.
To do so and when using the ML2 driver, edit the file `/etc/neutron/plugins/ml2/ml2_conf.ini`
and make sure it contains the line:

```bash
extension_drivers = port_security
```

Note that this feature is still in experimental mode.

== LDAP authentication

You can use an external LDAP server to authenticate OpenShift users. Update
parameters in `env_ldap.yaml` file and include this environment file
when you create the stack.

Example of `env_ldap.yaml` using an Active Directory server:

.LDAP parameter file `env_ldap.yaml
----
parameter_defaults:
ldap_hostname:
ldap_ip:
ldap_url: ldap://:389/CN=Users,DC=example,DC=openshift,DC=com?sAMAccountName
ldap_bind_dn: CN=Administrator,CN=Users,DC=example,DC=openshift,DC=com?sAMAccountName
ldap_bind_password:
----

```bash
heat stack-create my-openshift \
-e openshift_parameters.yaml \
-e openshift-on-openstack/env_ldap.yaml \
-f openshift-on-openstack/openshift.yaml
```

If your LDAP service uses SSL, you will also need to add a link:#ca-certificates[CA Certficate] for the LDAP communications.

== Using Custom Yum Repositories

You can set additional Yum repositories on deployed nodes by passing `extra_repository_urls`
parameter which contains list of Yum repository URLs delimited by comma:

```bash
heat stack-create my-openshift \
-e openshift_parameters.yaml \
-P extra_repository_urls=http://server/my/own/repo1.repo,http://server/my/own/repo2.repo
-f openshift-on-openstack/openshift.yaml
```

== Using Custom Docker Respositories

You can set additional Docker repositories on deployed nodes by passing `extra_docker_repository_urls`
parameter which contains list of docker repository URLs delimited by comma, if a repository is insecure
you can use `#insecure` suffix for the repository:

```bash
heat stack-create my-openshift \
-e openshift_parameters.yaml \
-P extra_docker_repository_urls='user.docker.example.com,custom.user.example.com#insecure'
-f openshift-on-openstack/openshift.yaml
```

== Using Persistent Cinder Volume for Docker Registry

When deploying OpenShift registry (`-P deploy_registry=true`) you can use either
an ephemeral or persistent Cinder volume. Ephemeral volume is used by default,
the volume is automatically created when creating the stack and is also
deleted when deleting the stack. Alternatively you can use an existing Cinder
volume by including the `env_registry_persistent.yaml` environment file and
`registry_volume_id` when you create the stack:

```bash
heat stack-create my-openshift \
-e openshift_parameters.yaml \
-f openshift-on-openstack/openshift.yaml \
-e openshift-on-openstack/env_registry_persistent.yaml \
-P registry_volume_id=
```

Persistent volume is not formatted when creating the stack, if you have a new
unformatted volume you can enforce formatting by passing
`-P prepare_registry=true`.

== Accessing OpenShift

From user point of view there are two entry points into the deployed OpenShift:

* OpenShift console and API URLs: these URLs usually point to the
loadbalancer host and can be obtained by:

```
heat output-show my-openshift console_url
heat output-show my-openshift api_url
```

* Router IP: the IP address which application OpenShift router service
listens on. This IP will be used for setting wildcard DNS for
.apps. subdomain. The IP can be obtained by:

```
heat output-show my-openshift router_ip
```

== Setting DNS

To make sure that console and API URLs resolving works properly, you have to
create a DNS record for the hostname used in `console_url` and `api_url` URLs.
The floating IP address can be obtained by:

```
heat output-show my-openshift loadbalancer_ip
```

For example if `console_url` is `https://default32-lb.example.com:8443/console/`
and loadbalancer_ip is `172.24.4.166` there should be a DNS record for domain
`example.com`:

```
default32-lb IN A 172.24.4.166
```

If OpenShift router was deployed (`-P deploy_router=true`) you also may want
to make sure that
https://docs.openshift.com/enterprise/3.2/install_config/install/prerequisites.html#wildcard-dns-prereq[wildcard DNS]
is set for application subdomain. For example if used domain is `example.com`
and `router_ip` is `172.24.4.168` there should be a DNS record for domain
`example.com`:
```
*.cloudapps.example.com. 300 IN A 172.24.4.168
```

[NOTE]
====
The above DNS records should be set on the DNS server authoritative for the
domain used in OpenShift cluster (`example.com` in the example above).
====

=== Dynamic DNS Updates

If your DNS servers support dynamic updates (as defined in RFC 2136),
you can pass the update key in the `dns_update_key` parameter and each
node will register its internal IP address to all the DNS servers in
the `dns_nameserver` list.

In addition, if you use the *dedicated load balancer*, the API and
wildcard entries will be created as well. Otherwise, you will need to
set them manually.

== Retrieving the OpenShift CA certificate

You can retrieve the CA certificate that was generated during the OpenShift
installation by running

```bash
heat output-show --format=raw my-openshift ca_cert > ca.crt
heat output-show --format=raw my-openshift ca_key > ca.key
```

== Container and volumes quotas

OpenShift has preliminary support for local emptyDir volume quotas. You can
set the `volume_quota` parameter to a resource quantity representing the desired
quota per FSGroup.

You can set quota on the maximum size of the containers using the
`container_quota` parameter in GB.

Example:
```yaml
volume_quota: 10
container_quota: 20
```

== Disabling Cinder volumes for Docker storage

By default, the Heat templates create a Cinder volume per OpenShift node
to host containers. This can be disabled by including both `volume_noop.yaml`
and `volume_attachment_noop.yaml` in your environment file:

resource_registry:
...
OOShift::DockerVolume: volume_noop.yaml
OOShift::DockerVolumeAttachment: volume_attachment_noop.yaml

[[IPFailover]]
== IP failover

These templates allow using IP failover for the OpenShift router. In this mode,
a virtual IP address is assigned for the OpenShift router. Multiple instances of
router may be active but only one instance at a time will have the virtual IP.
This ensures that minimal downtime in the case of the failure of the current active
router.

By default, IP failover is used when the load balancing mode is `Neutron LBaas` or
`None` (see section <>).

The virtual IP of the router can be retrieved with
```bash
heat output-show --format=raw my-openshift router_ip
```

== Scaling Up or Down

You can manually scale up or down OpenShift nodes by updating `node_count` heat
stack parameter to the desired new count:

```
heat stack-update -P node_count=5
```

If the stack has 2 nodes, 3 new nodes are added. If the stack has 7 nodes, 2
are removed. Any running pods are evacuated from the node being removed.

== Autoscaling

Scaling of OpenShift nodes can be automated by using Ceilometer metrics.
By default `cpu_util` metering is used. You can enable autoscaling by `autoscaling`
heat parameter and tweaking properties of `cpu_alarm_high` and `cpu_alarm_low` in
`openshift.yaml`.

== Removing or Replacing Specific Nodes

Sometimes it's necessary to remove or replace specific nodes from the stack.
For example because of a hardware issue. Because OpenShift "compute" nodes are
members of heat AutoScalingGroup adding or removing nodes is by default handled
by a scaling policy and when removing a node the oldest one is
selected by Heat by default. A specific node can be removed with following
steps though:

```bash
# delete the node
$ nova delete instance_name

# let heat detect the missing node
$ heat action-check stack_name

# update the stack with desired new number of nodes (same is before
# for replacement, decreased by 1 for removal)
$ heat stack-update -P node_count=
```

== Known Bugs

Here is the link:README_bugs.adoc[list] of bugs which are not fixed
and you may hit.

== Customize OpenShift installation ==

Those Heat templates make use of openshift-ansible to deploy OpenShift.
You can provide additional parameters to openshift-ansible by specifying
a JSON string as the `extra_openshift_ansible_params` parameter.
For example :

```bash
$ heat stack-create -P extra_openshift_ansible_params='{"osm_use_cockpit":true}'
```

This parameter must be used with caution as it may conflict with other
parameters passed to openshift-ansible by the Heat templates.

== Current Status

1. The CA certificate used with OpenShift is currently not configurable.
2. The apps cloud domain is hardcoded for now. We need to make this configurable.

== Prebuild images

A `customize-disk-image` script is provided to preinstall OpenShift packages.

```bash
./customize-disk-image --disk rhel7.2.qcow2 --sm-credentials user:password
```

The modified image must be uploaded into Glance and used as the server image
for the heat stack with the `server_image` parameter.

== Copyright

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.