An open API service indexing awesome lists of open source software.

https://github.com/rguske/openshift-agent-based-installer-airgapped

Installing Red Hat OpenShift using the installation method Agent Based Installer in an air-gapped environment.
https://github.com/rguske/openshift-agent-based-installer-airgapped

airgapped disconnected kubernetes openshift openshift-container-platform redhat-openshift

Last synced: 17 days ago
JSON representation

Installing Red Hat OpenShift using the installation method Agent Based Installer in an air-gapped environment.

Awesome Lists containing this project

README

          

# Installing OpenShift in an air-gapped environment using the Agent-Based Installer

A disconnected environment is an environment that does not have full access to the internet.

OpenShift Container Platform is designed to perform many automatic functions that depend on an internet connection, such as retrieving release images from a registry or retrieving update paths and recommendations for the cluster. Without a direct internet connection, you must perform additional setup and configuration for your cluster to maintain full functionality in the disconnected environment.

Source: [Understanding disconnected installation mirroring](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/installing_an_on-premise_cluster_with_the_agent-based_installer/understanding-disconnected-installation-mirroring)

---
- [Installing OpenShift in an air-gapped environment using the Agent-Based Installer](#installing-openshift-in-an-air-gapped-environment-using-the-agent-based-installer)
- [How it works](#how-it-works)
- [Connected Mirroring vs Disconnected Mirroring](#connected-mirroring-vs-disconnected-mirroring)
- [Bastion Host Preperation](#bastion-host-preperation)
- [Hostname](#hostname)
- [RHEL Subscription Manager](#rhel-subscription-manager)
- [Networking Bastion-Host](#networking-bastion-host)
- [SSH](#ssh)
- [Command Line Interfaces (CLIs)](#command-line-interfaces-clis)
- [Install Podman and Nmstate](#install-podman-and-nmstate)
- [Installing Podman Offline](#installing-podman-offline)
- [Installing the Mirror Registry on the Bastion Host](#installing-the-mirror-registry-on-the-bastion-host)
- [Prerequisites](#prerequisites)
- [Validating the installation](#validating-the-installation)
- [Login into the Mirror Registry](#login-into-the-mirror-registry)
- [Uninstalling the Mirror Registry](#uninstalling-the-mirror-registry)
- [Mirroring Images](#mirroring-images)
- [Creating the image set configuration](#creating-the-image-set-configuration)
- [unexpected status code 413 Request Entity Too Large](#unexpected-status-code-413-request-entity-too-large)
- [Installing a disconnected Cluster using the Agent Based Installer](#installing-a-disconnected-cluster-using-the-agent-based-installer)
- [Cluster Preperations](#cluster-preperations)
- [Configurations](#configurations)
- [Create Agent iso](#create-agent-iso)
- [Run a `httpd` webserver on the bastion to share the iso](#run-a-httpd-webserver-on-the-bastion-to-share-the-iso)
- [Using Operator Lifecycle Manager in disconnected environments](#using-operator-lifecycle-manager-in-disconnected-environments)
- [Troubleshooting](#troubleshooting)
- [Networking](#networking)
- [Logs](#logs)
- [Cluster Status validations](#cluster-status-validations)
- [Firewall is blocking images from pulling](#firewall-is-blocking-images-from-pulling)

---

## How it works

You can use a mirror registry for disconnected installations and to ensure that your clusters only use container images that satisfy your organization’s controls on external content. Before you install a cluster on infrastructure that you provision in a disconnected environment, you must mirror the required container images into that environment. To mirror container images, you must have a registry for mirroring.

## Connected Mirroring vs Disconnected Mirroring

**Connected Mirroring** is if you have a host that can access both the internet and your mirror registry, but not your cluster nodes, you can directly mirror the content from that machine.

**Disconnected Mirroring** is if you have no such host, you must mirror the images to a file system and then bring that host or removable media into your restricted environment.

## Bastion Host Preperation

### Hostname

Make sure that your `hostname` is set correctly!

```code
hostnamectl
```

The hostname must be a fqdn!

```code
sudo hostnamectl set-hostname rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com
sudo reboot
```

### RHEL Subscription Manager

Configure RHEL Subscription Manager:

`sudo subscription-manager register --username --password '' --auto-attach`

### Networking Bastion-Host

Setup a Bastion Host with two nics. One is connected to the "internet-zone" and the other one to the disconnected network.

![network-diagram](assets/openshift-mirror-registry-diagramm.png)

Configure the interfaces accordingly:

Configure the interface which has internet connection:

```code
nmcli con show

NAME UUID TYPE DEVICE
System eth0 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03 ethernet eth0
lo dd314177-6d3f-4ad3-a1af-2875d094c193 loopback lo
Wired connection 1 c8d40ef7-3d02-3ba5-b047-dacd6d013b24 ethernet --
```

```bash
nmcli con mod "System eth0" \
ipv4.addresses 10.32.96.145/20 \
ipv4.gateway 10.32.111.254 \
ipv4.dns "10.32.96.1,10.32.96.31" \
ipv4.method manual
```

Configure the interface which is connected to the disconnected network:

```bash
nmcli con mod "Wired connection 1" \
ipv4.addresses 192.168.69.208/24 \
ipv4.method manual
```

Bring both interfaces up:

```code
nmcli dev reapply eth0 && nmcli dev reapply eth1

Connection successfully reapplied to device 'eth0'.
Connection successfully reapplied to device 'eth1'.
```

Alternatively:

```code
nmcli con up "System eth0" && nmcli con up "Wired connection 1"

Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/25)
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/26)
```

Check the new config:

```code
ip -br a
lo UNKNOWN 127.0.0.1/8 ::1/128
eth0 UP 10.32.96.145/20 2620:52:0:2060:d8:6dff:fe0f:3ed3/64 fe80::d8:6dff:fe0f:3ed3/64
eth1 UP 192.168.69.208/24 fe80::a0eb:9896:d4e1:3fbb/64
```

The configuration is stored under:

```code
nmcli -f NAME,UUID,FILENAME con show
NAME UUID FILENAME
System eth0 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03 /etc/sysconfig/network-scripts/ifcfg-eth0
Wired connection 1 c8d40ef7-3d02-3ba5-b047-dacd6d013b24 /etc/NetworkManager/system-connections/Wired connection 1.nmconnection
lo dd314177-6d3f-4ad3-a1af-2875d094c193 /run/NetworkManager/system-connections/lo.nmconnection
```

Readable configuration:

```bash
less /etc/sysconfig/network-scripts/ifcfg-eth0

# Created by cloud-init automatically, do not edit.
#
AUTOCONNECT_PRIORITY=120
BOOTPROTO=none
DEVICE=eth0
HWADDR=02:D8:6D:0F:3E:D3
IPV6INIT=yes
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
PROXY_METHOD=none
BROWSER_ONLY=no
IPADDR=10.32.96.145
PREFIX=20
GATEWAY=10.32.111.254
DNS1=10.32.96.1
DNS2=10.32.96.31
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME="System eth0"
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03
```

And the other interface:

```bash
less /etc/NetworkManager/system-connections/Wired\ connection\ 1.nmconnection

[connection]
id=Wired connection 1
uuid=c8d40ef7-3d02-3ba5-b047-dacd6d013b24
type=ethernet
autoconnect-priority=-999
interface-name=eth1

[ethernet]

[ipv4]
address1=192.168.69.208/24
method=manual

[ipv6]
addr-gen-mode=default
method=auto

[proxy]
```

Enable IP Forwarding: `sysctl -w net.ipv4.ip_forward=1`

This allows the VM to forward packets between interfaces.

Make it permanently:

```bash
vi /etc/sysctl.conf
net.ipv4.ip_forward = 1
```

Apply the changes: `sysctl -p`

On Red Hat Enterprise Linux, `firewall-cmd` is provided by the firewalld package. It is not guaranteed to be installed in minimal VM images.

Check whether firewalld is installed:

```code
rpm -q firewalld
```

If it is missing, install and enable it:

```code
sudo dnf install -y firewalld
sudo systemctl enable --now firewalld
```

Allow forwarding:

`firewall-cmd --permanent --add-forward-port=port=22:proto=tcp:toport=22`

Add trusted zones for both interfaces:

```bash
firewall-cmd --permanent --zone=public --add-interface=eth0
firewall-cmd --permanent --zone=trusted --add-interface=eth1
```

Check the configs:

```code
firewall-cmd --get-active-zones
public
interfaces: eth0
trusted
interfaces: eth1
```

Also:

```code
firewall-cmd --zone=public --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: eth0
sources:
services: cockpit dhcpv6-client ssh
ports:
protocols:
forward: yes
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
```

```code
firewall-cmd --zone=trusted --list-all
trusted (active)
target: ACCEPT
icmp-block-inversion: no
interfaces: eth1
sources:
services:
ports:
protocols:
forward: yes
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
```

We will need to open up port 8443 for Quay:

```code
firewall-cmd --add-port 8443/tcp --permanent
```

Reload firewall rules: `firewall-cmd --reload`

Configure Routing (if needed).

If devices in 192.168.69.0/24 need internet access via the VM, you need NAT:

```bash
sudo firewall-cmd --permanent --add-masquerade
sudo firewall-cmd --reload
```

Check IP forwarding: `cat /proc/sys/net/ipv4/ip_forward`

### SSH

Configure `ssh`:

`cat ~/.ssh/id_ed25519.pub | ssh rguske@rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys && chmod 700 ~/.ssh"`

- Generating an SSH key pair on your Bastion-Host. You can use this key pair to authenticate into the OpenShift Container Platform cluster’s nodes after it is deployed.

`ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_ed25519`

### Command Line Interfaces (CLIs)

On the bastion host, download the necessary cli's from [Homepage](https://console.redhat.com/openshift/downloads):

You could use `curl -LO ` for it:

OpenShift Installer: `curl -LO https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.21.11/openshift-install-rhel9-amd64.tar.gz`

OpenShift Client: `curl -LO https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.21.11/openshift-client-linux-amd64-rhel9-4.21.11.tar.gz`

The "mirror" plugin for the OpenShift CLI client (oc) controls the process of mirroring all relevant container image for a full disconnected OpenShift installation in a central, declarative tool. Learn more(new window or tab)

RHEL 9 is FIPS compatible; RHEL 8 is non-FIPS compatible.

OpenShift Mirror CLI: `curl -LO https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/latest/oc-mirror.rhel9.tar.gz`

Download and install a local, minimal single instance deployment of Red Hat Quay to aid bootstrapping the first disconnected cluster.

OpenShift Tiny Quay Registry: `curl -LO https://mirror.openshift.com/pub/cgw/mirror-registry/latest/mirror-registry-amd64.tar.gz`

This helps untaring the packages:

```code
alias untar='tar -zxvf'
```

This is how it should like if you've downloaded all:

```code
tree
.
├── clis
│   ├── oc-mirror.rhel9.tar.gz
│   ├── openshift-client-linux-amd64-rhel9-4.21.11.tar.gz
│   └── openshift-install-rhel9-amd64.tar.gz
└── mirror-registry
└── mirror-registry-amd64.tar.gz
```

Unpack the `.gz` files, except execution-environment.tar, image-archive.tar and sqlite3.tar of the folder mirror-registry and move them into `/usr/local/bin`:

```code
sudp mv {kubectl,oc,oc-mirror,openshift-install-fips} /use/local/bin
```

Apply rights:

```code
sudo chown -R $USER /usr/local/bin/{kubectl,oc,oc-mirror,openshift-install-fips}
sudo chmod +x /usr/local/bin/{kubectl,oc,oc-mirror,openshift-install-fips}
```

```code
tree /usr/local/bin
/usr/local/bin
├── execution-environment.tar
├── firstboot-network-firewall.sh
├── image-archive.tar
├── kubectl
├── mirror-registry
├── oc
├── oc-mirror
├── openshift-install-fips
└── sqlite3.tar

0 directories, 9 file
```

If /usr/local/bin isn't included in the `$PATH`, run
`export PATH=/usr/local/bin:$PATH`

### Install Podman and Nmstate

In order to run the mirror registry, the bastion host needs a container-runtime installed.

Podman is the runtime of choice and is included in the `container-tools` package.

Install it via `sudo dnf install container-tools -y`.

The installer also uses `nmstatectl` for the creation of the agent.iso. Install it via `sudo dnf install nmstate -y`. Otherwise, you'll get the error:

```code
FATAL * failed to validate network yaml for host 0, install nmstate package, exec: "nmstatectl": executable file not found in $PATH
```

### Installing Podman Offline

1. Mount a RHEL installation iso file to the system (VM or BM via Board Management Controller).

Check e.g. with `lsblk` for the disconnected "cdrom" (iso) device:

```bash
[root@mirror-rguske ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 120G 0 disk
├─sda1 8:1 0 600M 0 part /boot/efi
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 118.4G 0 part
├─rhel-root 253:0 0 70G 0 lvm /
├─rhel-swap 253:1 0 7.9G 0 lvm [SWAP]
└─rhel-home 253:2 0 40.5G 0 lvm /home
sr0 11:0 1 11G 0 rom
```

2. Create a folder in which the iso content will be accessible:

`mkdir -p /mnt/rhel-iso`

3. Mount the connected iso accordingly:

`mount -o loop /path/to/rhel.iso /mnt/rhel-iso`

Example in my case with a VM: `mount -o loop /dev/sr0 /mnt/rhel-iso`

4. Create a Local Repository:

```code
cat < The OpenShift image registry cannot be used as the target registry because it does not support pushing without a tag, which is required during the mirroring process.

Install the mirror registry:

At this point it is important to understand that my bastion is dual-homed.

```code
dig +short rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com
10.32.96.145
dig +short rguske-rhel9-disco-bastion.disco.local
192.168.69.208
```

In this scenario it is important to use the DNS record which points to the IP in the disco subnet.

```code
[root@bastion-rguske mirror-registry]# mirror-registry install --quayHostname rguske-rhel9-disco-bastion.disco.local --quayRoot '/home/$USER/downloads/mirror-registry/root' --initPassword 'r3dh4t1!' --verbose

__ __
/ \ / \ ______ _ _ __ __ __
/ /\ / /\ \ / __ \ | | | | / \ \ \ / /
/ / / / \ \ | | | | | | | | / /\ \ \ /
\ \ \ \ / / | |__| | | |__| | / ____ \ | |
\ \/ \ \/ / \_ ___/ \____/ /_/ \_\ |_|
\__/ \__/ \ \__
\___\ by Red Hat
Build, Store, and Distribute your Containers

INFO[2026-04-22 05:16:03] Install has begun
DEBU[2026-04-22 05:16:03] Ansible Execution Environment Image: quay.io/quay/mirror-registry-ee:latest
DEBU[2026-04-22 05:16:03] Pause Image: registry.access.redhat.com/ubi8/pause:8.10-5
DEBU[2026-04-22 05:16:03] Quay Image: registry.redhat.io/quay/quay-rhel8:v3.12.14
DEBU[2026-04-22 05:16:03] Redis Image: registry.redhat.io/rhel8/redis-6:1-1766406130
INFO[2026-04-22 05:16:03] Found execution environment at /usr/local/bin/execution-environment.tar
INFO[2026-04-22 05:16:03] Loading execution environment from execution-environment.tar
DEBU[2026-04-22 05:16:03] Importing execution environment with command: /bin/bash -c /usr/bin/podman image import \
--change 'ENV PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' \
--change 'ENV HOME=/home/runner' \
--change 'ENV container=oci' \
--change 'ENTRYPOINT=["entrypoint"]' \
--change 'WORKDIR=/runner' \
--change 'EXPOSE=6379' \
--change 'VOLUME=/runner' \
--change 'CMD ["ansible-runner", "run", "/runner"]' \
- quay.io/quay/mirror-registry-ee:latest < /usr/local/bin/execution-environment.tar
Getting image source signatures
Copying blob 159de7f3f142 done |
Copying config 3055d6ebc0 done |
Writing manifest to image destination
sha256:3055d6ebc0dd81d1b676e94a7c1c06eef2c94ddc5abe2f197147b013184afd81
INFO[2026-04-22 05:16:13] Detected an installation to localhost
INFO[2026-04-22 05:16:13] Did not find SSH key in default location. Attempting to set up SSH keys.
INFO[2026-04-22 05:16:13] Generating SSH Key
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/quay_installer
Your public key has been saved in /root/.ssh/quay_installer.pub

[...]

TASK [mirror_appliance : Create init user] ***********************************************************************************************************************************************************************************************************************************************************************
included: /runner/project/roles/mirror_appliance/tasks/create-init-user.yaml for rguske@rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com

TASK [mirror_appliance : Creating init user at endpoint https://rguske-rhel9-disco-bastion.disco.local:8443/api/v1/user/initialize] ******************************************************************************************************************************************************************************
ok: [rguske@rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com]

TASK [mirror_appliance : Enable lingering for systemd user processes] ********************************************************************************************************************************************************************************************************************************************
changed: [rguske@rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com]

PLAY RECAP *******************************************************************************************************************************************************************************************************************************************************************************************************
rguske@rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com : ok=49 changed=29 unreachable=0 failed=0 skipped=15 rescued=0 ignored=0

INFO[2026-04-24 10:18:34] Quay installed successfully, config data is stored in /home/$USER/downloads/mirror-registry/root
INFO[2026-04-24 10:18:34] Quay is available at https://rguske-rhel9-disco-bastion.disco.local:8443 with credentials (init, r3dh4t1!)
```

Alternative customizations examples:

1.: `/mirror-registry install --quayHostname mirror.example.org --sslKey tls.key --targetHostname internal.mirror --quayRoot /var/mirror-registry --initPassword changeme`

Source: [Mirror registry for Red Hat OpenShift flags](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/disconnected_environments/installing-mirroring-creating-registry).

### Validating the installation

Validating the endpoint using `curl`:

```code
curl -k https://rguske-rhel9-disco-bastion.disco.local:8443/health/instance
{"data":{"services":{"auth":true,"database":true,"disk_space":true,"registry_gunicorn":true,"service_key":true,"web_gunicorn":true}},"status_code":200}
```

Checking the certificate:

```code
echo | openssl s_client -connect rguske-rhel9-disco-bastion.disco.local:8443 -showcerts
Connecting to 192.168.69.208
CONNECTED(00000003)
depth=1 C=US, ST=VA, L=New York, O=Quay, OU=Division, CN=rguske-rhel9-disco-bastion.disco.local
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=1 C=US, ST=VA, L=New York, O=Quay, OU=Division, CN=rguske-rhel9-disco-bastion.disco.local
verify return:1
depth=0 CN=quay-enterprise
verify return:1
---
Certificate chain
0 s:CN=quay-enterprise
i:C=US, ST=VA, L=New York, O=Quay, OU=Division, CN=rguske-rhel9-disco-bastion.disco.local
a:PKEY: RSA, 2048 (bit); sigalg: sha256WithRSAEncryption
v:NotBefore: Apr 24 14:17:18 2026 GMT; NotAfter: Apr 15 14:17:18 2027 GMT
-----BEGIN CERTIFICATE-----

[...]
```

Also, validate the certificate which we are going trust on our bastion host.

```code
openssl x509 -in ~/downloads/mirror-registry/root/quay-config/ssl.cert -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
16:06:dd:b0:be:99:47:81:76:52:85:9c:15:1e:76:0d:ab:99:35:ea
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, ST=VA, L=New York, O=Quay, OU=Division, CN=rguske-rhel9-disco-bastion.disco.local
Validity
Not Before: Apr 24 14:17:18 2026 GMT
Not After : Apr 15 14:17:18 2027 GMT
[...]
```

Systemd auto-start is also configured:

```code
sudo systemctl list-units --type service | grep quay
quay-app.service loaded active running Quay Container
quay-pod.service loaded active exited Infra Container for Quay
quay-redis.service loaded active running Redis Podman Container for Quay
```

### Login into the Mirror Registry

```code
podman login -u init -p 'r3dh4t1!' https://rguske-rhel9-disco-bastion.disco.local:8443 --tls-verify=false
```

It is also possible without using the option `--tls-verify=false` by trusting the newly created certificates which are stored in `root` / `quay-config`:

```code
tree
.
├── pause.tar
├── quay.tar
├── redis.tar
└── root
├── quay-config
│   ├── config.yaml
│   ├── openssl.cnf
│   ├── ssl.cert
│   ├── ssl.csr
│   └── ssl.key
└── quay-rootCA
├── rootCA.key
├── rootCA.pem
└── rootCA.srl
```

Copy the certs:

`sudo cp ~/downloads/mirror-registry/root/quay-config/ssl.cert /etc/pki/ca-trust/source/anchors/`

```code
tree /etc/pki/ca-trust/source/anchors/
└── ssl.cert

0 directories, 1 file
```

Update the trust:

`update-ca-trust`

Logout:

```code
podman logout https://rguske-rhel9-disco-bastion.disco.local:8443

Removed login credentials for rguske-rhel9-disco-bastion.disco.local:8443
```

Login again:

```code
podman login -u init -p 'r3dh4t1!' 'https://rguske-rhel9-disco-bastion.disco.local:8443'

Login Succeeded!
```

### Uninstalling the Mirror Registry

`mirror-registry uninstall`

## Mirroring Images

> You must have access to the internet to obtain the necessary container images. In this procedure, you place your mirror registry on a mirror host that has access to both your network and the internet. If you do not have access to a mirror host, use the [Mirroring Operator catalogs](https://docs.redhat.com/en/documentation/openshift_container_platform/4.17/html-single/disconnected_environments/index#olm-mirror-catalog_installing-mirroring-installation-images) for use with disconnected clusters procedure to copy images to a device you can move across network boundaries with.

Procedure and prerequisites:

- You configured a mirror registry to use in your disconnected environment.
- You identified an image repository location on your mirror registry to mirror images into.
- You provisioned a mirror registry account that allows images to be uploaded to that image repository.
- You have write access to the mirror registry.

Obtain your [pull secret from Red Hat OpenShift Cluster Manager](https://console.redhat.com/openshift/install/pull-secret) and paste the json content into $XDG_RUNTIME_DIR/containers/auth.json. Be careful! It must be in `json`.

Make a copy of your pull secret in JSON format by running the following command:

Paste the content in the file pull-scret. Then:

`cat ./pull-secret | jq . > $(pwd)/pull-secret.json`

Replace the existing `auth.json` file in $XDG_RUNTIME_DIR/containers/

```code
sudo mv pull-secret.json $XDG_RUNTIME_DIR/containers/auth.json
```

Next up is to generate the base64-encoded user name and password or token for your mirror registry by running the following command:

`echo -n ':' | base64 -w0`

For and , specify the user name and password that you configured for your registry.

Example:

`echo -n 'init:r3dh4t1!' | base64 -w0`

Edit the JSON file and add a section that describes your registry to it:

```json
"auths": {
"rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com:8443": {
"auth": "aW5pdDpyM2RoNHQxIQ==",
"email": "rguske@redhat.com"
},
"cloud.openshift.com": {

[...]
```

## Creating the image set configuration

Create an `ImageSetConfiguration` YAML file and modify it to include your required images.

List the available Operator using e.g.:

```code
oc mirror list operators --catalogs --version=4.21 --v1
```

```yaml
tee imagesetconfiguration.yaml > /dev/null <<'EOF'
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
platform:
channels:
- name: stable-4.21
type: ocp
shortestPath: true
minVersion: 4.21.10
maxVersion: 4.21.11
graph: true
operators:
- catalog: registry.redhat.io/redhat/redhat-operator-index:v4.21
packages:
## cincinnati-operator:v1
- name: cincinnati-operator
channels:
- name: v1
minVersion: '5.0.3'
# maxVersion: '5.0.3'
## kubernetes-nmstate-operator:stable
- name: kubernetes-nmstate-operator
channels:
- name: 'stable'
minVersion: '4.21.0-202604080925'
# maxVersion: '4.17.0-202502120148'
## kubevirt-hyperconverged:stable
- name: kubevirt-hyperconverged
channels:
- name: stable
minVersion: '4.21.3'
# maxVersion: '4.17.4'
## metallb-operator:stable
- name: metallb-operator
channels:
- name: stable
minVersion: '4.21.0-202604140043'
# maxVersion: 'v4.17.0'
## web-terminal:fast
- name: web-terminal
channels:
- name: fast
minVersion: '1.16.0'
# maxVersion: 'v1.15.0'
## web-terminal relies on devworkspace-operator
# - name: devworkspace-operator
# channels:
# - name: fast
# minVersion: '0.40-1776457293'
## OpenShift Data Foundation
- name: odf-operator
channels:
- name: stable-4.21
minVersion: '4.21.2-rhodf'
## OpenShift Local Storage Operator
- name: local-storage-operator
channels:
- name: stable
minVersion: '4.21.0-202604200440'
additionalImages:
- name: registry.redhat.io/ubi8/ubi:latest
- name: registry.redhat.io/rhel9/rhel-guest-image:latest
- name: quay.io/rhn_support_sreber/curl:latest
# Important for KMM & GPFS Build
- name: registry.redhat.io/ubi9/ubi-minimal:latest
- name: registry.redhat.io/ubi9/ubi@sha256:20f695d2a91352d4eaa25107535126727b5945bff38ed36a3e59590f495046f0
- name: quay.io/rguske/vddk@sha256:26d07e11f7f8dcca263e83a1d942fe9274c90418c5bfc17fad88b61ddabf95ed
- name: quay.io/rguske/simple-web-app@sha256:f1c474d0b214975d2fb95d14967b620daa0cdbef094ee509fec1659d55c3a6de
# Virtualization Images
- name: quay.io/containerdisks/centos-stream:9
- name: quay.io/containerdisks/fedora:latest
EOF
```

Ensure that clis are in your $PATH. Otherwise `export PATH=/usr/local/bin:$PATH`.

The oc-mirror plugin v2 automatically generates the following custom resources:

- `ImageDigestMirrorSet` (IDMS)
Handles registry mirror rules when using image digest pull specifications. Generated if at least one image of the image set is mirrored by digest.
- `ImageTagMirrorSet` (ITMS)
Handles registry mirror rules when using image tag pull specifications. Generated if at least one image from the image set is mirrored by tag.
- `CatalogSource`
Retrieves information about the available Operators in the mirror registry. Used by Operator Lifecycle Manager (OLM) Classic.
- `ClusterCatalog`
Retrieves information about the available cluster extensions (which includes Operators) in the mirror registry. Used by OLM v1.
- `UpdateService`
Provides update graph data to the disconnected environment. Used by the OpenShift Update Service.

Mirror the images from the specified image set configuration to the disk by running the following command:

```code
oc mirror -c $(pwd)/openshift/imagesetconfiguration.yaml file://$(pwd) --v2
```

```code
oc-mirror -c $(pwd)/openshift/imagesetconfiguration.yaml file:///home/rguske/openshift/mirror --v2
```

Result:

```code
2026/04/23 12:55:55 [INFO] : === Results ===
2026/04/23 12:55:55 [INFO] : ✓ 193 / 193 release images mirrored successfully
2026/04/23 12:55:55 [INFO] : ✓ 95 / 95 operator images mirrored successfully
2026/04/23 12:55:55 [INFO] : ✓ 7 / 7 additional images mirrored successfully
2026/04/23 12:55:55 [INFO] : 📦 Preparing the tarball archive...
2026/04/23 12:58:48 [INFO] : mirror time : 19m53.229903012s
2026/04/23 12:58:48 [INFO] : 👋 Goodbye, thank you for using oc-mirror
```

```code
ls -ltr
total 61212944
drwxr-xr-x. 12 rguske rguske 4096 Apr 23 12:39 working-dir
-rw-r--r--. 1 rguske rguske 62682043392 Apr 23 12:58 mirror_000001.tar
```

Upload to your mirror registry:

```code
oc mirror -c $(pwd)/imagesetconfiguration.yaml --from file://$(pwd)/mirror/ docker://rguske-rhel9-disco-bastion.disco.local:8443/disco --v2
```

Results:

```code
2026/04/24 11:34:43 [INFO] : === Results ===
2026/04/24 11:34:43 [INFO] : ✓ 193 / 193 release images mirrored successfully
2026/04/24 11:34:43 [INFO] : ✓ 95 / 95 operator images mirrored successfully
2026/04/24 11:34:43 [INFO] : ✓ 7 / 7 additional images mirrored successfully
2026/04/24 11:34:43 [INFO] : 📄 Generating IDMS file...
2026/04/24 11:34:43 [INFO] : /home/rguske/openshift/mirror/working-dir/cluster-resources/idms-oc-mirror.yaml file created
2026/04/24 11:34:43 [INFO] : 📄 Generating ITMS file...
2026/04/24 11:34:43 [INFO] : /home/rguske/openshift/mirror/working-dir/cluster-resources/itms-oc-mirror.yaml file created
2026/04/24 11:34:43 [INFO] : 📄 Generating CatalogSource file...
2026/04/24 11:34:43 [INFO] : /home/rguske/openshift/mirror/working-dir/cluster-resources/cs-redhat-operator-index-v4-21.yaml file created
2026/04/24 11:34:43 [INFO] : 📄 Generating ClusterCatalog file...
2026/04/24 11:34:43 [INFO] : /home/rguske/openshift/mirror/working-dir/cluster-resources/cc-redhat-operator-index-v4-21.yaml file created
2026/04/24 11:34:43 [INFO] : 📄 Generating Signature Configmap...
2026/04/24 11:34:43 [INFO] : /home/rguske/openshift/mirror/working-dir/cluster-resources/signature-configmap.json file created
2026/04/24 11:34:43 [INFO] : /home/rguske/openshift/mirror/working-dir/cluster-resources/signature-configmap.yaml file created
2026/04/24 11:34:43 [INFO] : 📄 Generating UpdateService file...
2026/04/24 11:34:43 [INFO] : /home/rguske/openshift/mirror/working-dir/cluster-resources/updateService.yaml file created
2026/04/24 11:34:43 [INFO] : mirror time : 50m26.676667165s
2026/04/24 11:34:43 [INFO] : 👋 Goodbye, thank you for using oc-mirror
```

### unexpected status code 413 Request Entity Too Large

If this hits you, try using the following options:

```code
oc mirror -c $(pwd)/imagesetconfiguration.yaml --from file://$(pwd)/mirror/ docker://rguske-rhel9-disco-bastion.
disco.local:8443/disco --image-timeout 2h --parallel-images=10 --parallel-layers=10 --retry-times=5 --retry-delay=10s --v2
```

Documented in [oc-mirror v2 fails with context deadline exceeded when mirroring large images to a local registry in RHOCP 4](https://access.redhat.com/solutions/7130341).

## Installing a disconnected Cluster using the Agent Based Installer

> When you use a disconnected mirror registry, you must add the certificate file that you created previously for your mirror registry to the additionalTrustBundle field of the install-config.yaml file.

Workflow:

- Create mirror registry content (oc mirror) ✅
- Create installation assets (install-config.yaml)
- Create the cluster
- Apply mirror configuration to the new cluster

### Cluster Preperations

Network 192.168.69.0/24
DNS: 192.168.69.6
GW: 192.168.69.254
VLAN ID 69

Collecting the necessary nic information:

| name | nic | mac | ipv4 | comment |
|---|---|---|---|---|
| rguske-ocp42-disco-1 | enp1s0 | 02:d8:6d:0f:3e:dc | 192.168.69.202 | Node 1 |
| rguske-ocp42-disco-2 | enp1s0 | 02:d8:6d:0f:3e:dd | 192.168.69.203 | Node 2 |
| rguske-ocp42-disco-3 | enp1s0 | 02:d8:6d:0f:3e:de | 192.168.69.204 | Node 3 |

BaseDomain: disco.local

## Configurations

Create the `agent-config.yaml as well as the install-config.yaml`

```code
tree rguske-ocp42-disco/
rguske-ocp42-disco/
└── conf
├── agent-config.yaml
└── install-config.yaml
```

```yaml
cat > agent-config.yaml << EOF
apiVersion: v1beta1
kind: AgentConfig
metadata:
name: rguske-ocp42-disco
rendezvousIP: 192.168.69.202
hosts:
- hostname: rguske-ocp42-disco-1.disco.local
role: master
interfaces:
- name: enp1s0
macAddress: 02:d8:6d:0f:3e:dc
networkConfig:
interfaces:
- name: enp1s0
type: ethernet
state: up
mac-address: 02:d8:6d:0f:3e:dc
ipv4:
enabled: true
address:
- ip: 192.168.69.202
prefix-length: 24
dhcp: false
dns-resolver:
config:
server:
- 192.168.69.6
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 192.168.69.254
next-hop-interface: enp1s0
table-id: 254
- hostname: rguske-ocp42-disco-2.disco.local
role: master
interfaces:
- name: enp1s0
macAddress: 02:d8:6d:0f:3e:dd
networkConfig:
interfaces:
- name: enp1s0
type: ethernet
state: up
mac-address: 02:d8:6d:0f:3e:dd
ipv4:
enabled: true
address:
- ip: 192.168.69.203
prefix-length: 24
dhcp: false
dns-resolver:
config:
server:
- 192.168.69.6
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 192.168.69.254
next-hop-interface: enp1s0
table-id: 254
- hostname: rguske-ocp42-disco-3.disco.local
role: master
interfaces:
- name: enp1s0
macAddress: 02:d8:6d:0f:3e:de
networkConfig:
interfaces:
- name: enp1s0
type: ethernet
state: up
mac-address: 02:d8:6d:0f:3e:de
ipv4:
enabled: true
address:
- ip: 192.168.69.204
prefix-length: 24
dhcp: false
dns-resolver:
config:
server:
- 192.168.69.6
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 192.168.69.254
next-hop-interface: enp1s0
table-id: 254
EOF
```

The ssl certificate of the mirror-registry which will be used in the `install-config.yaml` can be found within the `mirror-registry/root/quay-rootCA` folder. Example: `/home/rguske/downloads/mirror-registry/root/quay-rootCA`

```json
{
"auths": {
"rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com:8443": {
"auth": "aW5pdDpyM2RoNHQxIQ==",
"email": "rguske@redhat.com"
}
}
}
```

```yaml
cat > install-config.yaml << EOF
apiVersion: v1
baseDomain: disco.local
ImageDigestSources:
- mirrors:
- rguske-rhel9-disco-bastion.disco.local:8443/disco/openshift/release
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
- mirrors:
- rguske-rhel9-disco-bastion.disco.local:8443/disco/openshift/release-images
source: quay.io/openshift-release-dev/ocp-release
additionalTrustBundle: |
-----BEGIN CERTIFICATE-----
MIIEHDCCAwSgAwIBAgIUFY/Z+WmgJ+8SIREIa3Cl3FRj9jYwDQYJKoZIhvcNAQEL
BQAwgYAxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJWQTERMA8GA1UEBwwITmV3IFlv
cmsxDTALBgNVBAoMBFF1YXkxETAPBgNVBAsMCERpdmlzaW9uMS8wLQYDVQQDDCZy
Z3Vza2UtcmhlbDktZGlzY28tYmFzdGlvbi5kaXNjby5sb2NhbDAeFw0yNjA0MjQx
NDE3MTZaFw0yOTAyMTExNDE3MTZaMIGAMQswCQYDVQQGEwJVUzELMAkGA1UECAwC
...
ZyYMLJyR83M5sD7sVbbuSOkYgNt20ZdIcqigIkyABRkqcahC7kOypXXJbhkj3fYL
kyGukgbJRF96hCB9oO8bW3evact/P40arsjHT6qKRIZf0kKm7CYUVRjI4+jlz+oV
n7iB3Rs8P16UvuFB2LfWmyNfuu21InZhXLmJ+rZJc0qnpq6Rm8iXAq0n8L5ycCHc
gPt4JJQZJ8JP6bSREgAhqfNSngfLj73O1+S2fuN7i3mCQEv0UajEhQgHQtcZ6r1C
-----END CERTIFICATE-----
compute:
- name: worker
replicas: 0
controlPlane:
name: master
replicas: 3
metadata:
name: rguske-ocp42-disco
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 192.168.69.0/24
serviceNetwork:
- 172.30.0.0/16
networkType: OVNKubernetes
platform:
baremetal:
apiVIPs:
- 192.168.69.200
ingressVIPs:
- 192.168.69.201
fips: false
pullSecret: '{
"auths": {
"rguske-rhel9-disco-bastion.disco.local:8443": {
"auth": "aW5pdDpyM2RoNHQxIQ==",
"email": "rguske@redhat.com"
}
}
}'
sshKey: 'ssh-ed25519 AAAAC3NzaC1lZ...VzGQ/Ur5Ek0v9gF rguske@rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com'
EOF
```

Create a new `openshift-install-fips` binary which only points to your mirror-registry.

```code
export LOCAL_SECRET_JSON='/home/rguske/openshift/new-pull-secret.json' \
export LOCAL_REGISTRY='rguske-rhel9-disco-bastion.disco.local:8443' \
export LOCAL_REPOSITORY='disco/openshift/release-images' \
export OCP_RELEASE='4.21.10' \
export ARCHITECTURE='x86_64'
```

```code
oc adm release extract -a ${LOCAL_SECRET_JSON} --idms-file=/home/rguske/openshift/mirror/working-dir/cluster-resources/idms-oc-mirror.yaml --command=openshift-install-fips "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}"
```

```code
tree -L 1
.
├── downloads
├── oc-mirror-web-app
├── openshift
└── openshift-install-fips
```

Validate it:

```code
./openshift-install-fips version
./openshift-install-fips 4.21.10
built from commit 6285755d199e7aa7bf29db5fe6964ce7f3684ed9
release image rguske-rhel9-disco-bastion.disco.local:8443/disco/openshift/release-images@sha256:5d591a70c92a6dfa3b6b948ffe5e5eac7ab339c49005744006aa0dd9d6d98898
Release Image Architecture is unknown
release architecture unknown
default architecture amd64
```

This binary will be used for the creation of the agent.iso.

## Create Agent iso

Create the install-config.yaml and agent-install.yaml file.

Run `./openshift-install-fips agent create image --dir /home/rguske/openshift/rguske-ocp42-disco/conf/`

Example output :

```code
INFO Configuration has 3 master replicas, 0 arbiter replicas, and 0 worker replicas
WARNING The imageDigestSources configuration in install-config.yaml should have at least one source field matching the releaseImage value rguske-rhel9-disco-bastion.disco.local:8443/disco/openshift/release-images@sha256:5d591a70c92a6dfa3b6b948ffe5e5eac7ab339c49005744006aa0dd9d6d98898
INFO The rendezvous host IP (node0 IP) is 192.168.69.202
INFO Extracting base ISO from release payload
INFO Base ISO obtained from release and cached at [/home/rguske/.cache/agent/image_cache/coreos-x86_64.iso]
INFO Consuming Install Config from target directory
INFO Consuming Agent Config from target directory
INFO Generated ISO at /home/rguske/openshift/rguske-ocp42-disco/conf/agent.x86_64.iso.
```

If you still have problemes with the release-image reference use:

```code
export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE='rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com:8443/disco/openshift/release-images@sha256:5d591a70c92a6dfa3b6b948ffe5e5eac7ab339c49005744006aa0dd9d6d98898'
```

Mount the `agent.x86_64.iso` on the machines (BM or VM).

If you're running the bastion host on e.g. OpenShift Virtualization, you can use the following command in order to upload the ise to a pvc:

```code
VERSION=$(curl -s https://api.github.com/repos/kubevirt/kubevirt/releases/latest | grep tag_name | cut -d '"' -f 4)
curl -L -o virtctl https://github.com/kubevirt/kubevirt/releases/download/$VERSION/virtctl-$VERSION-linux-amd64
chmod +x virtctl
sudo mv virtctl /usr/local/bin/
```

Login into your existing OpenShift cluster, change into your project and upload the iso:

```code
virtctl image-upload pvc rguskeagentiso-disco --size 2Gi --storage-class coe-netapp-nas --access-mode ReadWriteMany --image-path ~/openshift/rguske-ocp42-disco/conf/agent.x86_64.iso --insecure
```

Boot the machines and wait until the installation is done.

Validate the installer progress using `./openshift-install-fips wait-for install-complete --dir /home/rguske/openshift/rguske-ocp42-disco/conf/ --log-level=debug`

## Run a `httpd` webserver on the bastion to share the iso

Depending on your environment, providing the created iso can be cumbersome.

One quick and easy way could be by making it downloadable via a webserver.

Install `httpd` on the bastion host.

```bash
dnf install httpd
sudo systemctl enable --now httpd
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --reload
```

Validate the service is running:

```bash
sudo ss -tuln | grep :80
curl -I http://localhost
sudo tail -f /var/log/httpd/error_log
```

Copy (`scp agent.x86_64.iso root@192.168.69.208:/root/download/`) the agent.iso from the mirror registry to the bastion host which has access to the target (ESXi server for example).

Copy the created iso into `/var/www/html/` on the bastion host.

Download the iso by using e.g. `wget http:///agent.x86_64.iso`.

Example:

```code
wget http://bastion-rguske.rguske.coe.muc.redhat.com/agent.x86_64.iso
Connecting to bastion-rguske.rguske.coe.muc.redhat.com (10.32.96.138:80)
saving to 'agent.x86_64.iso'
agent.x86_64.iso 21% ******************************************** | 261M 0:00:10 ETA
```

![disco-cluster-operator](assets/disco-cluster-operators.png)

## Using Operator Lifecycle Manager in disconnected environments

Docs: [Using Operator Lifecycle Manager in disconnected environments](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/disconnected_environments/olm-restricted-networks)

Disable the sources for the default catalogs by adding disableAllDefaultSources: true to the OperatorHub object:

```code
oc patch OperatorHub cluster --type json \
-p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'
```

Create a CatalogSource object that references your index image.

```yaml
oc create -f - <