https://github.com/mvallim/cloud-image-ubuntu-from-scratch
This procedure shows how to create a cloud image Ubuntu from scratch to run on Cloud environments (EC2, GCE, Azure, QEMU and VirtualBox).
https://github.com/mvallim/cloud-image-ubuntu-from-scratch
aws azure cloud ec2 gce google image qemu scratch virtualbox
Last synced: 5 months ago
JSON representation
This procedure shows how to create a cloud image Ubuntu from scratch to run on Cloud environments (EC2, GCE, Azure, QEMU and VirtualBox).
- Host: GitHub
- URL: https://github.com/mvallim/cloud-image-ubuntu-from-scratch
- Owner: mvallim
- License: gpl-3.0
- Created: 2020-01-25T21:07:17.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2024-09-15T18:16:27.000Z (almost 2 years ago)
- Last Synced: 2024-09-15T19:37:21.322Z (almost 2 years ago)
- Topics: aws, azure, cloud, ec2, gce, google, image, qemu, scratch, virtualbox
- Homepage: https://mvallim.github.io/cloud-image-ubuntu-from-scratch/
- Size: 845 KB
- Stars: 81
- Watchers: 5
- Forks: 16
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# How to create a Cloud image Ubuntu from scratch
This procedure shows how to create a cloud image Ubuntu from scratch to run on Cloud environments (EC2, GCE, Azure, QEMU, OpenStack and VirtualBox).
## Prerequisites (GNU/Linux Debian/Ubuntu)
Install applications we need to build the environment.
```shell
sudo apt-get install \
debootstrap \
qemu-utils \
qemu-system \
genisoimage
```
```shell
mkdir $HOME/cloud-image-ubuntu-from-scratch
```
## Create loop device
1. Access build directory
```shell
cd $HOME/cloud-image-ubuntu-from-scratch
```
2. Create empty virtual hard drive file (`30Gb`)
```shell
dd if=/dev/zero of=cloud-ubuntu-image.raw bs=1 count=0 seek=32212254720 status=progress
```
3. Create partitions on file
```shell
sed -e 's/\s*\([\+0-9a-zA-Z]*\).*/\1/' << EOF | sudo fdisk cloud-ubuntu-image.raw
o # clear the in memory partition table
n # new partition
p # primary partition
1 # partition number 1
# default - start at beginning of disk
+512M # 512 MB boot parttion
n # new partition
p # primary partition
2 # partion number 2
# default, start immediately after preceding partition
# default, extend partition to end of disk
a # make a partition bootable
1 # bootable partition is partition 1 -- /dev/loop0p1
p # print the in-memory partition table
w # write the partition table
q # and we're done
EOF
```
4. Start loop device
```shell
sudo losetup --show -fP cloud-ubuntu-image.raw
```
A similar output to this
```text
/dev/loop18
```
Check loop device
```shell
sudo losetup -l /dev/loop18
```
A similar output to this
```text
NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC
/dev/loop18 0 0 0 0 /home/scratch/cloud-image-ubuntu-from-scratch/cloud-ubuntu-image.raw 0 512
```
5. Check partitions on loop device
```shell
sudo fdisk -l /dev/loop18
```
A similar output to this
```text
Disk /dev/loop18: 30 GiB, 32212254720 bytes, 62914560 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x64e50bc1
Device Boot Start End Sectors Size Id Type
/dev/loop18p1 * 2048 1050623 1048576 512M 83 Linux
/dev/loop18p2 1050624 62914559 61863936 29.5G 83 Linux
```
## Format partitions loop device
1. Format device loop0p1 (/boot)
```shell
sudo mkfs.ext4 /dev/loop18p1
```
A similar output to this
```text
mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: done
Creating filesystem with 131072 4k blocks and 32768 inodes
Filesystem UUID: 08b9e410-eb6d-4ba7-8a83-c3c960bca98c
Superblock backups stored on blocks:
32768, 98304
Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done
```
2. Format device loop0p2 (/)
```shell
sudo mkfs.ext4 /dev/loop18p2
```
A similar output to this
```text
mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: done
Creating filesystem with 7732992 4k blocks and 1933312 inodes
Filesystem UUID: 1c8f95b7-ea0f-4b62-a6ca-2da0c42209e2
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
```
## Mount loop devices
1. Access build directory
```shell
cd $HOME/cloud-image-ubuntu-from-scratch
```
2. Create `chroot` directory
```shell
mkdir chroot
```
3. Mount `root` partition
```shell
sudo mount /dev/loop18p2 chroot/
```
4. Mount `boot` partition
First you need create directory...
```shell
sudo mkdir chroot/boot
```
... and mount `boot` partition
```shell
sudo mount /dev/loop18p1 chroot/boot
```
## Bootstrap and Configure Ubuntu
* Checkout bootstrap
```shell
sudo debootstrap \
--arch=amd64 \
--variant=minbase \
--components "main,universe" \
--include "ca-certificates,cron,iptables,isc-dhcp-client,libnss-myhostname,ntp,ntpdate,rsyslog,ssh,sudo,dialog,whiptail,man-db,curl,dosfstools,e2fsck-static" \
bionic \
$HOME/cloud-image-ubuntu-from-scratch/chroot \
http://us.archive.ubuntu.com/ubuntu/
```
> **debootstrap** is used to create a Debian base system from scratch, without requiring the availability of **dpkg** or **apt**. It does this by downloading .deb files from a mirror site, and carefully unpacking them into a directory which can eventually be **chrooted** into.
* Configure external mount points
```shell
sudo mount --bind /dev $HOME/cloud-image-ubuntu-from-scratch/chroot/dev
sudo mount --bind /run $HOME/cloud-image-ubuntu-from-scratch/chroot/run
```
As we will be updating and installing packages (grub among them), these mount points are necessary inside the chroot environment, so we are able to finish the installation without errors.
## Define chroot environment
*A chroot on Unix operating systems is an operation that changes the apparent root directory for the current running process and its children. A program that is run in such a modified environment cannot name (and therefore normally cannot access) files outside the designated directory tree. The term "chroot" may refer to the chroot system call or the chroot wrapper program. The modified environment is called a chroot jail.*
> Reference: https://en.wikipedia.org/wiki/Chroot
1. **Access chroot environment**
```shell
sudo chroot $HOME/cloud-image-ubuntu-from-scratch/chroot
```
2. **Configure mount points, home and locale**
```shell
mount none -t proc /proc
mount none -t sysfs /sys
mount none -t devpts /dev/pts
export HOME=/root
export LC_ALL=C
```
These mount points are necessary inside the chroot environment, so we are able to finish the installation without errors.
3. **Set a custom hostname**
```shell
echo "ubuntu-image" > /etc/hostname
```
4. **Configure apt sources.list**
```shell
cat < /etc/apt/sources.list
deb http://us.archive.ubuntu.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ bionic main restricted universe multiverse
deb http://us.archive.ubuntu.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ bionic-security main restricted universe multiverse
deb http://us.archive.ubuntu.com/ubuntu/ bionic-updates main restricted universe multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ bionic-updates main restricted universe multiverse
EOF
```
5. **Configure `fstab`**
```shell
cat < /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
#
/dev/sda2 / ext4 errors=remount-ro 0 1
/dev/sda1 /boot ext4 defaults 0 2
EOF
```
6. **Update indexes packages**
```shell
apt-get update
```
7. **Install systemd**
```shell
apt-get install -y systemd-sysv
```
> **systemd** is a system and service manager for Linux. It provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux control groups, maintains mount and automount points and implements an elaborate transactional dependency-based service control logic.
8. **Configure machine-id and divert**
```shell
dbus-uuidgen > /etc/machine-id
ln -fs /etc/machine-id /var/lib/dbus/machine-id
```
> The `/etc/machine-id` file contains the unique machine ID of the local system that is set during installation or boot. The machine ID is a single newline-terminated, hexadecimal, 32-character, lowercase ID. When decoded from hexadecimal, this corresponds to a 16-byte/128-bit value. This ID may not be all zeros.
```shell
dpkg-divert --local --rename --add /sbin/initctl
ln -s /bin/true /sbin/initctl
```
> **dpkg-divert** is the utility used to set up and update the list of diversions.
9. **Install packages needed for system**
```shell
apt-get install -y \
os-prober \
ifupdown \
network-manager \
locales \
build-essential \
module-assistant \
cloud-init \
grub-pc \
grub2 \
linux-generic
```
The next steps will appear, as a result of the packages that will be installed from the previous step, this will happen without anything having to be informed or executed.
1. Configure grub
2. Don’t select any options
3. Only confirm “Yes”
10. **Configure `interfaces`**
```shell
cat < /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
EOF
```
11. **Reconfigure packages**
1. Generate locales
```shell
dpkg-reconfigure locales
```
1. *Select locales*
2. *Select default locale*
2. Configure network-manager
```shell
cat < /etc/NetworkManager/NetworkManager.conf
[main]
rc-manager=none
plugins=ifupdown,keyfile
dns=systemd-resolved
[ifupdown]
managed=false
EOF
```
4. Reconfigure network-manager
```shell
dpkg-reconfigure network-manager
```
12. **Install `grub`**
1. Install
```shell
grub-install /dev/loop0
```
A similar output to this
```text
Installing for i386-pc platform.
Installation finished. No error reported.
```
2. Update grub configuration
```shell
update-grub
```
A similar output to this
```text
Sourcing file `/etc/default/grub'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.15.0-74-generic
Found initrd image: /boot/initrd.img-4.15.0-74-generic
Adding boot menu entry for EFI firmware configuration
done
```
2. Update packages
```shell
apt-get -y upgrade
```
## VirtualBox
If you plan to use this image in **VirtualBox**, install **VirtualBox Guest Additions**
1. Download VirtualBox Guest Additions
```shell
curl --progress-bar https://download.virtualbox.org/virtualbox/6.1.38/VBoxGuestAdditions_6.1.38.iso -o VBoxGuestAdditions_6.1.38.iso
```
2. Mount ISO
```shell
mount -o loop VBoxGuestAdditions_6.1.38.iso /mnt
```
3. Install
```shell
/mnt/VBoxLinuxAdditions.run --nox11
```
Output like this
```text
Verifying archive integrity... All good.
Uncompressing VirtualBox 6.1.38 Guest Additions for Linux........
VirtualBox Guest Additions installer
Copying additional installer modules ...
Installing additional modules ...
depmod: ERROR: could not open directory /lib/modules/4.19.0-6-amd64: No such file or directory
depmod: FATAL: could not search modules: No such file or directory
VirtualBox Guest Additions: Starting.
VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel
modules. This may take a while.
VirtualBox Guest Additions: To build modules for other installed kernels, run
VirtualBox Guest Additions: /sbin/rcvboxadd quicksetup
VirtualBox Guest Additions: or
VirtualBox Guest Additions: /sbin/rcvboxadd quicksetup all
VirtualBox Guest Additions: Kernel headers not found for target kernel
4.19.0-6-amd64. Please install them and execute
/sbin/rcvboxadd setup
modprobe vboxguest failed
The log file /var/log/vboxadd-setup.log may contain further information.
Running in chroot, ignoring request: daemon-reload
```
4. Generate modules inside `chroot` environment
```shell
ls -al /lib/modules
```
Output
```text
total 12
drwxr-xr-x 3 root root 4096 Jan 25 22:29 .
drwxr-xr-x 14 root root 4096 Jan 25 22:29 ..
drwxr-xr-x 5 root root 4096 Jan 25 22:29 4.15.0-74-generic
```
Use the same name listed before 4.15.0-74-generic
```shell
rcvboxadd quicksetup 4.15.0-74-generic
```
Output like this
```text
VirtualBox Guest Additions: Building the modules for kernel 4.15.0-74-generic.
update-initramfs: Generating /boot/initrd.img-4.15.0-74-generic
Warning: /sbin/fsck.vfat doesn't exist, can't install to initramfs, ignoring.
```
5. Umount and remove ISO
```shell
umount /mnt
rm -rf VBoxGuestAdditions_6.1.38.iso
```
6. Fix `vboxadd-service`
```shell
sed -i -e 's/ systemd-timesyncd.service//g' /lib/systemd/system/vboxadd-service.service
```
## Azure
If you plan to use this image in **Azure**, install the **Azure agent**
1. Install the latest package version
```shell
apt-get install walinuxagent
```
2. Ensure auto update is enabled
First, check to see if it is enabled:
```shell
cat /etc/waagent.conf | grep AutoUpdate.Enabled
```
Find 'AutoUpdate.Enabled'. If you see this output, it is enabled:
```text
# AutoUpdate.Enabled=y
```
3. To enable run:
```shell
sudo sed -i 's/# AutoUpdate.Enabled=n/AutoUpdate.Enabled=y/g' /etc/waagent.conf
```
## Cleanup the chroot environment
1. If you installed software, be sure to run
```shell
truncate -s 0 /etc/machine-id
```
2. Remove the diversion
```shell
rm /sbin/initctl
dpkg-divert --rename --remove /sbin/initctl
```
3. Clean up
```shell
apt-get clean
rm -rf /tmp/* ~/.bash_history
umount /proc
umount /sys
umount /dev/pts
export HISTSIZE=0
exit
```
## Unbind mount points
```shell
sudo umount $HOME/cloud-image-ubuntu-from-scratch/chroot/dev
sudo umount $HOME/cloud-image-ubuntu-from-scratch/chroot/run
```
## Umount loop partitions
```shell
sudo umount $HOME/cloud-image-ubuntu-from-scratch/chroot/boot
sudo umount $HOME/cloud-image-ubuntu-from-scratch/chroot
```
## Leave loop device
```shell
sudo losetup -D
```
## Conclusion
At the end the image produced is in `cloud-ubuntu-image.raw`.
Now you can use this raw image and import it into your favorite cloud.
Each cloud has a process for importing which we will not deal with here.
## Test image
1. **Convert `raw` image to `qcow2`**
```shell
qemu-img convert -f raw cloud-ubuntu-image.raw -O qcow2 ubuntu-image.qcow2
```
2. **Create a simple `user-data` intilizer `cloud-init`**
Generate hash passwd 'ubuntu'
```shell
openssl passwd -6 ubuntu
```
Output
```text
$6$vcSilJc5EkAaj/sE$RfHgGqzKQ/iVHvFObi8acrKFeLUcNAHH7YPT7hP7euIB5m8p.rbxxntrgyFalFG6eKlKE/OLCq6L2Fu/NWZi4/
```
Use this value on user-data yml file
```shell
cat < user-data
#cloud-config
users:
- name: ubuntu
passwd: \$6\$vcSilJc5EkAaj/sE\$RfHgGqzKQ/iVHvFObi8acrKFeLUcNAHH7YPT7hP7euIB5m8p.rbxxntrgyFalFG6eKlKE/OLCq6L2Fu/NWZi4/
lock_passwd: false
sudo: ['ALL=(ALL) NOPASSWD:ALL']
shell: /bin/bash
power_state:
mode: reboot
timeout: 30
condition: true
EOF
```
3. **Create a simple `meta-data` intilizer `cloud-init`**
```shell
cat < meta-data
local-hostname: instance-test
EOF
```
4. **Create a disk to attach with Cloud-Init configuration**
```shell
genisoimage \
-output instance-test-cidata.iso \
-input-charset utf-8 \
-volid cidata \
-joliet \
-rock \
user-data meta-data
```
Output
```text
Total translation table size: 0
Total rockridge attributes bytes: 331
Total directory bytes: 0
Path table size(bytes): 10
Max brk space used 0
183 extents written (0 MB)
```
5. **Launch virtual machine**
```shell
qemu-system-x86_64 \
-m 512 \
-hda ubuntu-image.qcow2 \
-cdrom instance-test-cidata.iso \
-enable-kvm \
-net nic \
-net user
```
6. **Login on image**
Use user `ubuntu` and password `ubuntu`, previously configured in step **2**
## Contributing
Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on our code of conduct, and the process for submitting pull requests to us.
## Versioning
We use [GitHub](https://github.com/mvallim/cloud-image-ubuntu-from-scratch) for versioning. For the versions available, see the [tags on this repository](https://github.com/mvallim/cloud-image-ubuntu-from-scratch/tags).
## Authors
* **Marcos Vallim** - *Initial work, Development, Test, Documentation* - [mvallim](https://github.com/mvallim)
See also the list of [contributors](CONTRIBUTORS.txt) who participated in this project.
## License
This project is licensed under the Apache License - see the [LICENSE](LICENSE) file for details