Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/mvallim/live-custom-ubuntu-from-scratch

(Yes, the project is still alive 😃) This procedure shows how to create a bootable and installable Ubuntu Live (along with the automatic hardware detection and configuration) from scratch. A Linux to call your own.
https://github.com/mvallim/live-custom-ubuntu-from-scratch

bionic bios bootstrap focal linux live-cd scratch ubuntu uefi

Last synced: 22 days ago
JSON representation

Host: GitHub
URL: https://github.com/mvallim/live-custom-ubuntu-from-scratch
Owner: mvallim
License: gpl-3.0
Created: 2019-06-26T16:33:13.000Z (over 5 years ago)
Default Branch: master
Last Pushed: 2024-09-15T15:53:41.000Z (3 months ago)
Last Synced: 2024-09-15T17:09:43.878Z (3 months ago)
Topics: bionic, bios, bootstrap, focal, linux, live-cd, scratch, ubuntu, uefi
Language: Shell
Homepage: https://mvallim.github.io/live-custom-ubuntu-from-scratch/
Size: 687 KB
Stars: 402
Watchers: 23
Forks: 189
Open Issues: 12
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

awesome-hacking-lists - mvallim/live-custom-ubuntu-from-scratch - (Yes, the project is still alive 😃) This procedure shows how to create a bootable and installable Ubuntu Live (along with the automatic hardware detection and configuration) from scratch. (Shell)

README

# How to create a custom Ubuntu live from scratch

This project guides you through building a fully customized version of Ubuntu Linux from scratch. It covers creating a live ISO image that includes pre-installed packages, configurations, and custom scripts tailored to your needs. The steps walk you through setting up the environment, configuring the chroot, installing software, modifying the kernel, and finally generating the ISO image. This is ideal for those who want complete control over their Linux distribution, whether for personal or professional use.

## Requirements

* Proficiency with Linux shell commands and scripting.
* Sufficient disk space and memory for building an ISO.

## Steps
1. **Prepare the Environment**: Install necessary dependencies.
2. **Create a Base System**: Use debootstrap to set up a minimal Ubuntu system.
3. **Customize Packages**: Add/remove software, configure kernel.
4. **Generate ISO**: Package the system into a bootable ISO.

## Authors

* **Marcos Vallim** - *Founder, Author, Development, Test, Documentation* - [mvallim](https://github.com/mvallim)
* **Ken Gilmer** - *Commiter, Development, Test, Documentation* - [kgilmer](https://github.com/kgilmer)

See also the list of [contributors](CONTRIBUTORS.txt) who participated in this project.

## Ways of Using this Tutorial

* (Recommended) follow the directions step by step below to understand how to build an Ubuntu ISO.
* Run the `build.sh` script in the `scripts` directory after checking this repo out locally.
* Fork this repo and run the github action `build`. This will generate an ISO in your github account.

[![build-bionic](https://github.com/mvallim/live-custom-ubuntu-from-scratch/actions/workflows/build-bionic.yml/badge.svg)](https://github.com/mvallim/live-custom-ubuntu-from-scratch/actions/workflows/build-bionic.yml)
[![build-focal](https://github.com/mvallim/live-custom-ubuntu-from-scratch/actions/workflows/build-focal.yml/badge.svg)](https://github.com/mvallim/live-custom-ubuntu-from-scratch/actions/workflows/build-focal.yml)
[![build-jammy](https://github.com/mvallim/live-custom-ubuntu-from-scratch/actions/workflows/build-jammy.yml/badge.svg)](https://github.com/mvallim/live-custom-ubuntu-from-scratch/actions/workflows/build-jammy.yml)
[![build-noble](https://github.com/mvallim/live-custom-ubuntu-from-scratch/actions/workflows/build-noble.yml/badge.svg)](https://github.com/mvallim/live-custom-ubuntu-from-scratch/actions/workflows/build-noble.yml)

## Terms

* `build system` - the computer environment running the build scripts that generate the ISO.
* `live system` - the computer environment that runs from the live OS, generated by a `build system`. This may also be referred to as the `chroot environment`.
* `target system` - the computer environment that runs after installation has completed from a `live system`.

## Prerequisites (GNU/Linux Ubuntu)

Install packages we need in the `build system` required by our scripts.

```shell
sudo apt-get install \
debootstrap \
squashfs-tools \
xorriso
```

```shell
mkdir $HOME/live-ubuntu-from-scratch
```

## Bootstrap and Configure Ubuntu

`debootstrap` is a program for generating OS images. We install it into our `build system` to begin generating our ISO.

* Checkout bootstrap

```shell
sudo debootstrap \
--arch=amd64 \
--variant=minbase \
noble \
$HOME/live-ubuntu-from-scratch/chroot \
http://us.archive.ubuntu.com/ubuntu/
```

> **debootstrap** is used to create a Debian base system from scratch, without requiring the availability of **dpkg** or **apt**. It does this by downloading .deb files from a mirror site, and carefully unpacking them into a directory which can eventually be **chrooted** into.

* Configure external mount points

```shell
sudo mount --bind /dev $HOME/live-ubuntu-from-scratch/chroot/dev

sudo mount --bind /run $HOME/live-ubuntu-from-scratch/chroot/run
```

As we will be updating and installing packages (grub among them), these mount points are necessary inside the chroot environment, so we are able to finish the installation without errors.

## Define chroot environment

*A chroot on Unix operating systems is an operation that changes the apparent root directory for the current running process and its children. A program that is run in such a modified environment cannot name (and therefore normally cannot access) files outside the designated directory tree. The term "chroot" may refer to the chroot system call or the chroot wrapper program. The modified environment is called a chroot jail.*

> Reference:

From this point we will be configuring the `live system`.

1. **Access chroot environment**

```shell
sudo chroot $HOME/live-ubuntu-from-scratch/chroot
```

2. **Configure mount points, home and locale**

```shell
mount none -t proc /proc

mount none -t sysfs /sys

mount none -t devpts /dev/pts

export HOME=/root

export LC_ALL=C
```

These mount points are necessary inside the chroot environment, so we are able to finish the installation without errors.

3. **Set a custom hostname**

```shell
echo "ubuntu-fs-live" > /etc/hostname
```

4. **Configure apt sources.list**

```shell
cat < /etc/apt/sources.list
deb http://us.archive.ubuntu.com/ubuntu/ noble main restricted universe multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ noble main restricted universe multiverse

deb http://us.archive.ubuntu.com/ubuntu/ noble-security main restricted universe multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ noble-security main restricted universe multiverse

deb http://us.archive.ubuntu.com/ubuntu/ noble-updates main restricted universe multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ noble-updates main restricted universe multiverse
EOF
```

5. **Update indexes packages**

```shell
apt-get update
```

6. **Install systemd**

```shell
apt-get install -y libterm-readline-gnu-perl systemd-sysv
```

> **systemd** is a system and service manager for Linux. It provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux control groups, maintains mount and automount points and implements an elaborate transactional dependency-based service control logic.

7. **Configure machine-id and divert**

```shell
dbus-uuidgen > /etc/machine-id

ln -fs /etc/machine-id /var/lib/dbus/machine-id
```

> The `/etc/machine-id` file contains the unique machine ID of the local system that is set during installation or boot. The machine ID is a single newline-terminated, hexadecimal, 32-character, lowercase ID. When decoded from hexadecimal, this corresponds to a 16-byte/128-bit value. This ID may not be all zeros.

```shell
dpkg-divert --local --rename --add /sbin/initctl

ln -s /bin/true /sbin/initctl
```

> **dpkg-divert** is the utility used to set up and update the list of diversions.

8. **Upgrade packages**

```shell
apt-get -y upgrade
```

9. **Install packages needed for Live System**

```shell
apt-get install -y \
sudo \
ubuntu-standard \
casper \
discover \
laptop-detect \
os-prober \
network-manager \
net-tools \
wireless-tools \
wpagui \
locales \
grub-common \
grub-gfxpayload-lists \
grub-pc \
grub-pc-bin \
grub2-common \
grub-efi-amd64-signed \
shim-signed \
mtools \
binutils
```

```shell
apt-get install -y --no-install-recommends linux-generic
```

10. **Graphical installer**

```shell
apt-get install -y \
ubiquity \
ubiquity-casper \
ubiquity-frontend-gtk \
ubiquity-slideshow-ubuntu \
ubiquity-ubuntu-artwork
```

The next steps will appear, as a result of the packages that will be installed from the previous step, this will happen without anything having to be informed or executed.

1. Configure keyboard

2. Console setup

11. **Install window manager**

```shell
apt-get install -y \
plymouth-themes \
ubuntu-gnome-desktop \
ubuntu-gnome-wallpapers
```

12. **Install useful applications**

```shell
apt-get install -y \
clamav-daemon \
terminator \
apt-transport-https \
curl \
vim \
nano \
less
```

13. **Install Visual Studio Code (optional)**

1. Download and install the key

```shell
curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg

install -o root -g root -m 644 microsoft.gpg /etc/apt/trusted.gpg.d/

echo "deb [arch=amd64] https://packages.microsoft.com/repos/vscode stable main" > /etc/apt/sources.list.d/vscode.list

rm microsoft.gpg
```

2. Then update the package cache and install the package using

```shell
apt-get update

apt-get install -y code
```

14. **Install Google Chrome (optional)**

1. Download and install the key

```shell
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -

echo "deb http://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google-chrome.list
```

2. Then update the package cache and install the package using

```shell
apt-get update

apt-get install google-chrome-stable
```

15. **Install Java JDK 8 (optional)**

```shell
apt-get install -y \
openjdk-8-jdk \
openjdk-8-jre
```

16. **Remove unused applications (optional)**

```shell
apt-get purge -y \
transmission-gtk \
transmission-common \
gnome-mahjongg \
gnome-mines \
gnome-sudoku \
aisleriot \
hitori
```

17. **Remove unused packages**

```shell
apt-get autoremove -y
```

18. **Reconfigure packages**

1. Generate locales

```shell
dpkg-reconfigure locales
```

1. *Select locales*

2. *Select default locale*

2. Configure network-manager

1. Create config file

```shell
cat < /etc/NetworkManager/NetworkManager.conf
[main]
rc-manager=none
plugins=ifupdown,keyfile
dns=systemd-resolved

[ifupdown]
managed=false
EOF
```

2. Reconfigure network-manager

```shell
dpkg-reconfigure network-manager
```

## Create the image directory and populate it

We are now back in our `build environment` after setting up our `live system` and will continue creating files necessary to generate the ISO.

1. Create directories

```shell
mkdir -p /image/{casper,isolinux,install}
```

2. Copy kernel images

```shell
cp /boot/vmlinuz-**-**-generic /image/casper/vmlinuz

cp /boot/initrd.img-**-**-generic /image/casper/initrd
```

3. Copy memtest86+ binary (BIOS and UEFI)

```shell
wget --progress=dot https://memtest.org/download/v7.00/mt86plus_7.00.binaries.zip -O /image/install/memtest86.zip
unzip -p /image/install/memtest86.zip memtest64.bin > /image/install/memtest86+.bin
unzip -p /image/install/memtest86.zip memtest64.efi > /image/install/memtest86+.efi
rm -f /image/install/memtest86.zip
```

### GRUB menu configuration

1. Create base point access file for grub

```shell
touch /image/ubuntu
```

2. Create image/isolinux/grub.cfg

```shell
cat < /image/isolinux/grub.cfg

search --set=root --file /ubuntu

insmod all_video

set default="0"
set timeout=30

menuentry "Try Ubuntu FS without installing" {
linux /casper/vmlinuz boot=casper nopersistent toram quiet splash ---
initrd /casper/initrd
}

menuentry "Install Ubuntu FS" {
linux /casper/vmlinuz boot=casper only-ubiquity quiet splash ---
initrd /casper/initrd
}

menuentry "Check disc for defects" {
linux /casper/vmlinuz boot=casper integrity-check quiet splash ---
initrd /casper/initrd
}

grub_platform
if [ "\$grub_platform" = "efi" ]; then
menuentry 'UEFI Firmware Settings' {
fwsetup
}

menuentry "Test memory Memtest86+ (UEFI)" {
linux /install/memtest86+.efi
}
else
menuentry "Test memory Memtest86+ (BIOS)" {
linux16 /install/memtest86+.bin
}
fi
EOF
```

### Create manifest

Next we create a file `filesystem.manifest` to specify each package and it's version that is installed on the `live system`. We create another file `filesystem.manifest-desktop` which specifies which files will be installed on the `target system`. Once the Ubiquity installer completes, it will
remove packages specified in `filesystem.manifest` that are *not* listed in `filesystem.manifest-desktop`.

1. Generate manifest

```shell
dpkg-query -W --showformat='${Package} ${Version}\n' | sudo tee /image/casper/filesystem.manifest

cp -v /image/casper/filesystem.manifest image/casper/filesystem.manifest-desktop

sed -i '/ubiquity/d' /image/casper/filesystem.manifest-desktop

sed -i '/casper/d' /image/casper/filesystem.manifest-desktop

sed -i '/discover/d' /image/casper/filesystem.manifest-desktop

sed -i '/laptop-detect/d' /image/casper/filesystem.manifest-desktop

sed -i '/os-prober/d' /image/casper/filesystem.manifest-desktop
```

### Create diskdefines

**README** file often found on Linux LiveCD installer discs, such as an Ubuntu Linux installation CD; typically named “**README.diskdefines**” and may be referenced during installation.

1. Create file /image/README.diskdefines

```shell
cat < /image/README.diskdefines
#define DISKNAME Ubuntu from scratch
#define TYPE binary
#define TYPEbinary 1
#define ARCH amd64
#define ARCHamd64 1
#define DISKNUM 1
#define DISKNUM1 1
#define TOTALNUM 0
#define TOTALNUM0 1
EOF
```

### Creating image

1. Access image directory

```shell
cd /image
```

2. Copy EFI loaders

```shell
cp /usr/lib/shim/shimx64.efi.signed.previous isolinux/bootx64.efi
cp /usr/lib/shim/mmx64.efi isolinux/mmx64.efi
cp /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed isolinux/grubx64.efi
```

3. Create a FAT16 UEFI boot disk image containing the EFI bootloaders

```shell
(
cd isolinux && \
dd if=/dev/zero of=efiboot.img bs=1M count=10 && \
mkfs.vfat -F 16 efiboot.img && \
LC_CTYPE=C mmd -i efiboot.img efi efi/ubuntu efi/boot && \
LC_CTYPE=C mcopy -i efiboot.img ./bootx64.efi ::efi/boot/bootx64.efi && \
LC_CTYPE=C mcopy -i efiboot.img ./mmx64.efi ::efi/boot/mmx64.efi && \
LC_CTYPE=C mcopy -i efiboot.img ./grubx64.efi ::efi/boot/grubx64.efi && \
LC_CTYPE=C mcopy -i efiboot.img ./grub.cfg ::efi/ubuntu/grub.cfg
)
```

4. Create a grub BIOS image

```shell
grub-mkstandalone \
--format=i386-pc \
--output=isolinux/core.img \
--install-modules="linux16 linux normal iso9660 biosdisk memdisk search tar ls" \
--modules="linux16 linux normal iso9660 biosdisk search" \
--locales="" \
--fonts="" \
"boot/grub/grub.cfg=isolinux/grub.cfg"
```

5. Combine a bootable Grub cdboot.img

```shell
cat /usr/lib/grub/i386-pc/cdboot.img isolinux/core.img > isolinux/bios.img
```

6. Generate md5sum.txt

```shell
/bin/bash -c "(find . -type f -print0 | xargs -0 md5sum | grep -v -e 'isolinux' > md5sum.txt)"
```

### Cleanup the chroot environment

1. If you installed software, be sure to run

```shell
truncate -s 0 /etc/machine-id
```

2. Remove the diversion

```shell
rm /sbin/initctl

dpkg-divert --rename --remove /sbin/initctl
```

3. Clean up

```shell
apt-get clean

rm -rf /tmp/* ~/.bash_history

umount /proc

umount /sys

umount /dev/pts

export HISTSIZE=0

exit
```

## Unbind mount points

```shell
sudo umount $HOME/live-ubuntu-from-scratch/chroot/dev

sudo umount $HOME/live-ubuntu-from-scratch/chroot/run
```

## Compress the chroot

After everything has been installed and preconfigured in the **chrooted** environment, we need to generate an image of everything that was done by following the next steps in the `build environment`.

1. Access build directory

```shell
cd $HOME/live-ubuntu-from-scratch
```

2. Move image artifacts

```shell
sudo mv chroot/image .
```

3. Create squashfs

```shell
sudo mksquashfs chroot image/casper/filesystem.squashfs \
-noappend -no-duplicates -no-recovery \
-wildcards \
-comp xz -b 1M -Xdict-size 100% \
-e "var/cache/apt/archives/*" \
-e "root/*" \
-e "root/.*" \
-e "tmp/*" \
-e "tmp/.*" \
-e "swapfile"
```

> **Squashfs** is a highly compressed read-only filesystem for Linux. It uses zlib compression to compress both files, inodes and directories. Inodes in the system are very small and all blocks are packed to minimize data overhead. Block sizes greater than 4K are supported up to a maximum of 64K.
> **Squashfs** is intended for general read-only filesystem use, for archival use (i.e. in cases where a .tar.gz file may be used), and in constrained block device/memory systems (e.g. **embedded systems**) where low overhead is needed.

4. Write the filesystem.size

```shell
printf $(sudo du -sx --block-size=1 chroot | cut -f1) | sudo tee image/casper/filesystem.size
```

## Create ISO Image for a LiveCD (BIOS + UEFI + Secure Boot)

1. Access build directory

```shell
cd $HOME/live-ubuntu-from-scratch/image
```

2. Create iso from the image directory using the command-line

```shell
sudo xorriso \
-as mkisofs \
-iso-level 3 \
-full-iso9660-filenames \
-J -J -joliet-long \
-volid "Ubuntu from scratch" \
-output "../ubuntu-from-scratch.iso" \
-eltorito-boot isolinux/bios.img \
-no-emul-boot \
-boot-load-size 4 \
-boot-info-table \
--eltorito-catalog boot.catalog \
--grub2-boot-info \
--grub2-mbr ../chroot/usr/lib/grub/i386-pc/boot_hybrid.img \
-partition_offset 16 \
--mbr-force-bootable \
-eltorito-alt-boot \
-no-emul-boot \
-e isolinux/efiboot.img \
-append_partition 2 28732ac11ff8d211ba4b00a0c93ec93b isolinux/efiboot.img \
-appended_part_as_gpt \
-iso_mbr_part_type a2a0d0ebe5b9334487c068b6b72699c7 \
-m "isolinux/efiboot.img" \
-m "isolinux/bios.img" \
-e '--interval:appended_partition_2:::' \
-exclude isolinux \
-graft-points \
"/EFI/boot/bootx64.efi=isolinux/bootx64.efi" \
"/EFI/boot/mmx64.efi=isolinux/mmx64.efi" \
"/EFI/boot/grubx64.efi=isolinux/grubx64.efi" \
"/EFI/ubuntu/grub.cfg=isolinux/grub.cfg" \
"/isolinux/bios.img=isolinux/bios.img" \
"/isolinux/efiboot.img=isolinux/efiboot.img" \
"."
```

## Alternative way, if previous one fails, create an Hybrid ISO

1. Create a ISOLINUX (syslinux) boot menu

```shell
cat < isolinux/isolinux.cfg
UI vesamenu.c32

MENU TITLE Boot Menu
DEFAULT linux
TIMEOUT 600
MENU RESOLUTION 640 480
MENU COLOR border 30;44 #40ffffff #a0000000 std
MENU COLOR title 1;36;44 #9033ccff #a0000000 std
MENU COLOR sel 7;37;40 #e0ffffff #20ffffff all
MENU COLOR unsel 37;44 #50ffffff #a0000000 std
MENU COLOR help 37;40 #c0ffffff #a0000000 std
MENU COLOR timeout_msg 37;40 #80ffffff #00000000 std
MENU COLOR timeout 1;37;40 #c0ffffff #00000000 std
MENU COLOR msg07 37;40 #90ffffff #a0000000 std
MENU COLOR tabmsg 31;40 #30ffffff #00000000 std

LABEL linux
MENU LABEL Try Ubuntu FS
MENU DEFAULT
KERNEL /casper/vmlinuz
APPEND initrd=/casper/initrd boot=casper

LABEL linux
MENU LABEL Try Ubuntu FS (nomodeset)
MENU DEFAULT
KERNEL /casper/vmlinuz
APPEND initrd=/casper/initrd boot=casper nomodeset
EOF
```

2. Include syslinux bios modules

```shell
apt install -y syslinux-common && \
cp /usr/lib/ISOLINUX/isolinux.bin image/isolinux/ && \
cp /usr/lib/syslinux/modules/bios/* image/isolinux/
```

3. Access build directory

```shell
cd $HOME/live-ubuntu-from-scratch/image
```

4. Create iso from the image directory

```shell
sudo xorriso \
-as mkisofs \
-iso-level 3 \
-full-iso9660-filenames \
-J -J -joliet-long \
-volid "Ubuntu from scratch" \
-output "../ubuntu-from-scratch.iso" \
-isohybrid-mbr /usr/lib/ISOLINUX/isohdpfx.bin \
-eltorito-boot \
isolinux/isolinux.bin \
-no-emul-boot \
-boot-load-size 4 \
-boot-info-table \
--eltorito-catalog isolinux/isolinux.cat \
-eltorito-alt-boot \
-e /EFI/boot/efiboot.img \
-no-emul-boot \
-isohybrid-gpt-basdat \
-append_partition 2 0xef EFI/boot/efiboot.img \
"$HOME/live-ubuntu-from-scratch/image"
```

## Make a bootable USB image

It is simple and easy, using "dd"

```shell
sudo dd if=ubuntu-from-scratch.iso of= status=progress oflag=sync
```

## Summary

This completes the process of creating a live Ubuntu installer from scratch. The generated ISO may be tested in a virtual machine such as `VirtualBox` or written to media and booted from a standard PC.

## Contributing

Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on our code of conduct, and the process for submitting pull requests to us.

## Versioning

We use [GitHub](https://github.com/mvallim/live-custom-ubuntu-from-scratch) for versioning. For the versions available, see the [tags on this repository](https://github.com/mvallim/live-custom-ubuntu-from-scratch/tags).

## License

This project is licensed under the GNU GENERAL PUBLIC LICENSE - see the [LICENSE](LICENSE) file for details