{"id":22927296,"url":"https://github.com/abarrak/linux-sysops-handbook","last_synced_at":"2025-04-05T06:10:11.537Z","repository":{"id":39602616,"uuid":"453699604","full_name":"abarrak/linux-sysops-handbook","owner":"abarrak","description":"Essentials of Linux system administration.","archived":false,"fork":false,"pushed_at":"2022-09-16T15:42:04.000Z","size":1053,"stargazers_count":492,"open_issues_count":0,"forks_count":58,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-29T05:11:14.271Z","etag":null,"topics":["gitbook","linux","markdown"],"latest_commit_sha":null,"homepage":"https://abarrak.gitbook.io/linux-sysops-handbook","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abarrak.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-01-30T14:17:19.000Z","updated_at":"2025-03-07T14:23:09.000Z","dependencies_parsed_at":"2023-01-17T16:15:32.715Z","dependency_job_id":null,"html_url":"https://github.com/abarrak/linux-sysops-handbook","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abarrak%2Flinux-sysops-handbook","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abarrak%2Flinux-sysops-handbook/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abarrak%2Flinux-sysops-handbook/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abarrak%2Flinux-sysops-handbook/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abarrak","download_url":"https://codeload.github.com/abarrak/linux-sysops-handbook/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247294541,"owners_count":20915340,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gitbook","linux","markdown"],"created_at":"2024-12-14T09:14:09.098Z","updated_at":"2025-04-05T06:10:11.521Z","avatar_url":"https://github.com/abarrak.png","language":null,"readme":"# Linux SysOps Handbook\n\nAn essentials notebook for the common knowledge and tasks of a Linux system admin.\n\n[![License: GPL v3](https://img.shields.io/badge/License-GPL%20v3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)\n\u003ca href=\"https://abdullah-barrak.gitbook.io/linux-sysops-handbook\" alt=\"Gitbook link\"\u003e\u003cimg src=\"https://img.shields.io/badge/gitbook-link-success\" /\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/abarrak/linux-sysops-handbook\" alt=\"Github link\"\u003e\u003cimg src=\"https://img.shields.io/badge/github-link-important\" /\u003e\u003c/a\u003e\n![GitHub contributors](https://img.shields.io/github/contributors/abarrak/linux-sysops-handbook)\n\n## Table of Content\n\n- [1. Processes](#processes)\n- [2. User Management](#user-management)\n- [3. Shell Tips and Tricks](#shell-tips-and-tricks)\n- [4. File Permissions](#file-permissions)\n- [5. Background Services and Crons](#background-services-and-crons)\n- [6. Linux Distros](#linux-distros)\n- [7. Logs, Monitoring, and Troubleshooting](#logs-monitoring-and-troubleshooting)\n- [8. Network Essentials](#network-essentials)\n- [9. System Updates and Patching](#system-updates-and-patching)\n- [10. Storage](#storage)\n- [11. Notes \u0026 Additional Resources](#notes-and-additional-resources)\n\n## Processes\n\nList the current active process with their statuses, numbers, resource usage, etc. using the command `ps`.\n\n```shell\n$ ps auxc\n```\n\nQuoting man's page documentation on `ps`: \"A different set of processes can be selected for display by using any combination of the `-a, -G, -g, -p, -T, -t, -U, and -u` options.  If more than one of these options are given, then ps will select all processes which are matched by at least one of the given options\".\n\nThe daemon `systemd` process starts during boot time, and remains active until the shutdown. It's the parent process for all other process in the system.\n\nEach process contains several main parts, such as: PID, state, virtual space address (memory), threads, network and file descriptors, scheduler information, and links. Processes are controlled and respond to signals. The states that a process can transition among are depicted below:\n\n\u003cimg src=\"https://github.com/abarrak/linux-sysops-handbook/blob/main/images/process-states.png?raw=true\" width=\"700px\" /\u003e\n\nTo observe the states and other information of the processes interactively, use the `top` command.\n\nTo run executables as background process (job), append an ampersand to it:\n\n```shell\n$ echo \"Hi .. looping:\" | sleep 10000 | echo \"done.\" \u0026\n```\n\nTo view the current jobs, and their details run `job`, `ps j` commands respectively.\n\nTo bring back a job in the foreground in the current session, and send it back use the following:\n\n```shell\n$ fg %\u003cjob-no\u003e\n$ ctrl+z\n$ bg %\u003cjob-no\u003e\n```\n\nUse the command `kill -l` to see the available signals to send to processes, like interrupt, terminate, resume, etc.\n\n```shell\n$ kill -l\n$ kill -9 5921\n$ kill -SIGTERM 6152\n```\n\nUse `killall` to operate on multiple processes using their executable name. Use `pkill` for filtering with more options.\n\n```shell\n$ killall -15 nginx\n$ pkill -U tester\n```\n\nFinally, use `pstree` and `pgrep` to view process parent/child tree and search for processes by pattern.\n\n```shell\n$ psgrep -u abdullah -l\n```\n\n## User Management\n\nThe users and groups are managed in `/etc/passwd` and `/etc/group` files.\n\n```shell\n$ tail /etc/passwd\n$ tail /etc/group\n$ tail /etc/shadow\n```\n\nThe commands to manage a user are as follows:\n- `useradd`\n- `usermod`\n- `userdel`\n\nAnd for groups:\n- `groupadd`\n- `groupmod`\n- `groupdel`\n\nEach user in the system is associated with unique user id `uid`, and each group is associated with `gid`.\n\n```shell\n$ id abdullah\n```\n\nUse flags `-g` and `-aG` for users to replace group or append group, respectively:\n\n```shell\n$ sudo usermod -G admins abdullah\n$ sudo usermod -aG staff abdullah\n```\n\nTo lock or unlock a user account, us the `-L`, `-U` options respectively.\n\n```shell\n$ usermod -L \u003cusername\u003e\n$ usermod -U \u003cusername\u003e\n```\n\nTo restrict service user accounts (e.g. accounts for web servers), the shell can be set to `nologin`:\n\n```shell\n$ usermod -s /sbin/nologin nginx_usr1\n```\n\nTo change a user password, use the command `passwd` interactively. Additionally `change` command sets the password policy in the system.\n\n\nUse the command `su - \u003cusername\u003e` to switch to the specified user. which will promote for her password. Running the command without username will switch to\nthe root user. To avoid cases where password is not available, use `sudo` to switch accounts using current user password only and according to rules in `/etc/sudoers` directory. Use `sudo -i` to gain an interactive root shell.\n\n\n## Shell Tips and Tricks \n\n\nGetting used to [bash language and its fundamentals](https://learnxinyminutes.com/docs/bash/) like conditions, looping, functions, etc. is recommended.\n\nThe popular files and text processing and manipulation utilities are important to master, such as:\n\n- `cat`\n- `cp`\n- `rm`\n- `mkdir`\n- `rmdir`\n- `touch`\n- `less`\n- `more`\n- `head`\n- `tail`\n- `grep`\n- `find`\n- `locate`\n- `wc`\n- `sed`\n\nUse the command `date` to print the current date and time or others in the past and future:\n\n```shell\n$ date +%x\n```\n\nThe standard terminal channels in Linux are 3: `stdin`, `stdout`, and `stderr` where the first is for input stream and the latters for output and error streams. \n\nBy default the successful command results are outputted to `stdout` (equivalent to `\u003e`). You can explicity redirect to `stdout` or `stderr` as follows:\n\n```shell\n$ echo \"hi there!\" 1\u003e error_log.txt\n$ cat ~/incorrect-path 2\u003e error_log.txt\n# To both:\n$ (echo \"hi\" \u0026\u0026 cat ~/wrong) \u003e\u003e log.txt 2\u003e\u00261\n```\n\nTo discard output stream, redirect it to the special directory `/dev/null`.\n\nThe standard input can be captured via redirection or file pipes:\n\n```shell\n$ cat \u003c\u003cEOF\nThis is coming from the stdin\nEOF\n\n$ cat LICENSE | wc -l\n```\n\nThe `ssh` command used to connect to servers in secure manner using OpenSSH library using public key cryptography. The configuration and known hosts are kept under `/etc/ssh` system-wide or in `~/.ssh/` in current user's home directory. On the other hand `scp` is used for secure copy on secure shell fashion.\n\nThe following list of commands are used to generate and manage ssh keys between client and server:\n\n1. `ssh-keygen`: to generate new key pairs.\n2. `ssh-copy-id`: to copy the public key to the remote machines.\n3. `ssh-agent`: to simplify working with the private key passphrase if used.\n4. `ssh-add`: to cache the passphrase in the current session.\n\n\n## File Permissions\n\nA file permissions are considered in three dimensions: the owner user, the owner's group, and rest of other users. \n\nShowing the permisison of files and directories can be using `ls -l`, `ls -ld` respectively.\n\nThe basic permission types are: read (r), write (w), and execute (x) on both folders and files:\n\n```shell\n$ ls -l\n-rw-r--r--  1 abdullah  staff  35149 Jan 30 17:20 LICENSE\n```\n\nSetting the files and folders permission is done by `chmod` command and can be using symbols or digits. \n\nThe symbols/letter way is made for `u`, `g`, `o`, or `a` basis for the user, group, others, or all. Whereas, the digits are written for all at once in sequence for user, group, and others. Examples are below for both cases:\n\n```shell\n# Use + to add, - to remove, and = to reset.\n\n# adding execute permission to user\n$ chmod u+x my-file.txt\n# setting read, execute to all on a folder and its content\n$ chmod -R a=rX my-folder\n\n$ chmod 740 special.txt\n$ chmod -R 444 read-only-files/\n```\n\n`chown` is used to change the ownership of folder/files to users or groups respectively. `chgrp` is a shortcut to group change only. The root or the owner are only people can change ownership and in the latter, she needs to be part of the new target group before the change.\n\n```shell\n$ chown sarah file-10.txt\n$ chown sarah:staff file-12.txt\n\n$ chown :admins server_log.txt\n$ chgrp operators server_log.txt\n```\n\nLastly, a fourth dimension at the start can be added to represent the special permissions of `suid s`, `sgid s`, and `sticky t` which control executable nature of files to be of owner users, and groups regardless of the current user. The last is to restrict deletion for only the root and owner always.\n\n```shell\n$ chmod a+t protected-folder/\n$ chmod -R 1444 read-only-protected/\n```\n\n## Background Services and Crons\n\n`systemctl` is the command used to list, manage, and check background processes or so called `daemons`.\n\nTo list the available categories of daemons, run:\n\n```shell\n$ systemctl -t help\n```\n\nThere are 3 types of daemons: 1. services, 2. sockets, 3. paths. Use the following to see the system's processes in each:\n\n```shell\n$ systemctl\n$ systemctl list-units --type=service\n$ systemctl list-units --type=socket --state=LOAD\n$ systemctl list-units --type=path --all\n$ systemctl list-unit-files\n```\n\nThe states `enabled` and `disabled` indicate wether a service is lanuched on startup or not. The subcommands `enable` and `disable` can be used to control this aspect.\n\nTo view the status of a daemon use the `status` command or its state shortcuts:\n\n```shell\n$ systemctl status kubelet\n$ systemctl is-active dockerd\n$ systemctl is-enabled sshd.service\n```\n\nUse the subcommands `start`, `stop`, `restart`, and `reload`, `reload-or-restart` to control daemons.\n\nAdditionally, use the following to list a daemon dependencies:\n\n```shell\n$ systemctl list-dependencies nginx.service\n```\n\nFinally, to resolve conflicting services making them unavailable, the `mask` and `unmask` commands can be used to point a deamons config to `dev/null` then back to normal respectively.\n\n\nThe cron daemon `crond` is responsible for managing the user's and system's scheduled jobs. Use the command `crontab` to manage jobs and their files in the user account or in the system wide `/etc/crontab`, `/etc/cron.d/` locations.\n\n```shell\n$ sudo crontab -l\n$ sudo crontab -e\n$ vim /etc/cron.d/my-backup\n```\n\nThe syntax of crontab entries is captured by the diagram below. Use the [following tool to quick assistance.](https://crontab.guru/)\n\n\u003cimg src=\"https://github.com/abarrak/linux-sysops-handbook/blob/main/images/crontabs.jpg?raw=true\" /\u003e\n\nAn example of a cron entry that runs backup command, every day at 5:00 AM:\n\n```shell\n0 5 * * * /usr/bin/daily-backup\n```\n\n## Linux Distros\n\nIn 1991, Linux kernel was introduced by Linus Torvalds, and combined with GNU project, which was previously created in 1983-1984 as open source OS programs and components. This formed what we call today Linux distribution, a Unix-like operating system.\n\nToday the Linux operating system is supported on most hardware platforms.  [Linux works on almost every architecture from i386 to SPARC](https://www.linuxtrainingacademy.com/linux-distribution-intro/). Linux can be found on almost every type of device today, from watches, televisions, mobile phones, servers, desktops, and even vending machines.\n\nOne of the major distinction between Linux distributions is the package management part and how software is installed and managed. There are multiple package formats, and the most common ones are Debian (deb), RedHat Package Manager (RPM).\n\n\u003cimg src=\"https://github.com/abarrak/linux-sysops-handbook/blob/main/images/distros.jpg?raw=true\" width=\"700px\" /\u003e\n\nHere's a listing for the common Debian based distributions:\n\n- Debian.\n- Ubuntu.\n- Linux Mint.\n- Kali Linux.\n\nAnd here's for RPM based distributions:\n\n- Fedora.\n- RedHat Enterprise Linux (RHEL).\n- CentOS.\n- openSUSE.\n\n\u003cimg src=\"https://github.com/abarrak/linux-sysops-handbook/blob/main/images/timeline.png?raw=true\" width=\"700px\" /\u003e\n\n## Logs, Monitoring, and Troubleshooting\n\nYou can monitor the system's resources usage, uptime, and sessions' load leverages over time as follows:\n\n```shell\n$ top\n$ uptime\n$ w\n```\n\nUse `lscpu` to see the system's CPU in use and other details.\n\nThe system events and processes traces are usually kept in as logs in `/var/log` directory. There are two categories of logs: 1. essential system logs via `journald`, that are wiped across boots by default (can be configured to persist). 2. `rsyslog` logs that persist by default and organized inside `/var/log/` folder. Mainly, the logging mechanism in Linux follows the standard `syslog` protocol for the system's messages, events, security incidents, mailing, and jobs logs, while other programs may or may not follow `syslog` format identically.\n\n\nAs explored in section (3), use `cat`, `head`, `tail` commands to interactively see or follow the logs.\n\n```shell\n$ head -n 50 /var/logs/mail.log\n$ tail -f /var/logs/mysql.log\n```\n\nYou can configure `rsyslog` service and manage it as any daemon:\n\n```shell\n$ vim /etc/rsyslog.conf\n$ systemctl reload rsyslog\n```\n\nOn the other hand, use `journalctl` to view and follow the system's `journald` log entries, which resides in `/run/log/journal`.\n\n```shell\n$ journalctl -n 50 -p err \n$ journalctl -f\n$ journalctl _PID=6610\n```\n\n\n## Network Essentials\n\nFor effective work on the system network configurations and troubleshooting, it is essential to review network/internet protocols (TCP/UDP) and IPv4/IPv6 concepts [(Ref.1)](https://www.ibm.com/cloud/learn/networking-a-complete-guide), [(Ref.2)](https://www.cloudflare.com/learning/network-layer/what-is-a-protocol/).\n\n\nSee the hostname of current machine or set it as below:\n\n```shell\n$ hostname\n$ hostnamectl set-hostname rhel.n1.apps.com\n```\n\nThe host name is managed under `/etc/hostname`. \n\nThe host connection is either managed dynamically (`DHCP`) configured in `/etc/resolv.conf` or manually in `/etc/hosts` file.\n\nThe `ping` utiltiy helps for connectivity checking:\n\n```shell\n$ ping 172.168.9.13\n$ ping -c4 github.com\n$ ping6 2001:db8:3333:4444:5555:6666:7777:8888\n```\n\nTo see the network routing table and interfaces, use the following:\n\n```shell\n$ ip route\n$ ip -6 route\n$ ip help\n$ ip show link\n```\n\nUse the command `nmap` [for advanced network investigation and security monitor and scan.](https://www.cyberciti.biz/security/nmap-command-examples-tutorials/)\n\n```shell\n# Scan a single ip address\n$ nmap 192.168.1.1\n \n# Scan a host name \n$ nmap -v server1.cyberciti.biz\n\n# View open ports:\n$ nmap --open 192.168.2.18\n\n# Trace all pakets:\n$ nmap --packet-trace 192.168.1.1\n```\n\n`NetworkManager` is the kernel feature [to manage network configurations in Linux](https://en.wikipedia.org/wiki/NetworkManager). `nmcli` is the terminal utility.\n\n```shell\n$ nmcli device wifi list\n$ nmcli dev status\n$ nmcli general hostname centos-8.cluster.internal\n$ nmcli con show \n```\n\n\n## System Updates and Patching\n\nManaging the system packages varies depending on Linux distributions, but the essential parts are the same (installation, repositories, package managers, etc.). For Debian based distributions, `apt` is the package manager, whereas for Fedora / RHEL, `yum` is used.\n\nSearch for some package:\n\n```shell\n$ apt search \u003cKEYWORD\u003e\n$ yum search \u003cKEYWORD\u003e\n```\n\nInstall a package:\n\n```shell\n$ apt install \u003cNAME\u003e\n$ yum install \u003cNAME\u003e\n```\n\n\nUpdate a package or all packages:\n\n```shell\n$ apt upgrade \u003cNAME\u003e\n$ yum update \u003cNAME\u003e\n```\n\nRemove a package:\n\n```shell\n$ apt remove \u003cNAME\u003e\n$ yum remove \u003cNAME\u003e\n```\n\nShow details on a package:\n\n```shell\n$ apt show \u003cNAME\u003e\n$ yum info \u003cNAME\u003e\n```\n\nList all current packages on the system:\n\n```shell\n$ apt list --installed\n$ yum list\n```\n\nAudit the history of package management actions:\n\n```shell\n$ cat less /var/log/apt/history.log | less\n$ cat less /var/log/dnf.rpm.log | less\n$ yum history\n```\n\nAnd finally, the package source repositories can be set up and updated through the following:\n\n```shell\n# list current enabled repos\n$ yum repolist all\n$ apt-cache policy\n\n# manage and add repos in these directories:\n$ cat /etc/apt/sources.list /etc/apt/sources.list.d/*\n$ cat /etc/yum.repos.d/*\n```\n\n## Storage\n\nLinux is formed for a unified file-system consists of all file systems provided by the hardware or virtual storage devices attached to the system. Essentially, everything in Linux is a file. It can be viewed as a reversed tree of nested directories starting from the root directory `/`.\n\n\u003cimg src=\"https://github.com/abarrak/linux-sysops-handbook/blob/main/images/linux-file-system.png?raw=true\" width=\"700px\" /\u003e\n\nBlock devices are the mechanism that the kernel detects and identify raw storage devices (HDD, SSD, USBs, ..). [As the name indicates, the kernel interfaces and references them by fixed-size blocks (chunks of spaces)](https://www.digitalocean.com/community/tutorials/an-introduction-to-storage-terminology-and-concepts-in-linux). The block devices are stored in `/dev` directory by the OS, and has letters naming convention such as `/dev/sda`, `/dev/sdb`, `/dev/vda`, and appended numbers in case of partitions `/dev/sda3`. The attachment of the block device into the system is done through mounting it to a directory in the system.\n\nTwo operations are essential for using block storage:\n\n**1. Partitioning:**\n\n  Breaking the disk into reusable smaller units, each treated as own disk. \n  The main partitioning methods are MBR (Master Boot Record) and GPT (GUID Partition Table).\n  Use `parted` or equivalent commands to prepare partitions of block devices.\n\n**2. Formatting:**\n\n  Preparing the device as a file-system to be read and write to. Many file-system formats exists like:\n\n  - `Ext4`.\n  - `XFS`.\n  - `Btrfs`.\n  - `ZFS`.\n\n\nAdditionally, LVM concepts focus on building more extensible storage layout by grouping physical volumes (PV) into logical groups (VG), then creating logical volumes from, with possibility of extending or reduction later on.\n\nTo see the block devices and currently attached file system with mounts, and disk usage:\n\n```shell\n$ blkid\n$ mount\n$ df -h\n$ du -h /opt/data\n```\n\nThe `lsof` command lists all active processes using the block device.\n\nThe permanent mounting process rely on `/etc/fstab` file to determine devices to mount on the boot time.\n\nUse the commands `lsblk`, `mount`, and `unmount` to check and mount filesystem devices, respectively.\n\n\n## Notes and Additional Resources\n\nUse the `man` command to lookup the manual information on commands or topics in the system.\n\nAdditionally, the `info` command is the GNU documentation tool and provide more detailed materials.\n\nBoth provide shortcuts, navigation, and searching capabilities (e.g. `man -K \u003ckeyword\u003e` to search across manual).\n\n### Recommended Reading List\n\n**Books:**\n\n1. [How Linux Works What Every Superuser Should Know, Brian Ward, _2nd Edition, No Starch Press_.](https://www.amazon.com/How-Linux-Works-2nd-Superuser/dp/1593275676).\n2. [Linux Command Line and Shell Scripting Bible, R. Blum and C. Bresnahan, _3rd Edition, Wiley_.](https://www.amazon.com/Command-Scripting-Christine-Bresnahan-2015-01-20/dp/B01JNWWSZA)\n3. [Linux Bible, Christopher Negus, _9th Edition, Wiley_.](https://www.amazon.com/Linux-Bible-Christopher-Negus/dp/1119578884/)\n\n**Websites \u0026 Blogs:**\n\n1. [Digital Ocean Knowledge Hub.](https://www.digitalocean.com/community/tags/linux-basics?language=en)\n2. [9 to 5 Linux Blog.](https://9to5linux.com/)\n3. [nixCraft.](https://www.cyberciti.biz/)\n\n\n## License\n\n[GNU General Public License v3.0](https://github.com/abarrak/linux-sysops-handbook/blob/main/LICENSE).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabarrak%2Flinux-sysops-handbook","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabarrak%2Flinux-sysops-handbook","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabarrak%2Flinux-sysops-handbook/lists"}