{"id":13584190,"url":"https://github.com/miguelmota/bash-streams-handbook","last_synced_at":"2025-04-07T01:31:37.711Z","repository":{"id":45467228,"uuid":"268741950","full_name":"miguelmota/bash-streams-handbook","owner":"miguelmota","description":"💻 Learn Bash streams, pipelines and redirection, from beginner to advanced.","archived":true,"fork":false,"pushed_at":"2022-03-24T10:50:21.000Z","size":3547,"stargazers_count":242,"open_issues_count":0,"forks_count":19,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-11-06T01:39:30.629Z","etag":null,"topics":["awesome","bash","book","example","gnu","gnu-linux","handbook","learning","linux","named-pipes","pipeline","pipes","process-substitution","shell","standard-streams","streams","terminal","tty","tutorial","unix"],"latest_commit_sha":null,"homepage":"https://github.com/miguelmota/bash-streams-handbook","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/miguelmota.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-02T08:18:04.000Z","updated_at":"2024-11-01T19:12:34.000Z","dependencies_parsed_at":"2022-07-14T15:31:12.001Z","dependency_job_id":null,"html_url":"https://github.com/miguelmota/bash-streams-handbook","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miguelmota%2Fbash-streams-handbook","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miguelmota%2Fbash-streams-handbook/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miguelmota%2Fbash-streams-handbook/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miguelmota%2Fbash-streams-handbook/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/miguelmota","download_url":"https://codeload.github.com/miguelmota/bash-streams-handbook/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247577931,"owners_count":20961197,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["awesome","bash","book","example","gnu","gnu-linux","handbook","learning","linux","named-pipes","pipeline","pipes","process-substitution","shell","standard-streams","streams","terminal","tty","tutorial","unix"],"created_at":"2024-08-01T15:04:04.518Z","updated_at":"2025-04-07T01:31:32.690Z","avatar_url":"https://github.com/miguelmota.png","language":null,"funding_links":[],"categories":["Others"],"sub_categories":[],"readme":"# Bash Streams Handbook\n\n\u003e Learn Bash streams, pipes and redirects, from beginner to advanced.\n\n[![License](http://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/miguelmota/bash-pipe-handbook/master/LICENSE)\n[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](#contributing)\n![status-wip](https://img.shields.io/badge/status-work_in_progress-lightgrey.svg)\n\u003c!--\n[![Mentioned in Awesome Bash](https://awesome.re/mentioned-badge.svg)](https://github.com/awesome-lists/awesome-bash)\n--\u003e\n\n## Contents\n\n- [Standard streams](#standard-streams)\n  - [Standard input](#standard-input)\n  - [Standard output](#standard-output)\n  - [Standard error](#standard-error)\n- [Redirection](#redirection)\n- [Pipelines](#pipelines)\n- [Named pipes](#named-pipes)\n- [Command grouping](#command-grouping)\n- [Process Substitution](#process-substitution)\n- [Subshells](#subshells)\n- [Examples](#examples)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Standard streams\n\nA data stream in the context of Bash is a communication channel between a program and the environment where the command was launched from.\n\nThere are three data [standard streams](https://en.wikipedia.org/wiki/Standard_streams) that are created when a command is launched.\n\nThe three streams are:\n\n- **stdin** - standard input\n- **stdout** - standard output\n- **stderr** - standard error\n\nFurther more:\n\n- The standard input stream accepts text as it's input.\n- The text output from the command is sent to the shell though the standard output stream.\n- Error messages from the command is sent to the shell through the standard error stream.\n\nThese data streams are treated as files meaning you read from them and write to them as if they are regular files. Files are identified by a unique number called a **file descriptor**, which the process uses to perform read/write operations.\n\nWhen a command is launched, the first three file descriptors are allocated for the standard streams in the TTY. A [_TTY_](https://en.wikipedia.org/wiki/Computer_terminal#Text_terminals) is the input/output environment, which is the terminal. This is different than a _shell_, which refers to the command-line interpreter.\n\nStandard stream file descriptors:\n\n- **0**: stdin\n- **1**: stdout\n- **2**: stderr\n\nFile descriptor `0` is dedicated for standard input, `1` for standard output, and `2` for standard error.\n\nFile descriptors are maintained under `/proc/$pid/fd` where `$pid` is the process id. The current process can be referenced by `/proc/self/fd`.\n\nThe locations of these file descriptors are in `/proc/self/fd/0`, `/proc/self/fd/1`, and `/proc/self/fd/2` respectively.\n\n```bash\n$ ls -la /proc/self/fd\ntotal 0\ndr-x------ 2 mota mota  0 Sep 29 16:13 ./\ndr-xr-xr-x 9 mota mota  0 Sep 29 16:13 ../\nlrwx------ 1 mota mota 64 Sep 29 16:13 0 -\u003e /dev/pts/9\nlrwx------ 1 mota mota 64 Sep 29 16:13 1 -\u003e /dev/pts/9\nlrwx------ 1 mota mota 64 Sep 29 16:13 2 -\u003e /dev/pts/9\nlr-x------ 1 mota mota 64 Sep 29 16:13 3 -\u003e /proc/815170/fd/\nlrwx------ 1 mota mota 64 Sep 29 16:13 6 -\u003e /dev/pts/9\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/proc-self-fd.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nBash forks a child process when launching a command and inherits the file descriptors from the parent process.\n\nWe can use `$$` to get the parent process ID:\n\n```bash\n$ echo $$\n2317356\n\n$ ps -p $$\n    PID TTY          TIME CMD\n2317356 pts/9    00:00:00 bash\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/bash-ps-p.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nListing `/proc/$$/fd` will print the same information as before when using `self` because the `$$` is expanded to the same process ID:\n\n```bash\n$ ls -la /proc/$$/fd\ntotal 0\ndr-x------ 2 mota mota  0 Sep 27 19:33 ./\ndr-xr-xr-x 9 mota mota  0 Sep 27 19:33 ../\nlrwx------ 1 mota mota 64 Sep 27 19:33 0 -\u003e /dev/pts/9\nlrwx------ 1 mota mota 64 Sep 27 19:33 1 -\u003e /dev/pts/9\nlrwx------ 1 mota mota 64 Sep 27 19:33 2 -\u003e /dev/pts/9\nlrwx------ 1 mota mota 64 Sep 28 12:20 255 -\u003e /dev/pts/9\nlrwx------ 1 mota mota 64 Sep 27 19:33 6 -\u003e /dev/pts/9\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/proc-psid-fd.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nIn the above examples, `/dev/pts/9` is referencing the pseudo terminal device. A _pts_ is a pseudo terminal device emulated by another program, such as `xterm`, `tmux`, `ssh`, etc.\n\nType the `tty` command to see the pts device path.\n\n```bash\n$ tty\n/dev/pts/9\n```\n\nYou'll see a different number if you open up a new terminal window because it's a new terminal device:\n\n```bash\n$ tty\n/dev/pts/11\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/tty-command.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nIf we we're on a native terminal device (non-pseudo) meaning the backend is hardware or kernel emulated (e.g. the console before launching the desktop environment), then the tty path will look something like `/dev/tty1`.\n\nThe file descriptor table looks like this, where the standard streams are reading/writing from the TTY.\n\n- `0` -\u003e `/dev/pts/9`\n- `1` -\u003e `/dev/pts/9`\n- `2` -\u003e `/dev/pts/9`\n\nWe can write data to the stdout file descriptor and you'll see it be printed back at you:\n\n```bash\n$ echo \"hello world\" \u003e /proc/self/fd/1\nhello world\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/echo-proc-self-fd-1.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nSame thing will occur if writing to the stderr file descriptor:\n\n```bash\n$ echo \"hello world\" \u003e /proc/self/fd/2\nhello world\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/echo-proc-self-fd-2.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nWe can read from from the stdin file descriptor and echo the input:\n\n```bash\n$ echo $(\u003c/proc/self/fd/0)\na\nb\nc\nd\na b c d\n```\n\nAfter running the command and typing text, press `ctrl-d` to stop reading from stdin. The inputed text will be printed.\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/echo-proc-self-fd-0.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nIf you're not familar with the `$(...)` syntax, it allows you to use the result of the command as the the string argument since `$()` evaluates the expression. The `\u003c` is the standard input redirect operator which we'll go over in the redirection section.\n\n### symlinks\n\nFor convenience, the file descriptor path is symlinked to `/dev/fd`.\n\n```bash\n$ ls -l /dev/fd\nlrwxrwxrwx 1 root root 13 Aug 26 23:14 /dev/fd -\u003e /proc/self/fd\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/ls-dev-fd.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nFor convenience, the standard streams are symlinked to `/dev/stdin`, `/dev/stdout`, and `/dev/stderr` respectively.\n\n```bash\n$ ls -l /dev/std*\nlrwxrwxrwx 1 root root 15 Aug 26 23:14 /dev/stderr -\u003e /proc/self/fd/2\nlrwxrwxrwx 1 root root 15 Aug 26 23:14 /dev/stdin -\u003e /proc/self/fd/0\nlrwxrwxrwx 1 root root 15 Aug 26 23:14 /dev/stdout -\u003e /proc/self/fd/1\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/ls-dev-std.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nThese are the same:\n\n- `/dev/stdin` -\u003e `/proc/self/fd/0`\n- `/dev/stdout` -\u003e `/proc/self/fd/1`\n- `/dev/stderr` -\u003e `/proc/self/fd/2`\n\nThe symlinks are considered [POSIX extensions](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap02.html), so they might not be available in all [POSIX](https://en.wikipedia.org/wiki/POSIX) compliant systems.\n\n## Redirection\n\nRedirection operators allow control of where input and output streams should go. When you see `\u003e` it means redirection. The following are some of the redirect operators in Bash:\n\n- `\u003e` - redirect output, overwriting target if exists\n- `\u003e\u003e` - redirect output, appending instead of overwriting if target exists\n- `\u0026#\u003e` - redirect file descriptor _#_, where _#_ is the identifier\n- `\u003c` - redirect input\n\nStandard streams are can be referenced by their file descriptor identifiers. An ampersand `\u0026` followed by the identifier number, ie `\u00261`, references a file descriptor when redirecting.\n\nFor example:\n\n- `command 1\u003e out.log` - outputs file descriptor 1 (stdout) to file.\n- `command 2\u003e out.log` - outputs file descriptor 2 (stderr) to file.\n- `command 3\u003e out.log` - outputs file descriptor 3 (a custom file descriptor) to file.\n\nCommon redirects:\n\n- `1\u003e` - redirects stdout only. This is also the same as simply doing `\u003e`\n- `2\u003e` - redirects stderr only\n- `2\u003e\u00261` - redirects stderr to stdout. The `2\u003e` is redirecting the standard error output into file descriptor `1` which is standard out. The final output will contain both stdout and stderr output, if any.\n- '\u0026\u003e' - redirect stdout and stderr. THis is also the same as the above `2\u003e\u00261`\n\nFor example:\n\n- `command 2\u003e\u00261 \u003e out.log` - says \"point output of FD #2 to FD #1, and ouput FD #1 to out file\".\n\nThe reason an ampersand is required is because `command 2\u003e1` would be ambiguous; it wouldn't be clear if it was redirecting to file descriptor `1` or to a filename named `1`, so the `\u0026` is required to explicitly reference it as the file descriptor.\n\nDo note that `\u00261` (file descriptor `1`) is different than a single `\u0026` (run in background) and double `\u0026\u0026` (AND operator). The ampersand has different meaning depending on the way it's used.\n\nOther redirect operators (this will be explained in later sections):\n\n- `\u003c\u003c` - \"[Here documents](http://tldp.org/LDP/abs/html/here-docs.html)\", a special-purpose code block\n- `\u003c\u003c\u003c` - \"[Here strings](http://www.tldp.org/LDP/abs/html/x17837.html)\", a striped-down form of a here document.\n\n**Example:** demonstration of the different redirect operators:\n\nWrite stdout output of `ls` list format to `list.txt` file:\n\n```bash\n$ ls\narchive.zip book.pdf notes.txt\n\n$ ls -l \u003e list.txt\n\n$ cat list.txt\ntotal 0\n-rw-r--r-- 1 mota mota 0 Sep 30 14:17 archive.zip\n-rw-r--r-- 1 mota mota 0 Sep 30 14:17 book.pdf\n-rw-r--r-- 1 mota mota 0 Sep 30 14:19 list.txt\n-rw-r--r-- 1 mota mota 0 Sep 30 14:17 notes.txt\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/ls-write-stdout.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nAppend new stdout output of `ls` showing hidden files to same `list.txt` file:\n\n```bash\n$ ls -a\n. .. archive.zip  book.pdf  .cache  .config  notes.txt\n\n$ ls -a \u003e\u003e list.txt\n$ cat list.txt\ntotal 0\n-rw-r--r-- 1 mota mota 0 Sep 30 14:17 archive.zip\n-rw-r--r-- 1 mota mota 0 Sep 30 14:17 book.pdf\n-rw-r--r-- 1 mota mota 0 Sep 30 14:19 list.txt\n-rw-r--r-- 1 mota mota 0 Sep 30 14:17 notes.txt\n.\n..\narchive.zip\nbook.pdf\n.cache\n.config\nlist.txt\nnotes.txt\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/ls-append-stdout.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nSearch for particular filenames and write errors from stderr to `errors.txt` file:\n\n```bash\n$ ls *.json\nls: cannot access '*.json': No such file or directory\n\n$ ls *.json 2\u003e errors.txt\n\n$ cat errors.txt\nls: cannot access '*.json': No such file or directory\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/ls-write-stderr.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nRead `errors.txt` file as input to the `less` command:\n\n```bash\n$ less \u003c errors.txt\n\nls: cannot access '*.json': No such file or directory\nerrors.txt (END)\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/less-stdin.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\n### Standard input\n\nstdin (standard input) is an input stream where input data is sent to. The program reads the stream data as input for the program. stdin is a file descriptor we can write to. In most cases the standard input is input from the keyboard.\n\nThe stdin file descriptor is located at `/proc/self/fd/0` but we can use the symlink `/dev/stdin` as well.\n\nData to be sent to program as input is redirected with `\u003c`:\n\n```bash\ncommand \u003c input.txt\n```\n\n**Example:** Read stdin as input for bash script:\n\n`program.sh`:\n\n```bash\nwhile read line\ndo\n  echo \"hello $line\"\ndone \u003c /dev/stdin\n```\n\nCreate file with names:\n\n```bash\n$ printf \"alice\\nbob\\n\" \u003e file.txt\n```\n\nRun program:\n\n```bash\n$ chmod +x program.sh\n$ ./program.sh \u003c file.txt\nhello alice\nhello bob\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/bash-script-stdin-file-reader.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\n\n**Example:** For stdin demonstration purposes, you can send file data as input to the echo command by reading the file into a subshell and using the result as the echo arguments:\n\n```bash\n$ echo \"hello world\" \u003e file.txt\n$ echo $(\u003c file.txt)\nhello world\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/echo-stdin-file.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\n### Standard output\n\nstdout (standard output) is an output stream where data is sent to and then outputted by the terminal. stdout is a file descriptor we can write to.\n\nThe stdout file descriptor is located at `/proc/self/fd/1` but we can use the symlink `/dev/stdout` as well.\n\nThe standard output of a program is redirect with `1\u003e` or simply just `\u003e`:\n\n```bash\ncommand \u003e stdout.log\n```\n\nThe above is the same as `command 1\u003e stdout.log`\n\n**Example:** Redirect stdout to a file:\n\n```bash\n$ echo \"hello world\" \u003e stdout.log\n$ cat stdout.log\nhello world\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/echo-write-stdout.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nTrying to write to a file that can't be opened for writing will make the command fail:\n\n```bash\n$ touch stdout.log\n$ chmod -w stdout.log\n$ echo \"hello world\" \u003e stdout.log\nbash: stdout.log: Permission denied\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/echo-write-stdout-error.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nSometimes when we aren't interested in the program stdout output, we can redirected to `/dev/null` to silence the output. This device file acts like a black hole for data streams.\n\n```bash\ncommand \u003e /dev/null\n```\n\n### Standard error\n\nstderr (standard error) is an output stream where error data is sent to and then outputted by the terminal. stderr is a file descriptor we can write to.\n\nThe stderr file descriptor is located at `/proc/self/fd/2` but we can use the symlink `/dev/stderr` as well.\n\nThe standard error of a program is redirect with `2\u003e`:\n\n```bash\ncommand 2\u003e stdout.log\n```\n\n**Example:**  Redirect stdout to the stderr file descriptor. stderr messages are outputted to the terminal:\n\n```bash\n$ echo \"hello world\" \u003e /dev/stderr\nhello world\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/echo-write-stderr-fd.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\n**Example:** Redirect the standard error messages to a file.\n\nRedirecting with only `\u003e` captures stdout and not stderr:\n\n```bash\n$ ls /foo \u003e out.log\nls: cannot access '/foo': No such file or directory\n$ cat out.log\n\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/ls-error-stdout.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nWe use `2\u003e` to redirect stderr only:\n\n```bash\n$ ls /foo 2\u003e out.log\n$ cat out.log\nls: cannot access '/foo': No such file or directory\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/ls-error-stderr.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nOf course now the following won't write anything to the file because there is no error:\n\n```bash\n$ ls /home 2\u003e out.log\nmota/\n$ cat out.log\n\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/ls-no-error-stderr.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nWe can use `2\u003e\u00261` to redirect stderr to stdout, and then redirect stdout to the file with `\u003e` (or `\u003e\u003e` to append):\n\n```bash\n$ ls /home \u003e out.log 2\u003e\u00261\n$ cat out.log\nmota/\n\n$ ls /foo \u003e\u003e out.log 2\u003e\u00261\n$ cat out.log\nmota/\nls: cannot access '/foo': No such file or directory\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/ls-stderr-to-stdout-append.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nAlternatively, we can redirect stdout to stderr with `1\u003e\u00262` (or simply `\u003e\u00262`), and then redirect stderr to the file with `2\u003e` (or `2\u003e\u003e` to append):\n\n```bash\n$ ls /home 2\u003e out.log 1\u003e\u00262\n$ cat out.log\nmota/\n\n$ ls /foo 2\u003e\u003e out.log 1\u003e\u00262\n$ cat out.log\nmota/\nls: cannot access '/foo': No such file or directory\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/ls-stdout-to-stderr-append.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nSince `\u003e` is shorthand for `1\u003e`, we can replace `1\u003e\u00262` with `\u003e\u00262` and it'll work the same.\n\n**Order is important!**\n\nThe following is what probably seems more intuitive but it won't work as you'd expect:\n\n```bash\n$ ls /foo 2\u003e\u00261 \u003e out.log\nls: cannot access '/foo': No such file or directory\n$ cat out.log\n\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/ls-stderr-to-stdout-before-redirect.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nIt didn't write to the file, and the reason is because stderr was made a copy of stdout before stdout was redirected to the file. It's assigning the right operand to the left operand by copy and not by reference.\n\nBasically the above example is redirecting stderr to whatever stdout currently is (the TTY screen in this case) and then redirects stdout to the file.\n\nMoving the stderr redirect operator `2\u003e\u00261` to after the stdout `\u003e` part correctly copies the error stream to the output stream which is redirecting to the file.\n\n#### Shorthand\n\nWe already learned about the `\u003e` shorthand for `1\u003e`. There's another shorthand for redirecting stderr to stdout to a file or file descriptor. The redirection `\u003e file 2\u003e\u00261` can be shorthanded to `\u0026\u003e`\n\n```bash\n$ ls /home \u0026\u003e out.log\n$ cat out.log\nmota/\n\n$ ls /foo \u0026\u003e\u003e out.log\n$ cat out.log\nmota/\nls: cannot access '/foo': No such file or directory\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/ls-stderr-to-stdout-shorthand.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\n## Pipelines\n\nUsing the `|` pipe operator allows you to send the output of one program as input to another program.\n\n```bash\ncommand 1 | command2\n```\n\nA basic example is filtering output of a program. For example, to only display files that end in `.txt`\n\n```bash\n$ ls\narchive.zip   book.pdf   data.txt  My_Notes.txt\n\n$ ls | grep \"\\.txt$\"\ndata.txt\nMy_Notes.txt\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/ls-pipe-grep.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nYou can chain multiple commands creating a pipeline:\n\n```bash\ncommand 1 | command 2 | command3\n```\n\n**Example:** Add additional lowercase command:\n\n```bash\n$ ls | grep \"\\.txt$\" | tr '[A-Z]' '[a-z]'\ndata.txt\nmy_notes.txt\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/ls-pipe-grep-trim.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nIt's important to note that the commands in pipelines, ie `cmd1 | cmd2 | cmd3`, are all launched in parallel and not ran sequentially. The inputs and outputs are configured appropriately for each program\n\nFor running a series of commands in sequential order then use the following operators:\n\n- `\u0026\u0026` - run command if the last one did not fail (zero exit status code)\n- `||` - run command if the last one failed (non-zero exit status code)\n- `;` - run command regardless of the last exit code\n\n### `\u0026\u0026`\n\nThe AND operator `\u0026\u0026` (double ampersand) is used for separating commands and only running the command if the previous on succeeds:\n\n```bash\ncommand1 \u0026\u0026 command2\n```\n\n**Example:** Continue if condition is true. The `test` command returns exit code `0` if the condition is true.\n\n```bash\n$ test 2 -lt 5 \u0026\u0026 echo \"yes\"\nyes\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/test-exit-0-and-echo.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nIf the `test` condition is false then the circuit breaks because the exit code is non-zero and the execution order doesn't reach the echo command:\n\n```bash\n$ test 7 -lt 5 \u0026\u0026 echo \"yes\"\n\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/test-exit-1-and-echo.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nYou can chain as many commands as you need:\n\n```bash\ncommand1 \u0026\u0026 command2 \u0026\u0026 command3\n```\n\nIt's important to not confuse the `\u0026\u0026` double ampersand with a single `\u0026` ampersand since they do very different things. The single ampersand is used for launching the command list in the background. See [\"\u0026\" section](#\u0026).\n\n### `||`\n\nThe OR operator `||` (double pipe) is used for separating commands and only running the command if the previous one failed:\n\n```bash\ncommand1 || command2\n```\n\n**Example:** Continue if condition is false. The `test` command returns a non-zero exit code if the condition is false.\n\n```bash\n$ test 7 -lt 5 || echo \"yes\"\nyes\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/test-exit-1-or-echo.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nIf the `test` condition is true and exit code is 0 then the execution will stop at the OR statement:\n\n```bash\n$ test 2 -lt 5 || echo \"yes\"\n\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/test-exit-0-or-echo.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\n### `;`\n\nCommands separated by a ; are executed sequentially: one after another.\n  The shell waits for the finish of each command.\n\n     # command2 will be executed after command1\n     command1 ; command2\n\n### `\u0026`\n\nThe single ampersand is used for launching a command or command list in a new subshell in the background. The operator `\u0026` must be at the end of the command:\n\n```bash\ncommand \u0026\n```\n\n**Example:** Run program in background. This command is will be immediately launched in the background and after 5 seconds it will display a desktop notification:\n\n```bash\n$ sleep 5 \u0026\u0026 notify-send \"hello world\" \u0026\n[1] 2481042\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/sleep-and-notify.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nAfter running a command with `\u0026` you'll see the job ID and process ID returned. Run `jobs` to see the list of running processes launched in the background.\n\n```bash\n$ jobs\n[1]+  Running                 sleep 5 \u0026\u0026 notify-send \"hello world\" \u0026\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/jobs-command.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nAfter the command has completed and exited, the status will change to done:\n\n```bash\n$ jobs\n[1]+  Done                    sleep 5 \u0026\u0026 notify-send \"hello world\"\n```\n\nUse the `-l` flag to list the process ID as well:\n\n```bash\n$ jobs -l\n[1]+ 2481042 Done                 sleep 5 \u0026\u0026 notify-send \"hello world\"\n```\n\nIf the command hasn't completed yet, you can bring to the foreground with the `fg` command:\n\n```bash\n$ fg 1\nsleep 5 \u0026\u0026 notify-send \"hello world\"\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/jobs-fg.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nNotice how there's no `\u0026` at the end because the process is no longer running in the background.\n\n**Example:** Launch bash scripts or executible files in the background:\n\n```bash\n$ cat \u003e program.sh\nsleep 5 \u0026\u0026 notify-send \"hello world\"\n^D\n$ chmod +x program.sh\n$ ./program.sh \u0026\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/echo-and-notify-in-background.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\n## Named pipes\n\nThe `mkfifo` allows us to create a special type of file, a FIFO file, which can be opened for writing and reading and behave similar to a pipe. These files are referred to as named pipes.\n\nThe difference between a FIFO file and a regular file is that the FIFO file must be opened on both ends at the same time to let the program continue with input or output operations. The data is passed internally through the kernel without writing it to the file system (the file size is always 0 bytes). This means reading from the FIFO file will be blocked until it's opened for writing, and writing to it will be blocked will until it's opened for reading.\n\n**Example:** create a named piped for writing and reading\n\nFirst we create the named pipe with `mkfifo`:\n\n```bash\n$ mkfifo mypipe\n```\n\nListing the file information shows us that it's a pipe because the file type letter in the attributes is `p`\n\n```bash\n$ ls -l mypipe\nprw-r--r-- 1 mota mota 0 Oct 25 02:14 mypipe\n```\n\nIn one terminal, we redirect some standard output to the named pipe:\n\n```bash\n$ echo \"hello world\" \u003e mypipe\n\n```\n\nNotice how it appears to hang after running the command. The pipe is blocked until another process reads from the pipe.\n\nIn another terminal, redirect standard input of the pipe into `cat` to read and print the contents that were sent to the pipe in the first terminal. This also unblocks the pipe since both ends are simultaneously opened.\n\n```bash\n$ cat \u003c mypipe\nhello world\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/mkfifo-echo-cat.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nThe FIFO device file is on disk so we have to manually delete it if we're done using it:\n```bash\n$ rm mypipe\n```\n\nAnother option is to create FIFO files in `/tmp` which will get automatically wiped after a restart.\n\n## Command grouping\n\nCommands can be grouped using curly braces `{...}`\n\n```bash\n$ { command; command; command; }\n```\n\n**Important!** there must be a space separating the command and the curly brace and the last command needs to be terminated by a semicolon for the group to be executed correctly.\n\nAnother way to group commands is by using a subshell `(...)`.\n\n```bash\n$ (command; command; command)\n```\n\nGrouping with subshells does not require the space separation and last command semicolon as like grouping with curly braces. There are differences in grouping using a subshell and subshells are explained further in the [subshells section](#subshells)\n\nGrouping commands is useful for managing redirection. For example, we can redirect the output of multiple programs to a single location without adding redudant redirects.\n\nFor context:\n\n```bash\n$ ls\ndata.json  list.txt\n$ cat list.txt\narchive.zip\nbook.pdf\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/ls-cat-list.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nWe can take file write redirects like these:\n\n```bash\n$ date \u003e out.log\n$ ls \u003e\u003e out.log\n$ cat list.txt \u003e\u003e out.log\n$ cat out.log\nSat Oct 10 09:35:06 PM PDT 2020\ndata.json\nlist.txt\nout.log\narchive.zip\nbook.pdf\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/multiple-commands-stdout-write.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nGroup them to simplify things:\n\n```bash\n$ { date; ls; cat list.txt; } \u003e out.log\n$ cat out.log\nSat Oct 10 09:35:06 PM PDT 2020\ndata.json\nlist.txt\nout.log\narchive.zip\nbook.pdf\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/command-group-stdout-file.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nA command group can be piped to another command as if it were a single standard input:\n\n```bash\n$ { date; ls; cat list.txt; } | tail -n+2 | sort\narchive.zip\nbook.pdf\ndata.json\nlist.txt\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/command-group-stdout-pipe.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nSimilarly, grouping can be done with a subshell:\n\n```bash\n$ (date; ls; cat list.txt) | tail -n+2 | sort\narchive.zip\nbook.pdf\ndata.json\nlist.txt\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/subshell-command-group-stdout-pipe.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\n### pipefail\n\nBy default, a bash pipeline's exit status code will be whichever exit code the last command returned, meaning a non-zero exit code is not preserved throughout the pipeline.\n\n**Example:** here we have a program that has a failing pipeline however the exit code is `0`. The last exit code can be read from the variable `$?`.\n\n```bash\n$ cat \u003e program.sh\nls /foo | tee out.log; echo $?\necho \"done\"\n^D\n$ chmod +x program.sh\n$ ./program.sh\nls: cannot access '/foo': No such file or directory\n0\ndone\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/bash-script-pipeline-fail.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nThe bash `set` builtin command allows us to configure shell options. One important option is the `set -o pipefail` option which causes the pipeline's exit code to be preserved if a command fails in the pipeline:\n\n```bash\n$ cat \u003e program.sh\nset -o pipefail\nls /foo | tee out.log; echo $?\necho \"done\"\n^D\n$ chmod +x program.sh\n$ ./program.sh\nls: cannot access '/foo': No such file or directory\n2\ndone\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/bash-script-pipefail-option.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nThe `ls` man page mentions the following reasons for exit status codes:\n\n- `0` - if OK\n- `1` - if minor problems\n- `2` - if serious trouble\n\nNotice how the last echo command still got executed after the pipeline failed. We can combine the `pipefail` option with the `set -e` (`errexit`) option to immediately exit the script if any command fails:\n\n```bash\n$ cat \u003e program.sh\nset -eo pipefail\nls /foo | tee out.log; echo $?\necho \"done\"\n^D\n$ chmod +x program.sh\n$ ./program.sh\nls: cannot access '/foo': No such file or directory\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/bash-script-pipefail-errexit-option.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nThe program exited immediately after the failing `ls` command and didn't print any commands after it.\n\n## Process Substitution\n\nProcess substitution allows us to run a program and write to another program as if it were a file. The syntax for process substitution is `\u003e(command)` for writing to the program as an output file or `\u003c(command)` for using the program as an input file.\n\n- `\u003c(command)` - for programs that produce standard output\n- `\u003e(command)` - for programs that intake standard input\n\nThe operator `\u003c()` or `\u003e()` creates a temporary file descriptor that manages reading and writing the substituted program.\n\nIt's important that theres no space between the `\u003c` or `\u003e` and the parentheses `(` otherwise it would result in an error. Although it looks similar, process substitution is different than command grouping or subshells.\n\n**Example:** Print the file descriptor created by process substitution:\n\nWe can use the echo command to view the result of the expansion:\n\n```bash\n$ echo \u003c(date)\n/dev/fd/63\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/echo-process-substitution-date.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\n**Example:** Print the contents of the file created by process substitution:\n\n```bash\n$ cat \u003c(date)\nSat Oct 10 12:56:18 PM PDT 2020\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/cat-process-substitution-date.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\n**Example:** command tee stdout to cat\n\nThe tee command accepts only files to write to but using process substitution we can write the output to cat. This results in the date command being printed and the cat command printing the date as well.\n\n```bash\n$ date | tee \u003e(cat)\nSat Oct 10 01:27:15 PM PDT 2020\nSat Oct 10 01:27:15 PM PDT 2020\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/cat-tee-process-substitution.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\n**Example:** send command stderr to substituted file while also logging stdout and stderr:\n\n```bash\ncommand 2\u003e tee \u003e(cat \u003e\u00262)\n```\n\nThe `\u003e()` operator substitute the tee command as a file and within that process substitution the cat command is substituted as a file so tee can write to it. The `2\u003e` operator sends only stderr to outer substituted file. The operator `\u003e\u00262` copies stdout to stderr.\n\nIf there is no stderr from the command then nothing is sent to the tee substituted file:\n\n```bash\n$ ls /home 2\u003e \u003e(tee \u003e(cat \u003e\u00262))\nmota/\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/ls-tee-cat-process-substitution.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nIf there is stderr from the command then the tee process substitution will process it and log it:\n\n```bash\n$ ls /foo 2\u003e \u003e(tee \u003e(cat \u003e\u00262))\nls: cannot access '/foo': No such file or directory\nls: cannot access '/foo': No such file or directory\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/ls-tee-cat-process-substitution-stderr.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\n## Subshells\n\nA subshell executes commands in a child copy of the current shell. The environment is copied to the new instance of the shell when running subshelled commands. The copy of the environment is deleted once the subshell exits so changes, such as environment variables assignments, in the subhsell are lost when it exits. Command grouping is preferred to subshells in most cases because it's faster and uses less memory.\n\nWrap the command(s) in parentheses `(...)` to launch them in a subshell:\n\n```bash\n$ (command)\n```\n\n**Example:** running a command in a subshell:\n\nNotice how the second environment variable echo is not printed because the variable was set in the subshell environment:\n\n```bash\n$ (FOO=bar; echo $FOO); echo $FOO\nbar\n\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/subshell-environment-variable.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\n**Example:** using process substitution to get around subshell caveats:\n\nAs an example, the `read` command can be used for caching input. The read input is copied to the `$REPLY` environment variable.\n\n```bash\n$ read\nhello world\n$ echo $REPLY\nhello world\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/read-command.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nHowever if we pipe a string to the read command, it will not print the string as expected after reading it:\n\n```bash\n$ echo \"hello world\" | read\n$ echo $REPLY\n\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/echo-pipe-read-command.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nThis is because read command is launched in a subshell when it's in a pipeline and the `REPLY` variable copy is lost after it exits. Commands in pipelines are executed in subshell and any variable assignments will not be available after the subshell terminates.\n\nWe can use process substitution to get around this problem so a subshell doesn't to be initialized:\n\n```bash\n$ read \u003c \u003c(echo \"hello world\")\n$ echo $REPLY\nhello world\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/echo-read-process-substitution.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\n## Examples\n\nThe following are various examples utilizing bash pipelines and redirections:\n\n### Pipe only on stderr\n\n```bash\n# will echo message only if command is not found\n$ command -v mycommand \u0026\u003e/dev/null || echo \"command not found\"\n```\n\n### Echo to stderr\n\nCopy stderr file descriptor #1 to stdout file descriptor #2:\n\n```bash\necho \"this will go to stderr\" 1\u003e\u00262\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/echo-redirect-stdout-to-stderr.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nYou can omit the `1` since `\u003e` is the same as `1\u003e`:\n\n```bash\necho \"this will go to stderr\" \u003e\u00262\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/echo-redirect-stdout-to-stderr-shorthand.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\nTo make it more readable, the redirect can be moved to the front:\n\n```bash\n\u003e\u00262 echo \"this will go to stderr\"\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/echo-redirect-stdout-to-stderr-in-front.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\n### Diff two commands\n\nDiff the output of two commands using process substitution:\n\n```bash\ndiff \u003c(command) \u003c(command)\n```\n\n**Example 1:**\n\n```bash\n$ diff \u003c(xxd file1.bin) \u003c(xxd file2.bin)\n```\n\n**Example 2:**\n\n```bash\n$ diff \u003c(printf \"foo\\nbar/nqux\\n\") \u003c(printf \"foo\\nbaz\\nqux\\n\")\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/diff-two-commands.gif\" alt=\"example gif\" /\u003e\n\u003c/details\u003e\n\n### Record SSH session\n\nUse `tee` to record an SSH session:\n\n```bash\nssh user@server | tee /path/to/file\n```\n\n**Example:**\n\n```bash\n$ ssh root@example.com | tee session.log\n\n# after exiting server\n$ cat session.log\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/record-ssh-session.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\n### Split pipe into multiple streams\n\nSplit a pipe into two separate pipes using `tee` and process substitution:\n\n```bash\ncommand | tee \u003e(command)\n```\n\n**Example 1:** echo text and reverse the text in second stream:\n\n```bash\n$ echo \"split this pipe\" | tee \u003e(rev)\nsplit this pipe\nepip siht tilps\n```\n\nYou're not limited to just one; add as many additional streams as you like:\n\n```bash\n$ echo \"split this pipe\" | tee \u003e(rev) \u003e(tr ' ' '_') \u003e(tr a-z A-Z)\nsplit this pipe\nSPLIT THIS PIPE\nsplit_this_pipe\nepip siht tilps\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/split-pipe.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\n**Example 2:** Run command and copy output to clipboard:\n\n```bash\n$ echo \"hello world\" | tee \u003e(copy)\nhello world\n```\n\n### Send text to another terminal\n\nEcho text from one TTY to another TTY:\n\n```bash\ncommand | /dev/pts/{id}\n```\n\n**Example:**\n\nTerminal 1\n\n```bash\n$ tty\n/dev/pts/39\n```\n\nTerminal 2\n\n```bash\n$ echo \"this will show up in terminal 1\" \u003e /dev/pts/39\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/echo-to-tty.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\n### Pipe terminal output to another terminal\n\nPipe stdout and stderr output of current TTY to another TTY:\n\n```bash\n$ exec \u0026\u003e \u003e(tee \u003e(cat \u003e /dev/pts/{id}))\n```\n\n**Example:**\n\nTerminal 1\n\n```bash\n$ tty\n/dev/pts/39\n```\n\nTerminal 2\n\n```bash\n$ exec \u0026\u003e \u003e(tee \u003e(cat \u003e /dev/pts/39))\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/tty-output-to-tty.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\nAnother way is to use the `script` command. The `script` command allows you to record terminal sessions. Here we specify the TTY as the output file:\n\n```bash\nscript -q /dev/pts/{id} command\n```\n\n**Example:**\n\nTerminal 1:\n\n```bash\n$ tty\n/dev/pts/12\n```\n\nTerminal 2:\n\n```bash\n$ script -q /dev/pts/12 bash\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/tty-output-to-tty-using-script.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\n### Read pipe into variable\n\n```bash\n$ read varname \u003c \u003c(command)\n```\n\n**Example:**\n\n```bash\n$ read myvar \u003c \u003c(echo \"hello world\")\n\n$ echo $myvar\nhello world\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/read-pipe-into-variable.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\n### Pipe to file descriptor\n\n```bash\ncommand | tee /dev/fd/{id}\n```\n\n**Example:**\n\n```bash\n$ echo \"hello world\" | tee /dev/fd/3\n```\n\nHowever, the above won't work on all systems. The cross-platform compatible way is to use process substitution:\n\n```bash\n$ command \u003e \u003e(tee \u003e(cat \u003e\u00263))\n```\n\n### Read stdin line by line in Bash\n\nSet stdin as input file:\n\n```bash\nwhile read line\ndo\n  echo \"echo: $line\"\ndone \u003c /dev/stdin\n```\n\n**Example:**\n\n```bash\n$ cat | reader.sh\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/bash-stdin-reader.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\n### Read command output as line by line in Bash\n\n```bash\nwhile read line\ndo\n  echo \"$line\"\ndone \u003c \u003c(command)\n```\n\n**Example:**\n\n```bash\nwhile true; do date; sleep 1; done \u003e stream.log\n```\n\n```bash\nwhile read line\ndo\n  echo \"$line\"\ndone \u003c \u003c(tail -n0 -f stream.log)\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/bash-file-reader.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\nAnother way of reading command output line by line:\n\n```bash\ncommand | while read line\ndo\n  echo \"$line\"\ndone\n```\n\n**Example:**\n\n```bash\ntail -n0 -f stream.log | while read line\ndo\n  echo \"$line\"\ndone\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/bash-file-reader-2.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\n### Pipe terminal to another computer's terminal\n\nPipe your terminal to an open TCP socket file descriptor:\n\n```bash\n$ exec 3\u003c\u003e/dev/tcp/{hostname}/{port} \u0026\u0026 exec \u0026\u003e \u003e(tee \u003e(cat \u003e\u00263))\n```\n\n**Example:**\n\nTerminal 1\n\n```bash\n$ nc -l -p 1337\n```\n\nTerminal 2\n\n```bash\n$ exec 3\u003c\u003e/dev/tcp/127.0.0.1/1337 \u0026\u0026 exec \u0026\u003e \u003e(tee \u003e(cat \u003e\u00263))\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/pipe-tty-to-tcp.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\nAlternatively, you can use netcat to pipe your terminal:\n\n```bash\n$ exec \u0026\u003e \u003e(nc 127.0.0.1 1337)\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/pipe-tty-to-tcp-using-netcat.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\n### Redirect the output of multiple commands\n\n```bash\n{ command1; command2; command3; } \u003e stdout.log 2\u003e stderr.log\n```\n\n**Example:**\n\n```bash\n$ { date ; echo \"ok\"; \u003e\u00262 echo \"error!\"; } \u003e stdout.log 2\u003e stderr.log\n\n$ cat stdout.log\nSat 29 Aug 2020 11:16:39 AM PDT\n\n$ cat stderr.log\nerror!\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/multiple-commands-redirect-output.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\n### Stream audio\n\nStream audio to terminal audio player:\n\n```bash\ncurl -s {http-stream-url} | mpv -\n```\n\n**Example:** Streaming mp3 audio from [somafm](https://somafm.com/) to [mpv](https://mpv.io/) player:\n\n```bash\n$ curl -s http://ice1.somafm.com/defcon-128-mp3 | mpv -\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/stream-mp3.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\n**Example:** Using [`afplay`](https://ss64.com/osx/afplay.html) player (preinstalled on macOS). Note `afplay` doesn't support streaming so we create a file descriptor to stream to.\n\n```bash\n$ exec 3\u003c\u003e /tmp/file.mp3 \u0026\u0026 curl -s http://ice1.somafm.com/defcon-128-mp3 | tee \u003e\u00263 | (sleep 1; afplay /tmp/file.mp3)\n```\n\n**Example:** using [`ffplay`](https://ffmpeg.org/ffplay.html) player (preinstalled on Fedora):\n\n```bash\n$ curl -s http://ice1.somafm.com/defcon-128-mp3 | ffplay -nodisp -\n```\n\n**Example:** Using [`youtube-dl`](https://ytdl-org.github.io/youtube-dl/) to get the m3u8 playlist url for `mpv` to stream:\n\n```bash\n$ youtube-dl -f best -g https://www.youtube.com/watch?v=dQw4w9WgXcQ | xargs -I % curl -s % | mpv --no-video -\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/youtube-dl-pipe-to-mpv.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\n### Stream directory contents to remote server\n\nServer:\n\n```bash\nnc -l -s 0.0.0.0 -p 1337 | tar xf -\n```\n\nClient:\n\n```bash\ntar cf - /some/directory | nc {hostname} 1337\n```\n\n**Example:** pipe all content from current client directory to server:\n\nServer:\n\n```bash\n$ nc -l -p 1337 | tar xf -\n```\n\nClient:\n\n```bash\ntar cf - . | nc 127.0.0.1 1337\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/stream-directory-contents.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\nA thing to note is that it'd be very easy to for someone to stream all your home directory contents (SSH keys, AWS credentials, etc) if they're able to run this command on your machine! Only run trusted software and monitor outgoing HTTP requests using something like [OpenSnitch](https://github.com/evilsocket/opensnitch).\n\n### Take webcam picture on mouse move\n\n**Example:** trigger a webcam picture to be taken when mouse movement events is read from `/dev/input/mouse0`, and wait 10 seconds before listening for another mouse event again:\n\n```bash\nwhile true; do sudo cat /dev/input/mouse0 | read -n1; streamer -q -o cam.jpeg -s 640x480 \u003e /dev/null 2\u003e\u00261; sleep 10; done\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/mouse-move-webcam-picture.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\n### Group commands with OR\n\nGroup commands with OR operator to try different commands until one succeeds and pipe the result to the next command:\n\n```bash\n$ ( command || command || command ) | command\n```\n\n**Example:** attempt to deflate a gzipped file and pipe text to `less`:\n\n```bash\n$ echo \"hello world\" \u003e stream.log; gzip stream.log; FILE=stream.log.gz\n$ ( zcat $FILE || gzcat $FILE || bzcat2 $FILE ) | less\nhello world\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/group-commands-with-or.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\n### Writing standard input to a file\n\n```bash\n$ cat \u003e file.txt\nhello world\n^D\n\n$ cat file.txt\nhello world\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/cat-stdin-write.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\nIt's the same thing using the `-` argument in cat to indicate that you want to read from stdin, e.g. `cat - \u003e file.txt`\n\n### Concatenating files with standard input in between\n\n**Example:** With `cat` can use the `-` in place of a file name to read from stdin. Press _ctrl-d_ to exit the stdin prompt:\n\n```bash\n$ echo \"hello\" \u003e 1.txt\n$ echo \"world\" \u003e 3.txt\n$ cat 1.txt - 3.txt \u003e all.txt\nearth\n^D\n\n$ cat all.txt\nhello\nearth\nworld\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/cat-with-stdin-between-files.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\n### Send commands to terminal through a named pipe\n\n**Example:** Set standard input of terminal to FIFO file\n\nIn terminal 1, create a FIFO file and replace terminal standard input by using `exec`:\n\n```bash\n$ mkfifo myfifo\n$ exec \u003c myfifo\n```\n\nIn terminal 2, write to the FIFO file and see the command being executed in the first terminal. However, the first terminal will close right away. This is because the writer closed the FIFO and the reading process received `EOF`.\n\n```bash\n$ echo \"ls -l\" \u003e myfifo\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/mkfifo-exec-stdin-echo.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\nWe can create a file descriptor with an open connection to the FIFO pipe to prevent the terminal from closing when writing commands to it.\n\nIn temrinal 1, run `exec` again to replace standard input:\n\n```bash\n$ exec \u003c myfifo\n```\n\nIn terminal 2, use `exec` to create a custom file descriptor `3` and redirect the standard output to the named pipe. Now we can echo commands to this file descriptor and the first terminal will execute them and remain opened.\n\n```bash\n$ exec 3\u003e myfifo\n$ echo \"ls -l\" \u003e\u00263\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/mkfifo-exec-stdin-file-descriptor-echo.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\nUse the FD close operator `{fd}\u003e\u0026-` with `exec` to close the file descriptor opened for writing to the FIFO:\n\n```bash\n$ exec 3\u003e\u0026-\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/exec-close-file-descriptor.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\n### Filter input for reading with process substitution\n\nIn this example, we'll create a program that will intake a filtered output of `ls -l` and print a formatted string.\n\nPrint long form of `ls`:\n\n```bash\n$ ls -l\ntotal 8\n-rw-r--r-- 1 mota mota 2247 Oct 10 19:51 book.pdf\n-rw-r--r-- 1 mota mota  465 Oct 10 19:51 data.txt\n```\n\nStrip out first line:\n\n```bash\n$ ls -l | tail -n+2\n-rw-r--r-- 1 mota mota 2247 Oct 10 19:51 book.pdf\n-rw-r--r-- 1 mota mota  465 Oct 10 19:51 data.txt\n```\n\nPrint only the size and filename columns:\n\n```bash\n$ ls -l | tail -n+2 | awk '{print $5 \" \" $9}'\n2247 book.pdf\n465 data.txt\n```\n\nNow that we know what filter pipeline we'll use, let's create a program that reads line by line the output of the pipeline through process substitution as standard input and prints a formatted string for every line:\n\n`program.sh`\n\n```bash\nwhile read size filename; do\n  cat \u003c\u003c EOF\n$filename is $size bytes\nEOF\ndone \u003c \u003c(ls -l | tail -n+2 | awk '{print $5 \" \" $9}')\n```\n\n```bash\n$ ./program.sh\nbook.pdf is 2247 bytes\ndata.txt is 465 bytes\n```\n\n\u003cdetails\u003e\n  \u003csummary\u003eexample gif\u003c/summary\u003e\n  \u003cp\u003e\u003cimg src=\"gifs/bash-script-ls-process-substitution.gif\" alt=\"example gif\" /\u003e\u003c/p\u003e\n\u003c/details\u003e\n\n## Contributing\n\nPull requests are welcome!\n\nFor contributions please create a new branch and submit a pull request for review.\n\n## Resources\n\n- [GNU Bash Manual - Redirections](https://www.gnu.org/software/bash/manual/html_node/Redirections.html)\n- [Advanced Bash-Scripting Guide: Process Substitution](https://tldp.org/LDP/abs/html/process-sub.html)\n- [Introduction to Linux - I/O redirection](https://linux.die.net/Intro-Linux/chap_05.html)\n- [The Linux Command Line](http://linuxcommand.org/tlcl.php)\n- [Bash One-Liners Explained: All about redirections](https://catonmat.net/bash-one-liners-explained-part-three)\n- [Bash Redirection Cheat Sheet](https://catonmat.net/ftp/bash-redirections-cheat-sheet.pdf)\n\n## License\n\n[MIT](LICENSE) @ [Miguel Mota](https://github.com/miguelmota/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiguelmota%2Fbash-streams-handbook","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmiguelmota%2Fbash-streams-handbook","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiguelmota%2Fbash-streams-handbook/lists"}