{"id":29857966,"url":"https://github.com/jaalto/project--shell-script-performance-and-portability","last_synced_at":"2025-07-30T01:11:16.626Z","repository":{"id":276847293,"uuid":"930481782","full_name":"jaalto/project--shell-script-performance-and-portability","owner":"jaalto","description":"Shell programing, shell script performance tests. How can you make faster and more portable shell scripts? Keywords: shell, sh, POSIX, bash, ksh93, programming, optimization, performance, profiling, portability.","archived":false,"fork":false,"pushed_at":"2025-04-09T12:36:42.000Z","size":1003,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-09T13:23:32.043Z","etag":null,"topics":["ksh","ksh93","optimization","performance","portability","posix","posix-sh","posix-shell","profiling","programming","sh","shell"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jaalto.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-10T17:47:29.000Z","updated_at":"2025-04-09T12:36:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"942eca72-a045-458e-9105-6838f0c8bf71","html_url":"https://github.com/jaalto/project--shell-script-performance-and-portability","commit_stats":null,"previous_names":["jaalto/project--shell-script-performance","jaalto/project--shell-script-performance-and-portability"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jaalto/project--shell-script-performance-and-portability","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaalto%2Fproject--shell-script-performance-and-portability","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaalto%2Fproject--shell-script-performance-and-portability/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaalto%2Fproject--shell-script-performance-and-portability/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaalto%2Fproject--shell-script-performance-and-portability/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jaalto","download_url":"https://codeload.github.com/jaalto/project--shell-script-performance-and-portability/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaalto%2Fproject--shell-script-performance-and-portability/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267791106,"owners_count":24144895,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-29T02:00:12.549Z","response_time":2574,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ksh","ksh93","optimization","performance","portability","posix","posix-sh","posix-shell","profiling","programming","sh","shell"],"created_at":"2025-07-30T01:11:10.031Z","updated_at":"2025-07-30T01:11:16.541Z","avatar_url":"https://github.com/jaalto.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!--\nINFORMATION FOR EDITÍNG\n\n- Github Markdown Guide:\n  https://is.gd/nqSonp\n\n- VSCode:\n  Palette: C-p\n  Open preview markdown C-S-v\n\n- URL text fragments: #:~:text=\n  https://developer.mozilla.org/en-US/docs/Web/URI/Reference/Fragment/Text_fragments\n\n- About accessibility\n\n  To support viewing and editing GitHub\n  pages on phone displays, the maximum\n  column widths are described below.\n  Exception: The GNU License at the end\n  of file is included verbatim.\n\n  col type\n  ---------------------------------\n  35  Code: bullet: ``` ... ``´)\n  41  Regular text and paragaphs.\n      Github line limit to support\n      editing.\n  ---------------------------------------\n\n  In Emacs:\n  \u003cgo to the end of above \"-\"\u003e\n  C-u C-x f (M-x set-fill-column)\n  M-x display-fill-column-indicator-mode\n\nMISCELLANEOUS\n\n- To search POSIX.1-2024 in Google\n  site:pubs.opengroup.org inurl:9799919799 \u003csearch\u003e\n\n--\u003e\n\n# 1.0 SHELL SCRIPT PERFORMANCE AND PORTABILITY\n\nHow can you make shell scripts portable\nand run faster? That are the questions\nthese test cases aim to answer.\n\nTable of Contents\n\n- [1.0 SHELL SCRIPT PERFORMANCE AND PORTABILITY](#10-shell-script-performance-and-portability)\n  - [1.1 THE PROJECT STRUCTURE](#11-the-project-structure)\n  - [1.2 THE PROJECT DETAILS](#12-the-project-details)\n- [3.0 ABOUT PERFORMANCE](#30-about-performance)\n  - [3.1 GENERAL PERFORMANCE ADVICE](#31-general-performance-advice)\n  - [3.2 SHELLS AND PERFORMANCE](#32-shells-and-performance).\n  - [3.3 MAJOR PERFORMANCE GAINS](#33-major-performance-gains)\n  - [3.4 MODERATE PERFORMANCE GAINS](#34-moderate-performance-gains)\n  - [3.5 MINOR PERFORMANCE GAINS](#35-minor-performance-gains)\n  - [3.6 NO PERFORMANCE GAINS](#36-no-performance-gains)\n- [4.0 PORTABILITY](#40-portability)\n  - [4.1 LEGACY SHELL SCRIPTING](#41-legacy-shell-scripting)\n  - [4.2 REQUIREMENTS AND SHELL SCRIPTS](#42-requirements-and-shell-scripts)\n  - [4.3 WRITING POSIX COMPLIANT SHELL SCRIPS](#43-writing-posix-compliant-shell-scrips)\n  - [4.4 SHEBANG LINE IN SCRIPTS](#44-shebang-line-in-scripts)\n    - [4.4.1 About Bash and Shebang](#441-about-bash-and-shebang)\n    - [4.4.2 About Python and Shebang](#442-about-python-and-shebang)\n  - [4.5 PORTABILITY OF UTILITIES](#45-portability-of-utilities)\n    - [4.5.1 Case Study: sed](#451-case-study-sed)\n    - [4.5.2 Case Study: awk](#451-case-study-awk)\n  - [4.6 MISCELLANEUS NOTES](#46-miscellaneus-notes)\n- [5.0 RANDOM NOTES](#50-random-notes)\n- [6.0 FURTHER READING](#60-further-reading)\n  - [6.1 BASH PROGRAMMING](#61-bash-programming)\n  - [6.2 PORTABILITY AND UTILITIES](#62-portability-and-utilities)\n  - [6.3 STANDARDS](#63-standards)\n  - [6.4 MISCELLANEOUS LINKS](#64-miscellaneous-links)\n- [COPYRIGHT](#copyright)\n- [LICENSE](#license)\n\nThe tests reflect results under Linux\nusing GNU utilities. The focus is on the\nfeatures found in\n[Bash](https://www.gnu.org/software/bash)\nand\n[POSIX.1-2024](https://pubs.opengroup.org/onlinepubs/9799919799/)\ncompliant `sh` shells. The term compliant\nis used here as \"most POSIX compliant\",\nas there is no, and has never been,\nshell that is fully POSIX compliant.\nPOSIX is useful if you are looking for\nmore portable scripts. See also POSIX in\n[Wikipedia](https://en.wikipedia.org/wiki/POSIX).\n\n\u003e Please note that `sh` here refers to\n\u003e modern, best-of-effort POSIX-compatible,\n\u003e minimal shells like\n\u003e [dash](https://tracker.debian.org/pkg/dash)\n\u003e and\n\u003e [posh](https://tracker.debian.org/pkg/posh).\n\u003e See section [PORTABILITY, SHELLS AND POSIX](#posix-shells-and-portability).\n\nIn Linux like systems, for all rounded\nshell scripting, Bash is the sensible\nchoice for data manipulation in memory\nwith\n[arrays](https://www.gnu.org/software/bash/manual/html_node/Arrays.html),\nassociative arrays, and strings with an\nextended set of parameter expansions,\nregular expressions, including extracting\nregex matches and utilizing functions.\n\nIn other operating systems, for example\nBSD, the obvious choice for shell\nscripting would be fast\n[Ksh](https://en.wikipedia.org/wiki/KornShell)\n([ksh93](https://tracker.debian.org/pkg/ksh93u+m),\n[mksh](https://tracker.debian.org/pkg/mksh),\netc.).\n\nShell scripting is about combining\nredirections, pipes, calling external\nutilities, and gluing them all together.\nShell scripts are also quite portable by\ndefault, requiring no additional\ninstallation.\n[Perl](https://www.perl.org)\nor\n[Python](https://www.python.org)\nexcel in their respective fields, where\nthe requirements differ from those of the\nshell.\n\nCertain features in Bash are slow, but\nknowing the cold spots and using\nalternatives helps. On the other hand,\nsmall POSIX `sh`, for example\n[dash](https://tracker.debian.org/pkg/dash),\nscrips are much faster at calling\nexternal processes and functions. More\nabout this in section\n[SHELLS AND PERFORMANCE](#32-shells-and-performance).\n\nThe results presented in this README\nprovide only some highlighs from the test\ncases listed in RESULTS. Consider the raw\n[`time`](https://www.gnu.org/software/bash/manual/bash.html#Reserved-Words)\nresults only as guidance, as they reflect\nonly the system used at the time of\ntesting. Instead, compare the relative\norder in which each test case produced\nthe fastest results.\n\n## 1.1 THE PROJECT STRUCTURE\n\n- [RESULTS](./doc/RESULTS.md)\n- [RESULTS-BRIEF](./doc/RESULTS-BRIEF.txt)\n- [RESULTS-PORTABILITY](./doc/RESULTS-PORTABILITY.txt)\n- The test cases and code in [bin/](./bin/)\n- [USAGE](./USAGE.md)\n- [CONTRIBUTING](./CONTRIBUTING.md)\n\n```\n    bin/            The tests\n    doc/            Results by \"make doc\"\n    COPYING         License (GNU GPL)\n    INSTALL         Install instructions\n    USAGE.md        How to run the tests\n    CONTRIBUTING.md Writing test cases\n```\n\n## 1.2 THE PROJECT DETAILS\n\n- Homepage:\n  https://github.com/jaalto/project--shell-script-performance-and-portability\n\n- To report bugs:\n  see homepage.\n\n- Source repository:\n  see homepage.\n\n- **Depends**:\n  Bash, GNU coreutils, file\n  `/usr/share/dict/words`\n  (Debian package: wamerican).\n\n- **Optional depends**:\n  GNU make.\n  For some tests: GNU parallel.\n\n# 3.0 ABOUT PERFORMANCE\n\n## 3.1 GENERAL PERFORMANCE ADVICE\n\nRegardless of the shell you use for\nscripting\n([sh](https://tracker.debian.org/pkg/dash),\n[ksh](https://tracker.debian.org/pkg/ksh93u+m),\n[bash](https://www.gnu.org/software/bash)),\nconsider these factors.\n\n\n- If you run scripts on many small files,\n  set up a RAM disk and copy the files to\n  it. This can lead to massive speed\n  gains. In Linux, see\n  [tmpfs](https://en.wikipedia.org/wiki/Tmpfs),\n  which allows you to set a size limit,\n  unlike the memory-hogging\n  [ramfs](https://wiki.debian.org/ramfs),\n  which can fill all available memory and\n  potentially halt your server.\n\n- If you know the files beforehand,\n  preload them into memory. This can also\n  lead to massive speed gains.\n  In Linux, see\n  [vmtouch](https://hoytech.com/vmtouch/).\n\n- If you have tasks that can be run concurrently,\n  use\n  [Perl](https://www.perl.org)\n  based\n  [GNU parallel](https://www.gnu.org/software/parallel/)\n  for massive gains in performance.\n  See also how to use\n  [semaphores](https://www.gnu.org/software/parallel/sem.html#understanding-a-semaphore)\n  ([tutorial](https://www.gnu.org/software/parallel/parallel_examples.html#example-working-as-mutex-and-counting-semaphore))\n  to wait for all concurrent tasks to\n  finish before continuing with the rest\n  of the tasks in the pipeline. In some\n  cases, even parallelizing work with GNU\n[`xargs --max-procs=0`](https://www.gnu.org/software/findutils/manual/html_node/find_html/xargs-options.html)\ncan help.\n\n- Use GNU utilities. According to\n  benchmarks, like\n  [StackOverflow](https://stackoverflow.com/a/22661643),\n  the GNU `grep` is considerably faster\n  and more optimized than the operating\n  system's default. For shells, the GNU\n  utilities consist mainly of\n  [coreutils](https://tracker.debian.org/pkg/coreutils),\n  [grep](https://tracker.debian.org/pkg/grep) and\n  [awk](https://tracker.debian.org/pkg/gawk).\n  If needed, arrange `PATH` to prefer\n  GNU utilities (for example, on\n  macOS).\n\n- Minimize extra processes as much as\n  possible. In most cases, a single\n[awk](https://www.gnu.org/software/gawk/)\n  can handle all of\n  [`sed`](https://www.gnu.org/software/sed/),\n  [`cut`](https://www.gnu.org/software/coreutils/manual/html_node/index.html),\n  [`grep`](https://www.gnu.org/software/grep/)\n  etc.\n  chains. The `awk` binary program is *very*\n  fast and more efficient than\n  [Perl](https://www.perl.org)\n  or\n  [Python](https://www.python.org)\n  scripts where startup time and higher\n  memory consumption is a factor.\n  *Note*: If you need to process large\n  files, use a lot of regular\n  expressions, manipulate or work on\n  data extensively, there is probably\n  nothing that can replace the speed of\n  [Perl](https://www.perl.org)\n  unless you go even lower-level\n  languages like `C`. But then again,\n  we assume that you know how to choose\n  your tools in those cases.\n\n```bash\n    cmd | awk '{...}'\n\n    # ... could probably\n    # replace all of these\n\n    cmd | head ... | cut ...\n    cmd | grep ... | sed ...\n    cmd | grep ... | grep -v ... | cut ...\n```\n\n- *Note*: if you have hordes of RAM,\n  no shortage of cores, and large\n  files, then utilize pipelines `\u003ccmd\u003e\n  | ...` as much as possible because\n  the Linux Kernel will optimize things\n  in memory better. In more powerful\n  systems, many latency and performance\n  issues are not as relevant.\n\n- Use Shell built-ins\n  (see [Bash](https://www.gnu.org/software/bash/manual/html_node/Bash-Builtins.html))\n  and not binaries:\n\n```bash\n    echo     # not /usr/bin/echo\n    printf   # not /usr/bin/printf\n    [ ... ]  # not /usr/bin/test\n```\n\n## 3.2 SHELLS AND PERFORMANCE\n\nTODO\n\n## 3.3 MAJOR PERFORMANCE GAINS\n\n- In Bash, It is at least 60 times faster\n  to perform regular expression string\n  matching using the binary operator\n  [`=~`](https://www.gnu.org/software/bash/manual/bash.html#index-_005b_005b)\n  rather than to calling external\n  POSIX utilities\n  [`expr`](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/expr.html)\n  or\n  [`grep`](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/grep.html).\n\n  **NOTE**: In POSIX `sh`, like\n  [dash](https://tracker.debian.org/pkg/dash)\n  , calling utilities is\n  *extremely* fast. Compared to Bash's\n  [`[[]]`](https://www.gnu.org/software/bash/manual/bash.html#index-_005b_005b),\n  the `expr` in `dash` is only 5x\n  slower, which is negligible because\n  the time differences are measured in\n  mere few milliseconds.\n  See [code](./bin/t-string-match-regexp.sh)\n\n```bash\n    str=\"abcdef\"\n    re=\"b.*e\"\n\n    # Bash, Ksh\n    [[ $str =~ $re ]]\n\n    # In Bash, at least 60x slower\n    expr match \"$str\" \".*$re\"\n\n    # In Bash, at least 100x slower\n    echo \"$str\" | grep -E \"$re\"\n\n    # --------------------------------\n    # Different shells compared.\n    # --------------------------------\n\n    ./run.sh --shell dash,ksh93,bash t-string-match-regexp.sh\n\n    Run shell: dash\n    # t1     \u003cskip\u003e\n    # t2     real 0.010s  expr\n    # t3     real 0.010s  grep\n    Run shell: ksh93\n    # t1     real 0.001s [[ =~ ]]\n    # t2     real 0.139s expr\n    # t3     real 0.262s grep\n    Run shell: bash\n    # t1     real 0.003s [[ =~ ]]\n    # t2     real 0.200s expr\n    # t3     real 0.348s grep\n```\n\n- In Bash, it is about 50 times\n  faster to do string manipulation in\n  memory, than calling external\n  utilities. Seeing the measurements\n  just how expensive it is, reminds us\n  to utilize the possibilities of POSIX\n  `#`, `##`, `%` and `%%`\n  [parameter expansions](https://pubs.opengroup.org/onlinepubs/009604499/utilities/xcu_chap02.html#tag_02_06_02).\n  See more in\n  [Bash](https://www.gnu.org/software/bash/manual/bash.html#Shell-Parameter-Expansion).\n  See [code](./bin/t-string-file-path-components.sh).\n\n```bash\n    str=\"/tmp/filename.txt.gz\"\n\n    # (1) Almost instantaneous\n    # Delete up till first \".\"\n\n    ext=${str#*.}\n\n    # (2) In Bash, over 50x slower\n    #\n    # NOTE: identical in speed\n    # and execution to:\n    # cut -d \".\" -f 2,3 \u003c\u003c\u003c \"$str\"\n\n    ext=$(echo \"$str\" | cut -d \".\" -f 2,3)\n\n    # (3) In Bash, over 70x slower\n\n    ext=$(echo \"$str\" | sed 's/^[^.]\\+//')\n\n    # --------------------------------\n    # Different shells compared.\n    # --------------------------------\n\n    ./run.sh --shell dash,ksh93,bash t-string-file-path-components.sh\n\n    Run shell: dash\n    # t3aExt real 0.009s (1)\n    # t3cExt real 0.008s (2)\n    # t3eExt real 0.009s (3)\n    Run shell: ksh93\n    # t3aExt real 0.001s\n    # t3cExt real 0.193s\n    # t3eExt real 0.288s\n    Run shell: bash\n    # t3aExt real 0.004s\n    # t3cExt real 0.358s\n    # t3eExt real 0.431s\n\n```\n\n- In Bash, it is about 10 times faster\n  to read a file into memory as a\n  string and use\n  [pattern matching](https://www.gnu.org/software/bash/manual/bash.html#Pattern-Matching)\n  or regular expressions binary\n  operator\n  [`=~`](https://www.gnu.org/software/bash/manual/bash.html#index-_005b_005b)\n  on string. In-memory handling is much\n  more efficient than calling the\n  `grep` command in Bash on a file,\n  especially if multiple matches are\n  needed.\n  See [code](./bin/t-file-grep-vs-match-in-memory.sh).\n\n```bash\n    # Bash, Ksh\n    str=$(\u003c file)\n\n    if [[ $str =~ $regexp1 ]]; then\n        ...\n    elif [[ $str =~ $regexp2 ]]; then\n        ...\n    fi\n\n    # --------------------------------\n    # Different shells compared.\n    # --------------------------------\n\n    (1) read once + case..end\n    (2) loop do.. grep file ..done\n    (3) loop do.. case..end ..done\n\n    ./run.sh --shell dash,ksh93,bash t-file-grep-vs-match-in-memory.sh\n\n    Run shell: dash\n    # t1b    real 0.023s (1) once\n    # t2     real 0.018s (2) grep\n    # t3     real 0.021s (3) case\n    Run shell: ksh93\n    # t1b    real 0.333s (1) once\n    # t2     real 0.208s (2) grep\n    # t3     real 0.453s (3) case\n    Run shell: bash\n    # t1b    real 0.048s (1) once\n    # t2     real 0.277s (2) grep\n    # t3     real 0.415s (3) case\n```\n\n- In Bash, it is about 8 times faster,\n  to use\n  [nameref](https://www.gnu.org/software/bash/manual/bash.html#Shell-Parameters)\n  to return a value. In Bash, the\n  `ret=$(fn)` is inefficient to call\n  functions. On the other hand, in\n  POSIX `sh` shells, like\n  [dash](https://tracker.debian.org/pkg/dash),\n  there\n  is practically no overhead in using\n  `$(fn)`.\n  See [code](./bin/t-function-return-value-nameref.sh).\n\n```bash\n    # An exmaple only. Not needed in\n    # POSIX sh shells as ret=$(fn)\n    # is already fast.\n\n    fnNamerefPosix()\n    {\n        # NOTE: uses non-POSIX\n        # 'local' but it is widely\n        # supported in POSIX-compliant\n        # shells: dash, posh, mksh,\n        # ksh93 etc.\n\n        local retref=$1\n        shift\n        local arg=$1\n\n        eval \"$retref=\\$arg\"\n    }\n\n    fnNamerefBash()\n    {\n        local -n retref=$1\n        shift\n        local arg=$1\n\n        retref=$arg\n    }\n\n    # Return value returned to\n    # variable 'ret'\n\n    fnNamerefPosix ret \"arg\"\n    fnNamerefBash ret \"arg\"\n\n    # --------------------------------\n    # Different shells compared.\n    # --------------------------------\n\n    ./run.sh --shell dash,ksh93,bash t-function-return-value-nameref.sh\n\n    Run shell: dash\n    # t1     \u003cskip\u003e\n    # t2     real 0.006s fnNamerefPosix\n    # t3     real 0.005s ret=$(fn)\n\n    Run shell: ksh93\n    # t1     \u003cskip\u003e\n    # t2     real 0.004s fnNamerefPosix\n    # t3     real 0.005s ret=$(fn)\n\n    Run shell: bash\n    # t1     real 0.006s fnNamerefBash\n    # t2     real 0.006s fnNamerefPosix\n    # t3     real 0.094s ret=$(fn)\n```\n\n\n- In Bash, it is about 2 times faster\n  for line-by-line handling to read\n  the file into an array and then loop\n  through the array. The built-in\n  [`readarray`](https://www.gnu.org/software/bash/manual/html_node/Bash-Builtins.html#index-readarray)\n  is synonym for\n  [`mapfile`](https://www.gnu.org/software/bash/manual/bash.html#index-mapfile),\n  See [code](./bin/t-file-read-content-loop.sh).\n\n```bash\n    # Bash\n    readarray \u003c file\n\n    for line in \"${MAPFILE[@]}\"\n    do\n        ...\n    done\n\n    # POSIX. In bash, slower\n    while read -r line\n    do\n        ...\n    done \u003c file\n\n    # --------------------------------\n    # Different shells compared.\n    # --------------------------------\n\n    ./run.sh --shell dash,ksh93,bash t-file-read-content-loop.sh\n\n    Run shell: dash\n    # t1     \u003cskip\u003e\n    # t2     real 0.085  POSIX\n    Run shell: ksh93\n    # t1     \u003cskip\u003e\n    # t2     real 0.021  POSIX\n    Run shell: bash\n    # t1     real 0.045  readarray\n    # t2     real 0.108  POSIX\n\n```\n\n- In Bash, it is about 2 times faster to\n  prefilter with\n  [grep](https://tracker.debian.org/pkg/grep)\n  to process only\n  certain lines instead of reading the\n  whole file into a loop and then\n  selecting lines. The\n  [process substitution](https://www.gnu.org/software/bash/manual/html_node/Process-Substitution.html)\n  is more general because variables\n  persist after the loop. The\n  [dash](https://tracker.debian.org/pkg/dash)\n  is very fast compared to Bash.\n  See [code](./bin/t-file-read-match-lines-loop-vs-grep.sh).\n\n```bash\n    # Bash\n    while read -r ...\n    do\n        ...\n    done \u003c \u003c(grep \"$re\" file)\n\n    # POSIX\n    # Problem: while runs in\n    # a separate environment\n    grep \"$re\" file) |\n    while read -r ...\n    do\n        ...\n    done\n\n    # POSIX\n    # NOTE: extra calls\n    # required for tmpfile\n    grep \"$re\" file) \u003e tmpfile\n    while read -r ...\n    do\n        ...\n    done \u003c tmpfile\n    rm tmpfile\n\n    # Bash, Slowest,\n    # in-loop prefilter\n    while read -r line\n    do\n       [[ $line =~ $re ]] || continue\n       ...\n    done \u003c file\n\n    # --------------------------------\n    # Different shells compared.\n    # --------------------------------\n\n    ./run.sh --shell dash,ksh93,bash t-file-read-match-lines-loop-vs-grep.sh\n\n    Run shell: dash\n    # t1a    real 0.015s grep prefilter\n    # t2a    real 0.012s loop: case...esac\n    Run shell: ksh93\n    # t1a    real 2.940s\n    # t2a    real 1.504s\n    Run shell: bash\n    # t1a    real 4.567s\n    # t2a    real 10.88s\n\n```\n\n## 3.4 MODERATE PERFORMANCE GAINS\n\n- It is about 10 times faster to split\n  a string into an array using list\n  rather than using Bash here-string.\n  This is because\n[HERE STRING](https://www.gnu.org/software/bash/manual/bash.html#Here-Strings)\n  `\u003c\u003c\u003c` uses a pipe or temporary file,\n  whereas Bash list operates entirely\n  in memory. The pipe buffer behavor\n  was introduced in\n  [Bash 5.1 section c](https://github.com/bminor/bash/blob/master/CHANGES).\n  *Warning*: Please note that using the\n  `(list)` statement will undergo\n  pathname expansion so globbing\n  characters like `*`, `?`, etc. in\n  string would be a problem. The\n  pathname expansion can be disabled.\n  See [code](./bin/t-variable-array-split-string.sh).\n\n```bash\n    str=\"1:2:3\"\n\n    # Bash, Ksh. Fastest.\n    IFS=\":\" eval 'array=($str)'\n\n    fn() # Bash\n    {\n        local str=$1\n\n        # Make 'set' local\n        local -\n\n        # Disable pathname\n        # expansion\n        set -o noglob\n\n        local -a array\n\n        IFS=\":\" eval 'array=($str)'\n        ...\n    }\n\n    # Bash. Slower than 'eval'.\n    IFS=\":\" read -ra array \u003c\u003c\u003c \"$str\"\n\n    # In Linux, see what Bash uses\n    # for HERE STRING: pipe or\n    # temporary file\n    bash -c 'ls -l --dereference /proc/self/fd/0 \u003c\u003c\u003c hello'\n```\n\n- It is about 2 times faster to read\n  file into a string using Bash command\n  substitution\n  [`$(\u003c file)`](https://www.gnu.org/software/bash/manual/bash.html#Command-Substitution).\n  **NOTE**: In POSIX `sh`, like\n  `dash`, the `$(cat file)` is\n  extremely fast.\n  See [code](./bin/t-file-read-into-string.sh).\n\n```bash\n    # Bash\n    str=$(\u003c file)\n\n    # In Bash: 1.8x slower\n    # Read max 100 KiB\n    read -r -N $((100 * 1024)) str \u003c file\n\n    # In Bash: POSIX, 2.3x slower\n    str=$(cat file)\n\n    # --------------------------------\n    # Different shells compared.\n    # --------------------------------\n\n    ./run.sh --shell dash,ksh93,bash t-file-read-into-string.sh\n\n    Run shell: dash\n    # t1     \u003cskip\u003e\n    # t2     \u003cskip\u003e\n    # t3     real 0.013s $(cat ...)\n    Run shell: ksh93\n    # t1     real 0.088s $(\u003c ...)\n    # t2     real 0.095s read -N\n    # t3     real 0.267s $(cat ...)\n    Run shell: bash\n    # t1     real 0.139s $(\u003c ...)\n    # t2     real 0.254s read -N\n    # t3     real 0.312s $(cat ...)\n```\n\n## 3.5 MINOR PERFORMANCE GAINS\n\nAccording to the results, none of\nthese offer practical benefits.\n\n- The Bash\n  [brace expansion](https://www.gnu.org/software/bash/manual/bash.html#Brace-Expansion)\n  `{N..M}` might offer a\n  neglible advantage. However it may be\n  impractical because `N..M` cannot be\n  parameterized. Surprisingly, the\n  simple and elegant `$(seq N M)` is\n  fast, even though\n[command substitution](https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Command-Substitution)\n  uses a subshell. The last POSIX\n  `while` loop example was slightly\n  slower in all subsequent tests.\n  See [code](./bin/t-statement-arithmetic-for-loop.sh).\n\n```bash\n\n    N=1\n    M=100\n\n    # Bash\n    for i in {1..100}\n    do\n        ...\n    done\n\n    # POSIX, fast\n    for i in $(seq $N $M)\n    do\n        ...\n    done\n\n    # Bash, slow\n    for ((i=$N; i \u003c= $M; i++))\n    do\n        ...\n    done\n\n    # POSIX, slowest\n    i=$N\n    while [ \"$i\" -le \"$M\" ]\n    do\n        i=$((i + 1))\n    done\n```\n\n- One might think that choosing\n  optimized `grep` options would make a\n  difference. In practice, for typical\n  file sizes (below few Megabytes),\n  performance is nearly identical even\n  with the ignore case option included.\n  Nonetheless, there may be cases where\n  selecting `LANG=C`, using\n  `--fixed-strings`, and avoiding\n  `--ignore-case` might improve\n  performance, at least according to\n  StackOverflow discussions with large\n  files.\n  See [code](./bin/t-command-grep.sh).\n\n```bash\n    # The same performance. Regexp\n    # engine does not seem to be\n    # the bottleneck\n\n    LANG=C grep --fixed-strings ...\n    LANG=C grep --extended-regexp ...\n    LANG=C grep --perl-regexp ...\n    LANG=C grep --ignore-case ...\n```\n\n## 3.6 NO PERFORMANCE GAINS\n\nNone of these offer any advantages to speed up shell scripts.\n\n- The Bash-specific expression\n  [`[[]]`](https://www.gnu.org/software/bash/manual/bash.html#index-_005b_005b)\n  might offer a minuscule advantage but\n  only in loops of 10,000 iterations.\n  Unless the safeguards provided by\n  Bash `[[ ]]` are important, the POSIX\n  tests will do fine.\n  See [code](./bin/t-statement-if-test-posix-vs-bash.sh).\n\n```bash\n    [ \"$var\" = \"1\" ] # POSIX\n    [[ $var = 1 ]]   # Bash\n\n    [ ! \"$var\" ]     # POSIX\n    [[ ! $var ]]     # Bash\n    [ -z \"$var\" ]    # archaic\n```\n\n- There are no practical differences\n  between these. The POSIX\n  [arithmetic expansion](https://www.gnu.org/software/bash/manual/bash.html#Arithmetic-Expansion)\n  `$(())`\n  compound command will do fine. Note\n  that the null command\n  [`:`](https://www.gnu.org/software/bash/manual/html_node/Bourne-Shell-Builtins.html)\n  utilizes the command's side effect to\n  \"do nothing, but evaluate elements\"\n  and therefore may not be the most\n  readable option.\n  See [code](./bin/t-statement-arithmetic-increment.sh).\n\n```bash\n    i=$((i + 1))     # POSIX, preferred\n    : $((i++))       # POSIX, Uhm\n    : $((i = i + 1)) # POSIX, Uhm\n    ((i++))          # Bash, Ksh\n    let i++          # Bash, Ksh\n```\n\n- There is no performance\n  difference between a\n  Bash-specific expression\n  [`[[]]`](https://www.gnu.org/software/bash/manual/bash.html#index-_005b_005b)\n  for pattern matching compared to\n  POSIX `case..esac`. Interestingly\n  pattern matching is 4x slower under\n  [dash](https://tracker.debian.org/pkg/dash)\n  compared to Bash. However,\n  that means nothing because the time\n  differences are measured in minuscule\n  milliseconds (0.002s).\n  See [code](./bin/t-string-match-pattern.sh).\n\n```bash\n    string=\"abcdef\"\n    pattern=\"*cd*\"\n\n    # Bash\n    if [[ $string == $pattern ]]; then\n\t    ...\n    fi\n\n    # POSIX\n    case $string in\n        $pattern)\n            :    # Same as true\n            ;;\n        *)\n            false\n            ;;\n    esac\n\n    # --------------------------------\n    # Different shells compared.\n    # --------------------------------\n\n    ./run.sh --shell dash,ksh93,bash t-string-match-regexp.sh\n\n\tRun shell: dash\n\t# t1     \u003cskip\u003e\n\t# t2      real 0.011 POSIX\n\tRun shell: ksh93\n\t# t1     real 0.004  [[ == ]]\n\t# t2     real 0.002  POSIX\n\tRun shell: bash\n\t# t1     real 0.003  [[ == ]]\n\t# t2     real 0.002  POSIX\n```\n\n- There is no performance difference\n  between a regular while loop and a\n  [process substitution](https://www.gnu.org/software/bash/manual/html_node/Process-Substitution.html)\n  loop. However, the latter is more\n  general, as any variable set during\n  the loop will persist after *and*\n  there is no need to clean up\n  temporary files like in POSIX (1)\n  solution. The POSIX (1) loop is\n  marginally faster but the speed gain\n  is lost by the extra `rm` command.\n  See [code](./bin/t-command-output-vs-process-substitution.sh).\n\n```bash\n    # Bash, Ksh\n    while read -r ...\n    do\n        ...\n    done \u003c \u003c(command)\n\n    # POSIX (1)\n    # Same, but with\n    # temporary file\n    command \u003e file\n    while read -r ...\n    do\n        ...\n    done \u003c file\n    rm file\n\n    # POSIX (2)\n    # while is being run in\n    # separate environment\n    # due to pipe(|)\n    command |\n    while read -r ...\n    do\n        ...\n    done\n```\n\n- With `grep`, the use of\n  [GNU parallel](https://www.gnu.org/software/parallel/),\n  a `perl` program, makes things\n  notably slower for typical file\n  sizes. The idea of splitting a file\n  into chunks of lines and running the\n  search in parallel is intriguing, but\n  the overhead of starting Perl\n  interpreter with\n  `parallel` is orders of magnitude\n  more expensive compared to running\n  already optimized `grep` only once.\n  Usually the limiting factor when\n  grepping a file is the disk's I/O\n  speed. Otherwise, GNU `parallel` is\n  excellent for making full use of\n  multiple cores. Based on\n  StackOverflow discussions, if file\n  sizes are in the several hundreds of\n  megabytes or larger, GNU\n  [`parallel`](https://www.gnu.org/software/parallel/)\n  can help speed things up.\n  See [code](./bin/t-command-grep-parallel.sh).\n\n```bash\n    # Possibly add: --block -1\n    parallel --pipepart --arg-file \"$largefile\" grep \"$re\"\n```\n\n# 4.0 PORTABILITY\n\n## 4.1 LEGACY SHELL SCRIPTING\n\nIn typical cases, the legacy `sh`\n([Bourne Shell](https://en.wikipedia.org/wiki/Bourne_shell))\nis not a relevant target for shell\nscripting. The Linux and and modern\nUNIX operating systems have long\nprovided an `sh` that is\nPOSIX-compliant enough. Nowadays `sh`\nis usually a symbolic link to\n[dash](https://tracker.debian.org/pkg/dash)\n(on Linux since 2006),\n[ksh](https://tracker.debian.org/pkg/ksh93u+m)\n(on some BSDs), or it may point to\n[Bash](https://www.gnu.org/software/bash)\n(on macOS).\n\nExamples of pre-2000 shell scripting\npractises:\n\n```bash\n    if [ x$a = y ] ...\n\n    # Variable lenght is non-zero\n    if [ -n \"$a\" ] ...\n\n    # Variable lenght is zero\n    if [ -z \"$a\" ] ...\n\n    # Deprecated in next POSIX\n    # version. Operands are\n    # not portable.\n    # -o (OR)\n    # -a (AND)\n\n    if [ \"$a\" = \"y\" -o \"$b\" = \"y\" ] ...\n\n    # POSIX allows leading\n    # opening \"(\" paren\n    case abc in\n        (a*) true\n             ;;\n        (*)  false\n             ;;\n    esac\n\n```\n\nModern equivalents:\n\n```bash\n\n    # Equality\n    if [ \"$a\" = \"y\" ] ..\n\n    # Variable has something\n    if [ \"$a\" ] ...\n\n    # Variable is empty\n    if [ ! \"$a\" ] ...\n\n    if [ \"$a\" = \"y\" ] || [ \"$b\" = \"y\" ] ...\n\n    # Without leading \"(\" paren\n    case abc in\n         a*) :  # \"true\"\n             ;;\n         *)  false\n             ;;\n    esac\n\n```\n\n## 4.2 REQUIREMENTS AND SHELL SCRIPTS\n\nWriting shell scripts inherently\ninvolves considering several factors.\n\n- *Personal scripts.* When writing\n  scripts for personal or\n  administrative tasks, the choice of\n  shell is unimportant. On Linux, the\n  obvious choice is Bash. On BSD\n  systems, it would be Ksh. On macOS,\n  Zsh might be handy.\n\n- *Portable scripts.* If you intend to\n  use the scripts across some operating\n  systems — from Linux to Windows\n  ([Git Bash](https://gitforwindows.org/),\n  [Cygwin](https://cygwin.com),\n  [MSYS2](https://www.msys2.org) [\\*][\\*\\*]) —\n  the obvious choice would be Bash.\n  Between macOS and Linux, writing\n  scripts in Bash is generally more\n  portable than writing them in Zsh\n  because Linux doesn't have Zsh\n  installed by default. With macOS\n  however, the choice of Bash is a bit\n  more involved (see next).\n\n- *POSIX-compliant scripts*. If you\n  intend to use the scripts across a\n  variety of operating systems — from\n  Linux, BSD, and macOS to various\n  Windows Linux-like environments — the\n  issues become quite complex. You are\n  probably better off writing `sh`\n  POSIX-compliant scripts and testing\n  them with\n  [dash](https://tracker.debian.org/pkg/dash),\n  since relying on Bash can lead to\n  unexpected issues — different systems\n  have different Bash versions, and\n  there’s no guarantee that a script\n  written on Linux will run without\n  problems on older Bash versions, such\n  as the outdated 3.2 version in\n  `/bin/bash` on macOS. Requiring users\n  to install a newer version on macOS\n  is not trivial because `/bin/bash` is\n  not replaceable.\n\n[\\*] \"Git Bash\" is available with the\npopular native Windows installation of\n[Git for Windows](https://git-scm.com/downloads/win).\nUnder the hood, the installation is based on\nMSYS2, which in turn is based on\nCygwin. The common denominator of\nall native Windows Linux-like\nenvironments is the\n[Cygwin](https://cygwin.com)\nbase which, in all\npractical terms, provides the usual\ncommand-line utilities,\nincluding Bash. For curious readers,\nWindows software\n[MobaXterm](https://mobaxterm.mobatek.net),\noffers X server, terminals\nand other connectivity features, but also\nsomes with Cygwin-based\nBash shell with its own\n[Debian-style](https://www.debian.org/doc/manuals/debian-faq/pkgtools.en.html)\n`apt` package manager which allows installing\nadditional Linux utilities.\n\n[\\*\\*] In Windows, there is also\nthe Windows Subsystem for Linux\n([WSL](https://learn.microsoft.com/en-us/windows/wsl/)),\nwhere you can install (See `wsl --list --onlline`)\nLinux distributions like\n[Debian](https://en.wikipedia.org/wiki/Debian),\n[Ubuntu](https://en.wikipedia.org/wiki/Ubuntu),\n[OpenSUSE](https://en.wikipedia.org/wiki/OpenSUSE) and\n[Oracle Linux](https://en.wikipedia.org/wiki/Oracle_Linux).\nBash is the obvious choice for shell\nscripts in this environment.\n\n## 4.3 WRITING POSIX COMPLIANT SHELL SCRIPS\n\nAs this document is more focused on\nLinux, macOS, and BSD compatibility,\nand less on legacy UNIX operating\nsystems, for all practical purposes,\nthere is no need to attempt to write\n*pure* POSIX shell scripts. Stricter\nmeasures are required only if you\ntarget legacy UNIX operating systems\nwhose `sh` may not have changed in 30\nyears. your best guide probably is\nthe wealth of knowledge collected by\nthe GNU autoconf project; see\n[\"11 Portable Shell Programming\"](https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.72/autoconf.html#Portable-Shell).\nFor more discussion see\n[4.6 MISCELLANEUS NOTES](#46-miscellaneus-notes).\n\nLet's first consider the typical `sh`\nshells in order of their strictness to\nPOSIX:\n\n- [posh](https://tracker.debian.org/pkg/posh).\n  Minimal `sh`, Policy-compliant\n  Ordinary SHell. Very close to POSIX.\n  Stricter than\n  [dash](https://tracker.debian.org/pkg/dash).\n  Supports\n  `local` keyword to define local\n  variables in functions. The keyword\n  is not defined in POSIX.\n\n- [dash](https://tracker.debian.org/pkg/dash).\n  Minimal `sh`, Debian Almquish Shell.\n  Close to POSIX. Supports `local`\n  keyword. The shell aims to meet the\n  requirements of the Debian Linux\n  distribution.\n- [Busybox ash](https://www.busybox.net)\n  is based on\n  [dash](https://tracker.debian.org/pkg/dash)\n  with some more\n  features added. Supports `local`\n  keyword. See ServerFault\n  [\"What's the Busybox default shell?\"](https://serverfault.com/questions/241959/whats-the-busybox-default-shell)\n\nLet's also consider what the `/bin/sh`\nmight be in different Operating\nSystems. For more about the history of\nthe `sh` shell, see the well-rounded\ndiscussion on StackExchange.\n[What does it mean to be \"sh compatible\"?](https://unix.stackexchange.com/q/145522)\n\n\u003c!-- \u003ccontactme2016@tangentsoft.com\u003e --\u003e\n[Picture](https://tangentsoft.com/misc/unix-shells.svg)\n\"Bourne Family Shells\" by\n[tangentsoft.com](tangentsoft.com)\n\u003cpicture\u003e\n    \u003csource srcset=\"https://tangentsoft.com/misc/unix-shells-large.png\"\u003e\n    \u003cimg src=\"pic/unix-shells.svg\" alt=\"Bourne Family Shells\" style=\"width:auto;\"\u003e\n\u003c/picture\u003e\n\n- On Linux, most distributions already\n  use, or are moving towards using,\n  `sh` as a symlink to\n  [dash](https://tracker.debian.org/pkg/dash).\n  Older Linux versions (Red Hat,\n  Fedora, CentOS) used to have `sh` to\n  be a symlink to `bash`.\n\n- On the most conservative NetBSD,\n  it is `ash`, the old\n  [Almquist shell](https://en.wikipedia.org/wiki/Almquist_shell).\n  On FreeBSD, `sh` is also `ash`.\n  On\n  [OpenBSD, sh](https://man.netbsd.org/sh.1) is\n  [ksh93](https://tracker.debian.org/pkg/ksh93u+m)\n  from the\n  [Ksh family](https://en.wikipedia.org/wiki/KornShell).\n- On many commercial, conservative\n  UNIX systems `sh` is nowadays quite\n  capable\n  [ksh93](https://tracker.debian.org/pkg/ksh93u+m).\n- On macOS, `sh` points to `bash\n  --posix`, where the Bash version is\n  indefinitely stuck at version 3.2.x\n  due to Apple avoiding the GPL-3\n  license in later Bash versions. If\n  you write `/bin/sh` scripts in macOS,\n  it is good idea to check them for\n  portability with:\n\n```bash\n    # Check better /bin/sh\n    # compliance\n\n    dash -nx script.sh\n    posh -nx script.sh\n```\n\nIn practical terms, if you plan to aim\nfor POSIX-compliant shell scripts, the\nbest shells for testing would be\n[posh](https://tracker.debian.org/pkg/posh)\nand\n[dash](https://tracker.debian.org/pkg/dash).\nYou can also extend testing\nwith BSD Ksh shells and other shells.\nSee\n[FURTHER READING](#further-reading)\nfor external utilities to check and\nimprove shell scripts even more.\n\n    # Save in shell startup file\n    # like ~/.bashrc\n\n    shelltest()\n    {\n        local script shell\n\n        for script # Implicit \"$@\"\n        do\n            for shell in \\\n                posh \\\n                dash \\\n                \"busybox ash\" \\\n                mksh \\\n                ksh \\\n                bash \\\n                zsh\n            do\n                if command -v \"$shell\" \u003e /dev/null; then\n                    echo \"-- shell: $shell\"\n                    $shell -nx \"$script\"\n                fi\n            done\n        done\n    }\n\n    # Use is like:\n    shelltest script.sh\n\n    # External utility\n    shellcheck script.sh\n\n    # External utility\n    checkbashisms script.sh\n\n## 4.4 SHEBANG LINE IN SCRIPTS\n\nNote that POSIX does not define the\n[shebang](https://en.wikipedia.org/wiki/Shebang_(Unix))\n— the traditional first line that\nindicates which interpreter to use. See\nPOSIX C language's section \"exec family\nof functions\" and\n[RATIONALE](https://pubs.opengroup.org/onlinepubs/9799919799/functions/exec.html#:~:text=RATIONALE)\n\n\u003e (...) Another way that some\n\u003e historical implementations handle\n\u003e shell scripts is by recognizing the\n\u003e first two bytes of the file as the\n\u003e character string \"#!\" and using the\n\u003e remainder of the first line of the\n\u003e file as the name of the command\n\u003e interpreter to execute.\n\nThe first bytes of a script typically\ncontain two special ASCII codes, a\nspecial comment `#!` if you wish, which\nis read by the kernel. Note that this\nis a de facto convention, universally\nsupported even though it is not defined\nby POSIX.\n\n    #! \u003cinterpreter\u003e [word]\n    #\n    # 1. whitespace is allowed in\n    #    \"#!\" for readability.\n    #\n    # 2. The \u003cinterpreter\u003e must be\n    #     full path name. Not like:\n    #\n    #    #! sh\n    #\n    # 3. ONE word can be added\n    #    after the \u003cinterpreter\u003e.\n    #    Any more than that may not\n    #    be portable accross Linux\n    #    and some BSD Kernels.\n    #\n    #    #! /bin/sh -eu\n    #    #! /usr/bin/awk -f\n    #    #! /usr/bin/env bash\n    #    #! /usr/bin/env python3\n\n### 4.4.1 About Bash and Shebang\n\nNote that on macOS, `/bin/bash` is\nhard-coded to Bash version 3.2.57 where\nin 2025 lastest Bash is\n[5.2](https://tracker.debian.org/pkg/bash).\nYou cannot uninstall it, even with root\naccess, without disabling System\nIntegrity Protection. If you install a\nnewer Bash version with `brew install\nbash`, it will be located in\n`/usr/local/bin/bash`.\n\nOn macOS, to use the latest Bash, the\nuser must arrange `/usr/local/bin`\nfirst in\n[PATH](https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap08.html#:~:text=This%20variable%20shall%20represent%20the%20sequence%20of%20path%20prefixes).\nIf the script starts with `#!\n/bin/bash`, the user cannot arrange it\nto run under different Bash version\nwithout modifying the script itself, or\nafter modifying `PATH`, run it\ninconveniently with `bash \u003cscript\u003e`.\n\n    ... portable\n\n    #! /usr/bin/env bash\n\n    ... traditional\n\n    #! /bin/bash\n\n### 4.4.2 About Python and Shebang\n\nThere was a disruptive change from\nPython 2.x to Python 3.x in 2008. The\nolder programs did not run without\nchanges with the new version. In Python\nprograms, the shebang should specify\nthe Python version explicitly, either\nwith `python` (2.x) or `python3`.\n\n    ... The de facto interpreters\n\n    #! /usr/bin/python\n    #! /usr/bin/python3\n\n    .... not supported\n\n    #! /usr/bin/python2\n    #! /usr/bin/python3.13.2\n\nBut this is not all. Python is one\nof those languages which might require\nmultiple virtual environments based on\nprojects. It is typical to manage these\nenvironments with tools like\n[uv](https://docs.astral.sh/uv/pip/environments/)\nor older\n[virtualenv](https://virtualenv.pypa.io),\n[pyenv](https://github.com/pyenv/pyenv)\netc. For even better portability, the\nfollowing would allow user to use his\nactive Python environment:\n\n    ... portable\n\n    #! /usr/bin/env python3\n\nThe fine print here is that\n[`env`](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/env.html)\nis a standard POSIX utility, but its\npath is not mandated by POSIX. However,\nin 99.9% of cases, the de facto\nportable location is `/usr/bin/env`.\n\n## 4.5 PORTABILITY OF UTILITIES\n\nIn the end, the actual implementation\nof the shell you use (dash, bash,\nksh...) is less important than what\nutilities you use and how you use them.\n\nIt's not just about choosing to write\nin POSIX `sh`; the utilities\ncalled from the script also has to be\nconsidered. Those of `echo`, `cut`,\n`tail` make big part of of the scripts.\nIf you want to ensure portability,\ncheck options defined in\n[POSIX](https://pubs.opengroup.org/onlinepubs/9799919799/).\nSee top left menu \"Shell \u0026 Utilities\"\nfollowed by bottom left menu\n[\"4. Utilities\"](https://pubs.opengroup.org/onlinepubs/9799919799/idx/utilities.html)\n\nNotable observations:\n\n- Use POSIX\n  [`command -v`](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/command.html)\n  to check if command exists.\n  Note that POSIX also defines\n  [`type`](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/type.html),\n  as in `type \u003ccommand\u003e`\n  without any options. POSIX also\n  defines utility\n  [`hash`](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/type.html),\n  as in `hash \u003ccommand\u003e`. Problem with\n  `type` is that the semantics, return\n  codes, support or output are not\n  necessarily uniform. Problem with\n  `hash` are similar. Neither `type`\n  nor `hash` is supported by\n  [posh](https://tracker.debian.org/pkg/posh);\n  see table\n  [RESULTS-PORTABILITY](./doc/RESULTS-PORTABILITY.txt).\n  Note: The `which \u003ccommand\u003e` is\n  neither in POSIX nor portable. For\n  more information about `which`, see\n  shellcheck\n  [SC2230](https://github.com/koalaman/shellcheck/wiki/SC2230),\n  BashFAQ [081](https://mywiki.wooledge.org/BashFAQ/081),\n  StackOverflow discussion\n  [\"How can I check if a program exists from a Bash script?\"](https://stackoverflow.com/q/592620),\n  and Debian project plan about\n  deprecating the command in LWN\n  article\n  [\"Debian's which hunt\"](https://lwn.net/Articles/874049/).\n\n```bash\n    REQUIRE=\"sqlite curl\"\n\n    RequireFeatures ()\n    {\n        local cmd\n\n        for cmd # Implicit \"$@\"\n        do\n            if ! command -v \"$cmd\" \u003e /dev/null; then\n                echo \"ERROR: not in PATH: $cmd\" \u003e\u00262\n                return 1\n            fi\n        done\n    }\n\n    # Before program starts\n    RequireFeatures $REQUIRE || exit $?\n    ...\n```\n\n- Use plain\n  [`echo`](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/echo.html)\n  without any options. Use\n  [`printf`](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/printf.html)\n  when more functionality is needed.\n  Relying solely on `printf` may not be\n  ideal. In POSIX-compliant `sh` shells,\n  `printf` is not always a built-in\n  command (e.g., in\n  [posh](https://tracker.debian.org/pkg/posh)\n  or\n  [mksh](https://tracker.debian.org/pkg/mksh))\n  which can lead to performance overhead\n  due to the need to invoke an external\n  process.\n\n```bash\n    # POSIX\n    echo \"line\"            # (1)\n    echo \"line\"\n    printf \"no newline\"    # (2)\n\n    # Not POSIX\n    echo -e \"line\\nline\"   # (1)\n    echo -n \"no newline\"   # (2)\n\n```\n\n- Use [`grep -E`](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/grep.html).\n  In 2001 POSIX removed `egrep`.\n\n- [`read`](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/read.html).\n  POSIX requires a VARIABLE, so always\n  supply one. In Bash, the command would\n  default to variable `REPLY` if omitted.\n  You should also always use option `-r`\n  which is eplained in\n  shellcheck [SC2162](https://github.com/koalaman/shellcheck/wiki/SC2162),\n  BashFAQ [001](https://mywiki.wooledge.org/BashFAQ/001),\n  POSIX [IFS](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html#:~:text=A%20string%20treated%20as%20a%20list%20of%20characters)\n  and\n  BashWiki [IFS](https://mywiki.wooledge.org/IFS).\n  See in depth details\n  how the\n  [`read`](https://www.gnu.org/software/bash/manual/html_node/Bash-Builtins.html#index-read)\n  command does not reads characters and\n  not lines in StackExchange discussion\n  [Understanding \"IFS= read -r line\"](https://unix.stackexchange.com/a/209184).\n\n\n```bash\n   # POSIX\n   REPLY=$(cat file)\n\n   # Bash, Ksh\n   # Read max 100 KiB to $REPLY\n   read -rN $((100 * 1024)) \u003c file\n\n   case $REPLY in\n        *pattern*)\n            # match\n            ;;\n   esac\n```\n\n- Use [`shift N`](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/shift.html)\n  always with shell special parameter\n  [`$#`](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html#tag_18_05_02)\n\n```bash\n    set -- 1\n\n    # POSIX\n    # shift all positional args\n    shift $#\n\n    # Any greater number terminates\n    # the whole program in:\n    # dash, posh, mksh, ksh93 etc.\n    shift 2\n```\n\n### 4.5.1 Case Study: sed\n\nAs a case study, the Linux GNU `sed(1)`\n and its options differ or\nare incompatible. The Linux GNU\n[`sed`](https://www.gnu.org/software/sed/)\n`--in-place` option for replacing file\ncontent cannot be used in macOS and\nBSD. Additionally, in macOS and BSD,\nyou will find GNU programs under a\n`g`-prefix, such as `gsed(1)`, etc. See\nStackOverflow\n[\"sed command with -i option failing on Mac, but works on Linux\"](https://stackoverflow.com/q/4247068). For more\ndiscussions about the topic, see\nStackOverflow [1](https://stackoverflow.com/q/48712373),\nStackOverflow [2](https://stackoverflow.com/q/16745988),\nStackOverflow [3](https://stackoverflow.com/q/7573368).\n\n    # Linux (works)\n    #\n    # GNU sed(1). Replace 'this'\n    # with 'that'\n\n    sed -i 's/this/that/g' file\n\n    # macOS (does not work)\n    #\n    # This does not work. The '-i'\n    # option has different syntax\n    # and semantics. There is no\n    # workaround to make the '-i'\n    # option work across all\n    # operating systems.\n\n    sed -i 's/this/that/g' file\n\n    # Maybe portable\n    #\n    # In many cases Perl might be\n    # available although it is not\n    # part of the POSIX utilities.\n\n    perl -i -pe 's/this/that/g' file\n\n    # Portable\n    #\n    # Avoid -i option.\n\n    tmp=$(mktemp)\n    sed 's/this/that/g' file \u003e \"$tmp\" \u0026\u0026\n    mv \"$tmp\" file \u0026\u0026\n    rm -f \"$tmp\"\n\n### 4.5.2 Case Study: awk\n\nPOSIX\n[`awk`](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/awk.html),\ndoes not support the `-v` option to\ndefine variables. You can use\nassignments after the program instead.\n\n    # POSIX\n    awk '{print var}' var=1 file\n\n    # GNU awk\n    awk -v var=1 '{print var}' file\n\nHowever, don't forget that such\nassignments are not evaluated until\nthey are encountered, that is, after\nany `BEGIN` action. To use awk for\noperands without any files:\n\n    # POSIX\n    var=1 awk 'BEGIN {print ENVIRON[\"var\"] + 1}' \u003c /dev/null\n\n    # GNU awk\n    awk -v var=1 'BEGIN {print var + 1; exit}'\n\n## 4.6 MISCELLANEOUS NOTES\n\n- The shell's null command\n  [`:`](https://www.gnu.org/software/bash/manual/html_node/Bourne-Shell-Builtins.html)\n  might be slightly preferrable than\n  utlity\n  [`true`](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/true.html)\n  according to\n  GNU autoconf's manual\n  [\"11.14 Limitations of Shell Builtins\"](https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.72/autoconf.html#Limitations-of-Builtins)\n  which states that `:` might not be\n  always builtin and \"(...) the\n  portable shell community tends to\n  prefer using :\".\n\n```bash\n    while :\n    do\n        break\n    done\n\n    # Create an empty file\n    : \u003e file\n\n```\n\n- Prefer POSIX\n  [`$(cmd)`](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html#tag_18_06_03)\n\n  command substitution instead of\n  leagacy POSIX backtics as in \\`cmd\\`.\n  For more information, see\n  BashFaq [098](https://mywiki.wooledge.org/BashFAQ/082)\n  and shellcheck\n  [SC2006](https://github.com/koalaman/shellcheck/wiki/SC2006).\n  For 20 years all the modern `sh`\n  shells have supported `$()`.\n  Including UNIX like AIX, HP-UX and\n  conservative Oracle Solaris 10 (2005)\n  whose support ends in\n[2026](https://www.theregister.com/2024/01/29/oracle_extends_solaris_support/#:~:text=During%202023%2C%20Oracle%20added%20another,2027%20instead%20of%20during%202024)\n  (see Solaris\n  [version history](https://www.liquisearch.com/solaris_operating_system/version_history)).\n\n```bash\n        # Easily nested\n        lastdir=$(basename $(pwd))\n\n        # Readabilty problems\n        lastdir=`basename \\`pwd\\``\n```\n\n\u003c!--\nTODO:\nhttps://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.72/autoconf.html#Shellology\n--\u003e\n\n# 5.0 RANDOM NOTES\n\nSee the Bash manual how to use\n[`time`](https://www.gnu.org/software/bash/manual/bash.html#Pipeline)\nreserved word with\n[TIMEFORMAT](https://www.gnu.org/software/bash/manual/bash.html#index-TIMEFORMAT)\nvariable to display results in\ndifferent formats. The use of time as a\nreserved word permits the timing of\nshell builtins, shell functions, and\npipelines.\n\n    TIMEFORMAT='real: %R'  # '%R %U %S'\n\nYou could also drop kernel cache before\ntesting:\n\n    echo 3 \u003e /proc/sys/vm/drop_caches\n\n# 6.0 FURTHER READING\n\n## 6.1 BASH PROGRAMMING\n\n- Bash\n  [Manual](https://www.gnu.org/software/bash/manual/bash.html)\n- Greg's Bash Wiki and FAQ\n  https://mywiki.wooledge.org/BashGuide\n- List of which features were added to\n  specific releases of Bash\n  https://mywiki.wooledge.org/BashFAQ/061\n\n## 6.2 PORTABILITY AND UTILITIES\n\n-  GNU autoconf's manual section\n  [\"11 Portable Shell Programming\"](https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.72/autoconf.html#Portable-Shell)\n  **Note:** This presents information\n  intended to overcome operating system\n  portability issues dating back to the\n  1970s. Consider some tips with a grain\n  of salt, given the capabilities of more\n  modern POSIX-compliant shells.\n- For cross platform operating system\n  detection, see useful files to check:\n  http://linuxmafia.com/faq/Admin/release-files.html\n- `shellcheck` (Haskell)\n  can help to improve and write portable\n  POSIX scripts. It can statically Lint\n  scripts for potential mistakes. There\n  is also a web interface where you can\n  upload the script at\n  https://www.shellcheck.net. In Debian,\n  see package \"shellcheck\". The manual\n  page is at\nhttps://manpages.debian.org/testing/shellcheck/shellcheck.1.en.html\n- `checkbashisms` can help to improve and\n  write portable POSIX scripts. In\n  Debian, the command is available in\n  package \"devscripts\". The manual page\n  is at\n  https://manpages.debian.org/testing/devscripts/checkbashisms.1.en.html\n\n## 6.3 STANDARDS\n\nRelevant POSIX links from 2000 onward:\n\n- https://en.wikipedia.org/wiki/POSIX\n- POSIX.1-2024\n  IEEE Std 1003.1-2024\n  https://pubs.opengroup.org/onlinepubs/9799919799\n- POSIX.1-2018\n  IEEE Std 1003.1-2018\n  https://pubs.opengroup.org/onlinepubs/9699919799\n  https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/\n- POSIX.1-2008\n  IEEE Std 1003.1-2008\n  https://pubs.opengroup.org/onlinepubs/9699919799.2008edition/\n- POSIX.1-2004 (2001-2004)\n  IEEE Std 1003.1-2004\n  https://pubs.opengroup.org/onlinepubs/009695399\n\nRelevant UNIX standardization links.\n[Single UNIX Specification](https://en.wikipedia.org/wiki/Single_UNIX_Specification)\n(SUSv4) documents are derived from POSIX\nstandards. For an operating system to\nbecome UNIX certified, it must meet all\nspecified requirements—a process that is\nboth costly and arduous. The only\n\"Linux-based\" system that has undergone\nfull certification is Apple's\n[macOS](https://en.wikipedia.org/wiki/MacOS)\n10.5 Leopard in 2007. Read the story\nshared by Apple’s project lead,\nTerry Lambert, in a Quora's discussion\nforum\n[\"What goes into making an OS to be Unix compliant certified?\"](https://www.quora.com/What-goes-into-making-an-OS-to-be-Unix-compliant-certified)\n\n- The Single UNIX Specification,\n  Version 4\n  https://unix.org/version4/\n- See discussion at StackExchnage about\n  [\"Difference between POSIX, Single UNIX Specification, and Open Group Base Specifications?\"](https://unix.stackexchange.com/q/14368).\n\n## 6.4 MISCELLANEOUS LINKS\n\n- A comprehensive history of `ash`.\n  \"Ash (Almquist Shell) Variants\" by\n  Sven Mascheck\n  https://www.in-ulm.de/~mascheck/various/ash/\n- Late Jörg Shillings's\n  [schilitools](https://codeberg.org/schilytools/schilytools.git)\n  contains `pbosh` shell that can be used\n  for POSIX-sh-like testing.\n  See discussion of preserving\n  the project and some history at\n  [Reddit](https://www.reddit.com/r/linux/comments/w9vrpx/continued_development_of_j%C3%B6rg_schillings_tools/?rdt=56420).\n- Super simple `s` command interpreter\n  to write shell-like scripts (security\n  oriented):\n  https://github.com/rain-1/s\n\n# COPYRIGHT\n\nCopyright (C) 2024-2025 Jari Aalto\n\n# LICENSE\n\nThese programs are free software; you can redistribute it and/or modify\nthem under the terms of the GNU General Public License as published by\nthe Free Software Foundation; either version 2 of the License, or\n(at your option) any later version.\n\nThese programs are distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with these programs. If not, see \u003chttp://www.gnu.org/licenses/\u003e.\n\nLicense-tag: GPL-2.0-or-later\u003cbr\u003e\nSee https://spdx.org/licenses\n\nKeywords: shell, sh, POSIX, bash,\nksh, ksh93, programming,\noptimizing, performance, profiling,\nportability\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaalto%2Fproject--shell-script-performance-and-portability","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjaalto%2Fproject--shell-script-performance-and-portability","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaalto%2Fproject--shell-script-performance-and-portability/lists"}