Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/natelandau/shell-scripting-templates

Shell scripting utility functions and a bash script boilerplate template
https://github.com/natelandau/shell-scripting-templates

Last synced: 6 days ago
JSON representation

Shell scripting utility functions and a bash script boilerplate template

Awesome Lists containing this project

README 

        
# Shell Scripting Templates and Utilities



A collection of BASH utility functions and script templates used to ease the creation of portable and hardened BASH scripts with sane defaults.



## Usage



There are **two Script Templates** located in the root level of this repository. The usage of these templates is described in detail below.



BASH **Utility functions** are located within the `utilities/` folder.



Complex `sed` find/replace operations are supported with the files located in `sedfiles/`. Read [the usage instructions](sedfiles/README.md).



**Automated testing** is provided using [BATS](https://github.com/bats-core/bats-core). All tests are in the `tests/` repo. A git pre-commit hook provides automated testing is located in the `.hooks/` directory. Read about [how to install the hook](.hooks/README.md).



## Bash Script Templates Usage



To create a new script, copy one of the script templates to a new file and make it executable `chmod 755 [newscript].sh`. Place your custom script logic within the `_mainScript_` function at the top of the script.



### Script Templates



There are two templates located at the root level of this repository.



-   **`template.sh`** - A lean template which attempts to source all the utility functions from this repository. You will need to update the path to the utilities folder sent to `_sourceUtilities_` at the bottom of the script. This template will not function correctly if the utilities are not found.

-   **`template_standalone.sh`** - For portability, the standalone template does not assume that this repository is available. Copy and paste the individual utility functions under the `### Custom utility functions` line.



### Code Organization



The script templates are roughly split into three sections:



-   TOP: Write the main logic of your script within the `_mainScript_` function. It is placed at the top of the file for easy access and editing. However, it is invoked at the end of the script after options are parsed and functions are sourced.

-   MIDDLE: Functions and default variable settings are located just below `_mainScript_`.

-   BOTTOM: Script initialization (BASH options, traps, call to `_mainScript_`, etc.) is at the bottom of the template



### Default Options



These default options and global variables are included in the templates and used throughout the utility functions. CLI flags to set/unset them are:



-   **`-h, --help`**: Prints the contents of the `_usage_` function. Edit the text in that function to provide help

-   **`--logfile [FILE]`** Full PATH to logfile. (Default is `${HOME}/logs/$(basename "$0").log`)

-   **`loglevel [LEVEL]`**: Log level of the script. One of: `FATAL`, `ERROR`, `WARN`, `INFO`, `DEBUG`, `ALL`, `OFF` (Default is '`ERROR`')

-   **`-n, --dryrun`**: Dryrun, sets `$DRYRUN` to `true` allowing you to write functions that will work non-destructively using the `_execute_` function

-   **`-q, --quiet`**: Runs in quiet mode, suppressing all output to stdout. Will still write to log files

-   **`-v, --verbose`**: Sets `$VERBOSE` to `true` and prints all debug messages to stdout

-   **`--force`**: If using the `_seekConfirmation_` utility function, this skips all user interaction. Implied `Yes` to all confirmations.



You can add custom script options and flags to the `_parseOptions_` function.



### Script Initialization



The bottom of the script template file contains a block which initializes the script. Comment, uncomment, or change the settings here for your needs



```bash

trap '_trapCleanup_ ${LINENO} ${BASH_LINENO} "${BASH_COMMAND}" "${FUNCNAME[*]}" "${0}" "${BASH_SOURCE[0]}"' EXIT INT TERM SIGINT SIGQUIT SIGTERM ERR



# Trap errors in subshells and functions

set -o errtrace



# Exit on error. Append '||true' if you expect an error

set -o errexit



# Use last non-zero exit code in a pipeline

set -o pipefail



# Confirm we have BASH greater than v4

[ "${BASH_VERSINFO:-0}" -ge 4 ] || {

    printf "%s\n" "ERROR: BASH_VERSINFO is '${BASH_VERSINFO:-0}'.  This script requires BASH v4 or greater."

    exit 1

}



# Make `for f in *.txt` work when `*.txt` matches zero files

shopt -s nullglob globstar



# Set IFS to preferred implementation

IFS=$' \n\t'



# Run in debug mode

# set -o xtrace



# Source utility functions

_sourceUtilities_



# Initialize color constants

_setColors_



# Disallow expansion of unset variables

set -o nounset



# Force arguments when invoking the script

# [[ $# -eq 0 ]] && _parseOptions_ "-h"



# Parse arguments passed to script

_parseOptions_ "$@"



# Create a temp directory '$TMP_DIR'

# _makeTempDir_ "$(basename "$0")"



# Acquire script lock

# _acquireScriptLock_



# Source GNU utilities for use on MacOS

# _useGNUUtils_



# Run the main logic script

_mainScript_



# Exit cleanly

_safeExit_

```



# Utility Functions



The files within `utilities/` contain BASH functions which can be used in your scripts. Each included function includes detailed usage information. Read the inline comments within the code for detailed usage instructions.



## Including Utility Functions



Within the `utilities` folder are many BASH functions meant to ease development of more complicated scripts. These can be included in the template in two ways.



#### 1. Copy and paste into template_standalone.sh



You can copy any complete function from the Utilities and place it into your script. Copy it beneath the `### Custom utility functions` line. Scripts created this way are fully portable among systems



#### 2. Source all the utility files by using template.sh



`template.sh` contains a function to source all the utility files into the script. **IMPORTANT:** You will need to update the paths within the `_sourceUtilities_` function to ensure your script can find this repository.



## alerts.bash



-   **`_columns_`** Prints a two column output from a key/value pair

-   -**`_printFuncStack_`** Prints the function stack in use. Used for debugging, and error reporting

-   **`_alert_`** Performs alerting functions including writing to a log file and printing to screen

-   **`_centerOutput_`** Prints text in the center of the terminal window

-   **`_setColors_`** Sets color constants for alerting (**Note:** Colors default to a dark theme.)



Basic alerting, logging, and setting color functions (included in `scriptTemplate.sh` by default). Print messages to stdout and to a user specified logfile using the following functions.



```bash

debug "some text"     # Printed only when in verbose (-v) mode

info "some text"      # Basic informational messages

notice "some text"    # Messages which should be read. Brighter than 'info'

warning "some text"   # Non-critical warnings

error "some text"     # Prints errors and the function stack but does not stop the script.

fatal "some text"     # Fatal errors. Exits the script

success "some text"   # Prints a success message

header "some text"    # Prints a header element

dryrun "some text"    # Prints commands that would be run if not in dry run (-n) mode

```



The following global variables must be set for the alert functions to work



-   **`$DEBUG`** - If `true`, prints `debug` level alerts to stdout. (Default: `false`)

-   **`$DRYRUN`** - If `true` does not eval commands passed to `_execute_` function. (Default: `false`)

-   **`$LOGFILE`** - Path to a log file

-   **`$LOGLEVEL`** - One of: FATAL, ERROR, WARN, INFO, DEBUG, ALL, OFF (Default: `ERROR`)

-   **`$QUIET`** - If `true`, prints to log file but not stdout. (Default: `false`)



## arrays.bash



Utility functions for working with arrays.



-   **`_dedupeArray_`** Removes duplicate array elements

-   **`_forEachDo_`** Iterates over elements and passes each to a function

-   **`_forEachFilter_`** Iterates over elements, returning only those that are validated by a function

-   **`_forEachFind_`** Iterates over elements, returning the first value that is validated by a function

-   **`_forEachReject_`** Iterates over elements, returning only those that are NOT validated by a function

-   **`_forEachValidate_`** Iterates over elements and passes each to a function for validation

-   **`_inArray_`** Determine if a value is in an array

-   **`_isEmptyArray_`** Checks if an array is empty

-   **`_joinArray_`** Joins items together with a user specified separator

-   **`_mergeArrays_`** Merges the values of two arrays together

-   **`_randomArrayElement_`** Selects a random item from an array

-   **`_reverseSortArray_`** Sorts an array from highest to lowest

-   **`_setdiff_`** Return items that exist in ARRAY1 that are do not exist in ARRAY2

-   **`_sortArray_`** Sorts an array from lowest to highest



## checks.bash



Functions for validating common use-cases



-   **`_commandExists_`** Check if a command or binary exists in the PATH

-   **`_functionExists_`** Tests if a function is available in current scope

-   **`_isInternetAvailable_`** Checks if Internet connections are possible

-   **`_isAlpha_`** Validate that a given variable contains only alphabetic characters

-   **`_isAlphaDash_`** Validate that a given variable contains only alpha-numeric characters, as well as dashes and underscores

-   **`_isAlphaNum_`** Validate that a given variable contains only alpha-numeric characters

-   **`_isDir_`** Validate that a given input points to a valid directory

-   **`_isEmail_`** Validates that an input is a valid email address

-   **`_isFQDN_`** Determines if a given input is a fully qualified domain name

-   **`_isFile_`** Validate that a given input points to a valid file

-   **`_isIPv4_`** Validates that an input is a valid IPv4 address

-   **`_isIPv6_`** Validates that an input is a valid IPv6 address

-   **`_isNum_`** Validate that a given variable contains only numeric characters

-   **`_isTerminal_`** Checks if script is run in an interactive terminal

-   **`_rootAvailable_`** Validate we have superuser access as root (via sudo if requested)

-   **`_varIsEmpty_`** Checks if a given variable is empty or null

-   **`_varIsFalse_`** Checks if a given variable is false

-   **`_varIsTrue_`** Checks if a given variable is true



## dates.bash



Functions for working with dates and time.



-   **`_convertToUnixTimestamp_`** Converts a date to unix timestamp

-   **`_countdown_`** Sleep for a specified amount of time

-   **`_dateUnixTimestamp_`** Current time in unix timestamp

-   **`_formatDate_`** Reformats dates into user specified formats

-   **`_fromSeconds_`** Convert seconds to HH:MM:SS

-   **`_monthToNumber_`** Convert a month name to a number

-   **`_numberToMonth_`** Convert a month number to its name

-   **`_parseDate_`** Takes a string as input and attempts to find a date within it to parse into component parts (day, month, year)

-   **`_readableUnixTimestamp_`** Format unix timestamp to human readable format

-   **`_toSeconds_`** Converts HH:MM:SS to seconds



## debug.bash



Functions to aid in debugging BASH scripts



-   **`_pauseScript_`** Pause a script at any point and continue after user input

-   **`_printAnsi_`** Helps debug ansi escape sequence in text by displaying the escape codes

-   **`_printArray_`** Prints the content of array as key value pairs for easier debugging



## files.bash



Functions for working with files.



-   **`_backupFile_`** Creates a backup of a specified file with .bak extension or optionally to a specified directory.

-   **`_decryptFile_`** Decrypts a file with `openssl`

-   **`_encryptFile_`** Encrypts a file with `openssl`

-   **`_extractArchive_`** Extract a compressed file

-   **`_fileBasename_`** Gets the basename of a file from a file name

-   **`_fileContains_`** Tests whether a file contains a given pattern

-   **`_filePath_`** Gets the absolute path to a file

-   **`_fileExtension_`** Gets the extension of a file

-   **`_fileName_`** Prints a filename from a path

-   **`_json2yaml_`** Convert JSON to YAML uses python

-   **`_listFiles_`** Find files in a directory. Use either glob or regex.

-   **`_makeSymlink_`** Creates a symlink and backs up a file which may be overwritten by the new symlink. If the exact same symlink already exists, nothing is done.

-   **`_parseYAML_`** Convert a YAML file into BASH variables for use in a shell script

-   **`_printFileBetween_`** Prints block of text in a file between two regex patterns

-   **`_randomLineFromFile_`** Prints a random line from a file

-   **`_readFile_`** Prints each line of a file

-   **`_sourceFile_`** Source a file into a script

-   **`_createUniqueFilename_`** Ensure a file to be created has a unique filename to avoid overwriting other files

-   **`_yaml2json_`** Convert a YAML file to JSON with python



## macOS.bash



Functions useful when writing scripts to be run on macOS



-   **`_guiInput_`** Ask for user input using a Mac dialog box

-   **`_haveScriptableFinder_`** Determine whether we can script the Finder or not

-   **`_homebrewPath_`** Adds Homebrew bin directory to PATH

-   **`_useGNUUtils_`** Add GNU utilities to PATH to allow consistent use of sed/grep/tar/etc. on MacOS



## misc.bash



Miscellaneous functions



-   **`_acquireScriptLock_`** Acquire script lock to prevent running the same script a second time before the first instance exits

-   **`_detectLinuxDistro_`** Detects the host computer's distribution of Linux

-   **`_detectMacOSVersion_`** Detects the host computer's version of macOS

-   **`_detectOS_`** Detect the the host computer's operating system

-   **`_endspin_`** Clears output from the _spinner_

-   **`_execute_`** Executes commands with safety and logging options. Respects `DRYRUN` and `VERBOSE` flags.

-   **`_findBaseDir_`** Locates the real directory of the script being run. Similar to GNU readlink -n

-   **`_generateUUID_`** Generates a unique UUID

-   **`_progressBar_`** Prints a progress bar within a for/while loop

-   **`_runAsRoot_`** Run the requested command as root (via sudo if requested)

-   **`_seekConfirmation_`** Seek user input for yes/no question

-   **`_spinner_`** Creates a spinner within a for/while loop.

-   **`_trapCleanup_`** Cleans up after a trapped error.



## services.bash



Functions to work with external services



-   **`_haveInternet_`** Tests to see if there is an active Internet connection

-   **`_httpStatus_`** Report the HTTP status of a specified URL

-   **`_pushover_`** Sends a notification via Pushover (Requires API keys)



## strings.bash



Functions for string manipulation



-   **`_cleanString_`** Cleans a string of text

-   **`_decodeHTML_`** Decode HTML characters with sed. (Requires sed file)

-   **`_decodeURL_`** Decode a URL encoded string

-   **`_encodeHTML_`** Encode HTML characters with sed (Requires sed file)

-   **`_encodeURL_`** URL encode a string

-   **`_escapeString_`** Escapes a string by adding `\` before special chars

-   **`_lower_`** Convert a string to lowercase

-   **`_ltrim_`** Removes all leading whitespace (from the left)

-   **`_regexCapture_`** Use regex to validate and parse strings

-   **`_rtrim_`** Removes all leading whitespace (from the right)

-   **`_splitString_`** Split a string based on a given delimiter

-   **`_stringContains_`** Tests whether a string matches a substring

-   **`_stringRegex_`** Tests whether a string matches a regex pattern

-   **`_stripANSI_`** Strips ANSI escape sequences from text

-   **`_stripStopwords_`** Removes common stopwords from a string using a list of sed replacements located in an external file.

-   **`_trim_`** Removes all leading/trailing whitespace

-   **`_upper_`** Convert a string to uppercase



## template_utils.bash



Functions required to allow the script template and alert functions to be used



-   **`_makeTempDir_`** Creates a temp directory to house temporary files

-   **`_safeExit_`** Cleans up temporary files before exiting a script

-   **`_setPATH_`** Add directories to $PATH so script can find executables



# Coding conventions



-   Function names use camel case surrounded by underscores: `_nameOfFunction_`

-   Local variable names use camel case with a starting underscore: `_localVariable`

-   Global variables are in ALL_CAPS with underscores separating words

-   Exceptions to the variable an function naming rules are made for alerting functions and colors to ease my speed of programming. (Breaking years of habits is hard...) I.e. `notice "Some log item: ${blue}blue text${reset}` Where `notice` is a function and `$blue` and `$reset` are global variables but are lowercase.

-   Variables are always surrounded by quotes and brackets `"${1}"` (Overly verbose true, but a safe practice)

-   Formatting is provided by [shfmt](https://github.com/mvdan/sh) using 4 spaces for indentation

-   All scripts and functions are fully [Shellcheck](https://github.com/koalaman/shellcheck) compliant

-   Where possible, I follow [defensive BASH programming](https://kfirlavi.herokuapp.com/blog/2012/11/14/defensive-bash-programming/) principles.



## A Note on Code Reuse and Prior Art



I compiled these scripting utilities over many years without having an intention to make them public. As a novice programmer, I have Googled, GitHubbed, and StackExchanged a path to solve my own scripting needs. I often lift a function whole-cloth from a GitHub repo don't keep track of its original location. I have done my best within these files to recreate my footsteps and give credit to the original creators of the code when possible. Unfortunately, I fear that I missed as many as I found. My goal in making this repository public is not to take credit for the code written by others. If you recognize something that I didn't credit, please let me know.



## Contributing



### Setup



1. Install Python 3.11 and [Poetry](https://python-poetry.org)

2. Clone this repository. `git clone https://github.com/natelandau/shell-scripting-templates.git`

3. Install the Poetry environment with `poetry install`.

4. Activate your Poetry environment with `poetry shell`.

5. Install the pre-commit hooks with `pre-commit install --install-hooks`.



### Developing



-   Activate your Poetry environment with `poetry shell`.

-   This project follows the [Conventional Commits](https://www.conventionalcommits.org/) standard to automate [Semantic Versioning](https://semver.org/) and [Keep A Changelog](https://keepachangelog.com/) with [Commitizen](https://github.com/commitizen-tools/commitizen).

    -   When you're ready to commit changes run `cz c`

-   Run `poe` from within the development environment to print a list of [Poe the Poet](https://github.com/nat-n/poethepoet) tasks available to run on this project. Common commands:

    -   `poe lint` runs all linters and tests

-   Run `poetry add {package}` from within the development environment to install a runtime dependency and add it to `pyproject.toml` and `poetry.lock`.

-   Run `poetry remove {package}` from within the development environment to uninstall a runtime dependency and remove it from `pyproject.toml` and `poetry.lock`.

-   Run `poetry update` from within the development environment to upgrade all dependencies to the latest versions allowed by `pyproject.toml`.



## License



MIT