Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/egordm/nauman
A CI inspired approach for local job automation.
https://github.com/egordm/nauman
automation command command-line-tool cron runner-script rust task task-runner yaml
Last synced: about 1 month ago
JSON representation
A CI inspired approach for local job automation.
- Host: GitHub
- URL: https://github.com/egordm/nauman
- Owner: EgorDm
- License: mit
- Created: 2021-12-03T23:35:11.000Z (almost 3 years ago)
- Default Branch: master
- Last Pushed: 2022-09-06T11:13:49.000Z (about 2 years ago)
- Last Synced: 2023-12-08T15:35:44.110Z (12 months ago)
- Topics: automation, command, command-line-tool, cron, runner-script, rust, task, task-runner, yaml
- Language: Rust
- Homepage:
- Size: 117 KB
- Stars: 10
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
nauman
A CI inspired approach for local job automation.
Features •
Installation •
Usage •
FAQ •
Examples •
Job Syntax
## About
`nauman` is an easy-to-use job automation tool. It arose from a necessity to automate complex task flows while still preserving the ability to monitor and debug them.
It is heavily inspired by simplicity of [Github Actions](https://docs.github.com/en/actions), flexibility of [Fastlane](https://github.com/fastlane/fastlane) and extensibility of [Apache Airflow](https://airflow.apache.org/). This tool aims to bring the best of both to local job automation.
## Quick Start
See [Installation](#installation) for how to install just on your computer. Try running `nauman --version` to make sure that it’s installed correctly.
Once `nauman` is installed and working, create a job file named `hello-world.yml` in the root of your project with the following contents:
```yaml
name: Hello World!
tasks:
- name: Hello World!
run: echo "Hello World!"
- name: Greeting
run: echo "Greetings ${USER}!"
```
When you invoke `nauman hello-world.yml` it runs the job tasks in the order they are listed in the file. The output should be as follows:
--------------------------
--- Task: Hello World! ---
--------------------------
$ echo "Hello World!"
Hello World!
----------------------
--- Task: Greeting ---
----------------------
$ echo "Greetings ${USER}!"
Greetings egordm!
`nauman` prints the output of each task to the console. The defined tasks run within your default shell and capture all of their output.
## Examples
For more examples, see the [examples](examples) directory.
* [Using Hooks](#using-hooks)
* [Logging](#logging)
* [Using Environment Variables](#using-environment-variables)
* [More](https://github.com/EgorDm/nauman/tree/master/examples)
### Using Hooks
[Hooks](#hooks) are first class citizens in `nauman`. They represent various events and callbacks that can occur during the execution of a job.
Let's take a look at a simple use case of hooks to add health checks to a job and its tasks. Create a file named `health-checks.yml` in the root of your project with the following contents:
```yaml
name: Example Job Using Health Checks
policy: always
tasks:
- name: Run a successful program
run: sleep 2 && echo "Success!"
hooks:
on_success:
- run: curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/fb4c4863-a7f1-44f1-8298-3baabec653d4
- name: Run a failing program
run: sleep 2 && exit 1
hooks:
on_success:
- run: curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/0178d446-9b50-4158-b50d-7df098945c81
on_failure:
- name: Send failing status code to Health Check
run: curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/0178d446-9b50-4158-b50d-7df098945c81/$NAUMAN_PREV_CODE
hooks:
after_job:
- name: On completion of the job, ping a health check
run: curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/fb4c4863-a7f1-44f1-8298-3baabec653d4
```
When you invoke `nauman health-checks.yml` it runs all the tasks withing the job file despite the fact that the second task fails (see [Execution Policy: always](#execution-policies)). See the output below:
--------------------------------------
--- Task: Run a successful program ---
--------------------------------------
$ sleep 2 && echo "Success!"
Success!
-------------------------------------------------------------------------------------------------------------
--- Hook: curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/fb4c4863-a7f1-44f1-8298-3baabec653d4 ---
-------------------------------------------------------------------------------------------------------------
$ curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/fb4c4863-a7f1-44f1-8298-3baabec653d4
-----------------------------------
--- Task: Run a failing program ---
-----------------------------------
$ sleep 2 && exit 1
Task "Run a failing program" completed in 2s with a non-zero exit status: 1. This indicates a failure
------------------------------------------------------
--- Hook: Send failing status code to Health Check ---
------------------------------------------------------
$ curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/0178d446-9b50-4158-b50d-7df098945c81/$NAUMAN_PREV_CODE
-----------------------------------------------------------
--- Hook: On completion of the job, ping a health check ---
-----------------------------------------------------------
$ curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/fb4c4863-a7f1-44f1-8298-3baabec653d4
On success of the first task, a success hook is executed which sends a health-check. On failure of the second task, a fail hook is executed sending a failure health-check. Finally, an after job hook is executed sending a job completion health-check.
### Logging
[Logging](#logging) is a powerful feature of `nauman` that allows you to log the output of your tasks and hooks to different output streams.
Create a file named `logging.yml` in the root of your project with the following contents:
```yaml
name: Example Job Using Logs
options:
log_dir: ./logs
tasks:
- name: Print Hello World to stdout
run: echo "Hello World!"
- name: Print Hello World to stderr
run: echo "Hello World!" >&2
logging:
- type: file
name: Print stdout to a file
stdout: true
stderr: false
output: ./stdout.log
- type: file
name: Print stderr to a file
stdout: false
stderr: true
output: ./stderr.log
- type: file
name: Print both stdout and stderr to separate files per task
split: true
output: ./separate_logs
- type: console
```
Run `nauman logging.yml` and see the output below:
-----------------------------------------
--- Task: Print Hello World to stdout ---
-----------------------------------------
$ echo "Hello World!"
Hello World!
-----------------------------------------
--- Task: Print Hello World to stderr ---
-----------------------------------------
$ echo "Hello World!" >&2
Hello World!
Additionally, the following files are created:
* `logs/logging_2021-12-05T18:11:14/`
* `separate_logs/`
* `000_print-hello-world-to-stdout.log`
* `001_print-hello-world-to-stderr.log`
* `stderr.log`
* `stdout.log`
Where the logs if the specified root directory for the logs (See `log_dir` in [Logging](#logging) for more details). All the logs are placed in an `logging_` subdirectory with the current date and time of the job run.
`stdout.log` and `stderr.log` are created for each log stream.
`separate_logs/` is created for each task and contains the stdout and stderr logs for that task.
### Using Environment Variables
[Environment Variables](#environment-variables) allow you to set environment variables for your job. There are multiple ways to set environment variables:
* As system environment variables: `KEY=VALUE nauman`
* As cli arguments: `nauman -e KEY=VALUE`
* As job configuration: `.env.KEY: VALUE`
* As task configuration: `.tasks..env.KEY: VALUE`
By creating `env-vars.yml` in the root of your project with the following content you can test them all:
```yaml
name: Example Environment variables
env:
PING_CMD: curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/
CHECK_1: fb4c4863-a7f1-44f1-8298-3baabec653d4
tasks:
- name: Job env var
run: echo $PING_CMD$CHECK_1
- name: Task env var
run: echo $PING_CMD$CHECK_1
env:
CHECK_1: fb4c4863-a7f1-44f1-8298-3baabec653d4
- name: System env var
run: echo $PING_CMD$CHECK_2
- name: Built-in env vars
run: echo "Previous task \"$NAUMAN_PREV_NAME\" finished with status $NAUMAN_PREV_CODE"
```
When you run `nauman env-vars.yml -e CHECK_2=0178d446-9b50-4158-b50d-7df098945c81` you will see the following output:
-------------------------
--- Task: Job env var ---
-------------------------
$ echo $PING_CMD$CHECK_1
curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/fb4c4863-a7f1-44f1-8298-3baabec653d4
--------------------------
--- Task: Task env var ---
--------------------------
$ echo $PING_CMD$CHECK_1
curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/fb4c4863-a7f1-44f1-8298-3baabec653d4
----------------------------
--- Task: System env var ---
----------------------------
$ echo $PING_CMD$CHECK_2
curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/0178d446-9b50-4158-b50d-7df098945c81
-------------------------------
--- Task: Built-in env vars ---
-------------------------------
$ echo "Previous task \"$NAUMAN_PREV_NAME\" finished with status $NAUMAN_PREV_CODE"
Previous task "System env var" finished with status 0
In the last task we can see that the `NAUMAN_PREV_NAME` and `NAUMAN_PREV_CODE` environment variables are used. These variables are set by the `nauman` based on the previous task. See [Environment Variables](#environment-variables) for more context specific environment variables.
## Features
* [Hook everything](#hook-everything)
* [Flexible Logging](#flexible-logging)
* [Context variables](#context-variables)
* [Configurable task plan](#configurable-task-plan)
* [Different shell types](#different-shell-types)
* [Dry run](#dry-run)
* [Task Outputs](#task-outputs)
* [Multiline commands](#multiline-commands)
* [Dotenv files](#dotenv-files)
* [Change your working directory](#change-your-working-directory)
### Hook everything
You can create hooks for all the possible outcomes and events of your job or your task. Create job or task-local hooks like this:
```yaml
tasks:
...
- name: My Task
hooks:
on_failure:
...
on_success:
...
before_task:
...
after_task:
...
hooks:
before_job:
...
after_job:
...
on_failure:
...
on_success:
...
before_task:
...
after_task:
...
```
### Flexible Logging
You can log to single or multiple files, to console and even choose which log streams to used (stdout, stderr, or both).
```yaml
logging:
- name: Log only stdout
type: file
stdout: true
stderr: false
output: ./stdout.log
- name: Logs split in files per task
type: file
stdout: true
stderr: true
split: true
output: ./per_task_logs
- name: Logs to console
type: console
stdout: true
stderr: true
- name: Append output to a shared file
type: file
stdout: true
stderr: true
output: /var/log/nauman/my_job.log
```
### Context variables
Define more flexible tasks by using context variables.
Currently, following context variables are supported:
* `NAUMAN_JOB_NAME` - Name of the job
* `NAUMAN_JOB_ID` - ID of the job
* `NAUMAN_TASK_NAME` - Name of the current task
* `NAUMAN_TASK_ID` - ID of the current task
* `NAUMAN_PREV_NAME` - Name of the previous task
* `NAUMAN_PREV_ID` - ID of the previous task
* `NAUMAN_PREV_CODE` - Exit code of the previous task
```yaml
tasks:
...
- name: Use context vars as env vars
run: echo $NAUMAN_TASK_NAME
```
### Configurable task plan
When one task fails it does not stop the whole job. You can configure the task execution plan to decide how to proceed.
You can choose between the following options:
* `always` - Always execute the task regardless of prior task status.
* `prior_success` - Execute the task only if prior task has succeeded.
* `no_prior_failed` - Execute the task only if no other task has failed.
```yaml
# Policy can be defined at job level
policy: no_prior_failed
tasks:
...
- name: Always run this task
# And overridden at task level
policy: always
```
### Different shell types
Aside from the default `sh` shell you can use `bash`, `python`, `ruby`, `php` or specify path to your own desired shell.
```yaml
# Specify a default shell
shell: bash
shell_path: /bin/bash
tasks:
...
- name: Python task
shell: python
run: print('Hello World!')
- name: Virtual env python
shell: python
shell_path: '/app/venv/bin/python'
run: print('Hello World!')
- name: Ruby task
shell: ruby
run: print('Hello World!')
- name: PHP task
shell: php
run: echo 'Hello World!';
```
### Dry run
Want to make sure that your job is configured correctly? You can run your job in dry run mode. This will verify that all tasks are syntactically correct, all shells are usable and warn you about any potential issues (such as missing directories).
```shell
nauman --dry-run my_job.yml
```
### Task Outputs
During the execution of every task, a temporary file is created where you can store the output variables. These files are automatically deleted after the task is finished. The variables specified in the output files will be loaded into the global context as environment variables.
The output file accepts dotenv style syntax.
```yaml
tasks:
...
- name: Append output to the output file
run: echo "foo=bar" >> "$NAUMAN_OUTPUT_FILE"
- name: Use the output variable
run: echo $foo
```
### Multiline commands
Sometimes commands can take up more space than a single line. You can use multiline strings to define your commands.
```yaml
tasks:
...
- name: Multiline
shell: python
run: |
import os
print(os.environ['NAUMAN_TASK_NAME'])
```
### Dotenv files
You can use dotenv files to define variables for your tasks.
```yaml
options:
dotenv: /path/to/my_env.env
```
### Change your working directory
You can change your working directory by using the `cwd` option.
```yaml
cwd: /my/project/dir
tasks:
...
- name: Change working directory to /my/project/dir/task1
cwd: ./task1
run: pwd
```
## FAQ
### Why use nauman?
Picture this: you want to periodically run your tool that syncs your favorite movies between services. This can be done with a cron job, but what if you want, to add more dependent tasks (like, also syncing your movie collections)? Easy, create a shell script that runs them both.
Now you want to keep track of their output (for debugging), you want to add health-checks, single process locking, etc. Shell scripts are not the best way to do this and can easily get very messy.
With `nauman` you can create and run a job file that covers it all in a readable and maintainable way.
Additionally `nauman` is written in Rust and can be installed bloat free onto any system as a simple binary. (See [Installation](#installation) for more details).
### When not to use nauman?
You should not use `nauman` for tasks where you need:
* A makefile:
* `nauman` is not meant to be a replacement for makefiles.
* It is meant to run a job to automate one single chain of tasks.
* It does not support task parallelism, recursion or other complex workflows.
* A data automation tool:
* `nauman` is not meant to be a replacement for data automation tools.
* It can be used to chain multiple data processing tasks together.
* But it does not provide anything for data loading, data processing or visualization.
* A CI tool:
* `nauman` is not meant to be a replacement for CI tools.
* It does not include any CI-specific features such as caching, build uploads or integrations with build tools.
## Installation
The binary name for nauman is `nauman`.
[Archives of precompiled binaries for nauman are available for Windows, macOS and Linux. Linux and Windows binaries are static executables.](https://github.com/EgorDm/nauman/releases) Users of platforms not explicitly mentioned below are advised to download one of these archives.
If you're a Rust programmer, nauman can be installed with cargo.
* Note that numane is tested with Rust 1.57.0, although nauman may work with older versions.
* Note that the binary may be bigger than expected because it contains debug symbols. This is intentional. To remove debug symbols and therefore reduce the file size, run strip on the binary.
```shell
$ cargo install nauman
```
### Building from source
nauman is written in Rust, so you'll need to grab a [Rust installation](https://www.rust-lang.org/) in order to compile it. nauman compiles with Rust 1.57.0 (stable) or newer. In general, nauman tracks the latest stable release of the Rust compiler.
To build nauman:
```shell
$ git clone https://github.com/EgorDm/nauman
$ cd nauman
$ cargo build --release
$ ./target/release/nauman --version
```
## Usage
The usual way to invoke `nauman` is to use the `nauman ` command. If you want to specify more options or to override some job settings, refer to the below full usage:
USAGE:
nauman [OPTIONS] <JOB>
ARGS:
<JOB> Path to job yaml file
OPTIONS:
--ansi <ANSI> Include ansi colors in output (default: true)
--dry-run <DRY_RUN> Dry run to check job configuration (default: false)
-e <ENV> List of env variable overrides
-h, --help Print help information
-l, --level <LEVEL> A level of verbosity, and can be used multiple times (default:
info) [possible values: debug, info, warn, error]
--log-dir <LOG_DIR> Directory to store logs in (default: current directory)
--system-env <SYSTEM_ENV> Whether to use system environment variables (default: true)
-V, --version Print version information
## [Job Syntax](https://github.com/EgorDm/nauman/blob/master/JOB_SYNTAX.md)
## Alternatives
If this is not what you are looking for, check out these cool alternatives:
* Bash or Makefile
* [just](https://github.com/casey/just) - is a handy way to save and run project-specific commands
* [fastlane](https://github.com/fastlane/fastlane) - is a tool for iOS and Android developers to automate tedious tasks like generating screenshots, dealing with provisioning profiles, and releasing your application
* [Apache Airflow](https://airflow.apache.org/) - is a platform created by the community to programmatically author, schedule and monitor workflows.
## TODO
* [x] Add support for .env files
* [ ] Add more tests
* [ ] Add a way to natively run web requests
* [x] Add a way to write outputs of different tasks
* [ ] Add a templating system
* [ ] Add a way to specify per log whether ansi is enabled or not
* [ ] Add flock support
* [ ] Always add console logging (only specify whether stdout and stderr should be logged)
## Contributing
As this is a hobby project, contributions are very welcome!
The easiest way for you to contribute right now is to use nauman, and see where it's lacking.
If you have a use case nauman does not cover, please file an issue. This is immensely useful to me, to anyone wanting to contribute to the project, and to you as well if the feature is implemented.
If you're interested in helping fix an [existing issue](https://github.com/EgorDm/nauman/issues), or an issue you just filed, help is appreciated.
See [CONTRIBUTING](./CONTRIBUTING.md) for technical information on contributing.
## License
This project is licensed under the terms of the MIT license. See the [LICENSE](LICENSE) file.