Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/observiq/stanza

Fast and lightweight log transport and processing.
https://github.com/observiq/stanza

golang log-parsing logging observability telemetry

Last synced: 6 days ago
JSON representation

Fast and lightweight log transport and processing.

Host: GitHub
URL: https://github.com/observiq/stanza
Owner: observIQ
License: apache-2.0
Created: 2020-07-06T18:37:56.000Z (over 4 years ago)
Default Branch: main
Last Pushed: 2024-12-11T23:33:08.000Z (about 2 months ago)
Last Synced: 2025-01-19T07:06:23.186Z (13 days ago)
Topics: golang, log-parsing, logging, observability, telemetry
Language: Go
Homepage:
Size: 4.6 MB
Stars: 183
Watchers: 13
Forks: 22
Open Issues: 42
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
- Governance: GOVERNANCE.md

Awesome Lists containing this project

README

[![Action Status](https://github.com/observIQ/stanza/workflows/Build/badge.svg)](https://github.com/observIQ/stanza/actions)
[![Action Test Status](https://github.com/observIQ/stanza/workflows/Tests/badge.svg)](https://github.com/observIQ/stanza/actions)
[![codecov](https://codecov.io/gh/observIQ/stanza/branch/master/graph/badge.svg)](https://codecov.io/gh/observIQ/stanza)
[![Go Report Card](https://goreportcard.com/badge/github.com/observIQ/stanza)](https://goreportcard.com/report/github.com/observIQ/stanza)
[![License](https://github.com/observIQ/stanza/workflows/license/badge.svg)](https://github.com/observIQ/stanza/license)
[![Gosec](https://github.com/observIQ/stanza/actions/workflows/gosec.yml/badge.svg)](https://github.com/observIQ/stanza/actions/workflows/gosec.yml)

# About Stanza

Stanza is a fast and lightweight log transport and processing agent. It's designed as a modern replacement for Fluentd, Fluent Bit, and Logstash and can run as a standalone agent on all major operating systems. Stanza is also highly integrated to perform seamlessly with the applications in Google Cloud Platform (GCP) based production environments.

## OpenTelemetry

Stanza has been contributed to the [OpenTelemetry](https://opentelemetry.io/) project and will be intergrated into the [OpenTelemetry collector](https://github.com/open-telemetry/opentelemetry-collector).

# Features

- Flexible
- Supports many different input types such as file, journald, windows events, tcp / udp, and external APIs (cloudwatch, azure log analytics) as well as parsing with json and regex.
- Easily extended by writing an "operator" or "plugin" which is just a unit of code that performs a task such as reading data from a source, parsing data, or shipping data.
- Pre-built Plugins
- Over 80 Plugins have been pre-built and are ready to be configured.
- Lightweight with low resource consumption
- Uses next to no resource while idling. It does not pollute the system with tons of clutter, it exists strictly in /opt/observiq/stanza with just a few files.
- Written in pure Go
- Everything is self contained into a single binary, there are no external dependencies.
- High Performance
- Stanza is lightweight, blazing-fast, and designed to scale.

## Supported [Plugins](https://github.com/observIQ/stanza-plugins)

Utilize Plugins to get up and running quickly. Here's a quick list of Stanza's most popular plugins:

These are many of the Plugins supported by Stanza, with more being developed all the time. View a full list of Plugins [here](https://github.com/observIQ/stanza-plugins/tree/main/plugins).

## Supported Operating Systems

### Windows
- 2012
- 2012 r2
- 2016
- 2019
- 2022

### Linux
- Ubuntu 16.04, 18.04, 20.04
- Debian 9, 10, 11
- Alma, Rocky 8
- Red Hat 7, 8
- Suse 12, 15
- CentOS 7, 8 Stream

### MacOS
- 12 (Monterey)
- 11 (Big Sur)
- 10.15 (Catalina)
- 10.14 (Mojave)

# Quick Start

## Installation

### Linux Package Manager

Linux packages are available for the following Linux Distributions:
- RHEL / CentOS 7 and 8
- Oracle Linux 7 and 8
- Alma, Rocky Linux
- Fedora 30 and newer
- Debian 9 and newer
- Ubuntu LTS 16.04 and newer

Once installed, Stanza will be running under a systemd service named `stanza` as the user `stanza`.

#### RPM Install

On Red Hat based platforms, Stanza can be installed with:
```bash
sudo dnf install https://github.com/observIQ/stanza/releases/download/v1.6.1/stanza_1.6.1_linux_amd64.rpm
sudo systemctl enable --now stanza
```

On RHEL / Centos 7, use `yum` instead of `dns`.

On Suse based platforms, Stanza can be installed with:
```bash
sudo zypper install https://github.com/observIQ/stanza/releases/download/v1.6.1/stanza_1.6.1_linux_amd64.rpm
sudo systemctl enable --now stanza
```

Be sure to replace the URL with the version you require. You can find Stanza versions [here](https://github.com/observIQ/stanza/releases).

#### DEB Install

On Debian / Ubuntu based platforms, Stanza can be installed with:

```bash
curl -L -o stanza.deb https://github.com/observIQ/stanza/releases/download/v1.6.1/stanza_1.6.1_linux_amd64.deb
sudo apt-get install -f ./stanza.deb
sudo systemctl enable --now stanza
```

#### Changing the Runtime User

Sometimes it may be nessisary to have Stanza run as `root`. This can be
accomplished by creating a systemd override.

Run `sudo systemctl edit stanza` and paste:
```
[Service]
User=root
Group=root
```

Restart Stanza: `sudo systemctl restart stanza`.

### Linux / macOS Script

- Single command install, requires the `curl` command
- Stanza will automatically be running as a service
- On Linux, Stanza will be running as the `root` user. On Macos, Stanza will be running as your current user.
- `sudo` is always required for Linux installations while macOS requires it if the invoking user does not have write permissions to `/usr/local/bin`.

```shell
sh -c "$(curl -fsSlL https://github.com/observiq/stanza/releases/latest/download/unix-install.sh)" unix-install.sh
```

### Windows Script

```pwsh
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 ; Invoke-Expression ((New-Object net.webclient).DownloadString('https://github.com/observiq/stanza/releases/latest/download/windows-install.ps1')); Log-Agent-Install
```

### Kubernetes

To deploy Stanza to Kubernetes, AKS, EKS, GKE or Openshift check out the installation guides [here](/docs/examples/k8s).

## Configuration

To get started navigate to the `config.yaml` file in the Stanza install directory, located in the following locations by default:

- Linux: `/opt/observiq/stanza`
- MacOS: `/Users//observiq/stanza`
- Windows: `C:\observiq\stanza`

You can utilize [operators](/docs/operators/README.md) and [plugins](/docs/plugins.md) in a pipeline to easily configure Stanza to ship logs to your target destination.

Stanza also offers several outputs to be configured for sending data, including:

- [Stdout](/docs/operators/stdout.md)
- [File](/docs/operators/file_output.md)
- [Google Cloud Logging](/docs/operators/google_cloud_output.md)

In the below examples, Stanza is configured to ship logs to Google Cloud logging using the file_input operator, and the MySQL plugin. You will need to have a `credentials.json` for your GCP environment which can be generated by following Google's documentation [here](https://cloud.google.com/iam/docs/creating-managing-service-account-keys).

### Operators
This `config.yaml` collects logs from a file and sends them to Google Cloud. A full list of available operators can be found [here](/docs/operators/README.md).

```yaml
pipeline:
# An example input that monitors the contents of a file.
# For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/file_input.md
- type: file_input
include:
- /sample/file/path.log

# An example output that sends captured logs to Google Cloud.
# For more info: https://github.com/observIQ/stanza/blob/master/docs/operators/google_cloud_output.md
- type: google_cloud_output
credentials_file: /tmp/credentials.json
```

### Plugins
This `config.yaml` collects logs from MySQL via a plugin and sends them to Google Cloud. By default, MySQL plugin collects general, slow query, and error logs. More details of the MySQL plugin can be viewed [here](https://github.com/observIQ/stanza-plugins/blob/main/plugins/mysql.yaml). A full list of available plugins can be found [here](https://github.com/observIQ/stanza-plugins/blob/main/plugins/).

```yaml
pipeline:
# An example input that configures a MySQL plugin.
# For more info: https://github.com/observIQ/stanza/blob/master/docs/plugins.md
- type: mysql
enable_general_log: true
general_log_path: "/var/log/mysql/general.log"

That's it! Logs should be streaming to Google Cloud.

For more details on installation and configuration, check out our full [Install Guide](/docs/README.md)!

### Common Scenarios

To see specific examples of Stanza configuration, check out the [scenarios](/docs/examples/scenarios). Below are some of our more popular scenarios:

- [Syslog](/docs/examples/scenarios/syslog.md)
- [MySQL](/docs/examples/scenarios/mysql.md)
- [Windows Events](/docs/examples/scenarios/windows_events.md)
- [File](/docs/examples/scenarios/file.md)
- [Custom Parsing](/docs/examples/scenarios/custom_parsing.md)

# Community

Stanza is an open source project. If you'd like to contribute, take a look at our [contribution guidelines](/CONTRIBUTING.md) and [developer guide](/docs/development.md). We look forward to building with you.

## Code of Conduct

Stanza follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md). Please report violations of the Code of Conduct to any or all [maintainers](MAINTAINERS.md).

# Other questions?

Check out our [FAQ](/docs/faq.md), send us an [email](mailto:[email protected]), or open an issue with your question. We'd love to hear from you!