https://github.com/krivachy/logzio-search-export
CLI to export/dump a search result from Logz.io
https://github.com/krivachy/logzio-search-export
cli javascript logzio logzio-search nodejs
Last synced: 4 months ago
JSON representation
CLI to export/dump a search result from Logz.io
- Host: GitHub
- URL: https://github.com/krivachy/logzio-search-export
- Owner: krivachy
- License: mit
- Created: 2020-04-19T16:47:34.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2022-03-26T13:28:47.000Z (over 4 years ago)
- Last Synced: 2025-08-09T10:18:04.388Z (11 months ago)
- Topics: cli, javascript, logzio, logzio-search, nodejs
- Language: JavaScript
- Homepage:
- Size: 135 KB
- Stars: 6
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# logzio-search-export
> CLI to export/dump a search result from Logz.io up to millions of log entries
This CLI uses Logz.io's scroll API to export a search in batches of 1000. It can write to a file or pipe to stdout for
further processing.
## Install
Make sure you have node + npm installed (e.g. `brew install node`)
```
npm install -g logzio-search-export
```
To run without installing globally:
```
npx logzio-search-export
```
## Usage
```
$ export LOGZIO_API_TOKEN=
$ export LOGZIO_API_REGION=
$ logzio-search-export --help
Usage: logzio-search-export [options]
Options:
-t, --api-token Logz.io API token [envvar: LOGZIO_API_TOKEN]
-r, --region Logz.io region for account, defaults to eu [envvar: LOGZIO_API_REGION]
-s, --search A simple search term. For more complex queries pipe in via stdin.
-e, --extract Log entry fields to extract in output (can be provided multiple times) (default: all fields are returned) (default: [])
--start A Logz,io compatible query start time (default: "now-5m")
--end A Logz.io compatible end time (default: "now")
-f, --format Output format [json, csv] (default: "json")
-o, --output Output file to write results to (default: stdout)
-v, --verbose Print verbose output
-h, --help display help for command
```
## Examples
#### Simple example
- extracting the @timestamp and message fields
- query string for java errors
- a date range for yesterday
- an output file to write to
```shell script
logzio-search-export -e @timestamp -e message -s 'level: ERROR AND "java"' --start now-1d/d --end now/d -o log_lines.json
```
#### Stdout example
Note: Piping to stdout to a file is slower than using the `-o` flag
```shell script
# To a file
logzio-search-export -e message -s 'level: ERROR AND "java"' > log_lines.json
# Or to another program
logzio-search-export -e message -s 'level: ERROR AND "java"' | jq .
```
#### Complex query example
- extracting the @timestamp and message fields
- output to log lines
- query.json is a file that contains an [elastic search query JSON][1]
```shell script
echo "{ ... }" > query.json
logzio-search-export -e @timestamp -e message -o log_lines.json < query.json
```
#### CSV example
```shell script
logzio-search-export --format csv -o log_lines.csv < query.json
```
#### Minimalistic example
Will retain all fields on the log entry (quite a lot of data)
```shell script
logzio-search-export -o log_lines.json < query.json
```
#### Different regions
Your account is limited to one [Logz.io region][2] so make sure you're selecting one of:
- `eu` (default): app-eu.logz.io
- `us`: app.logz.io
- `au`: app-au.logz.io
- `ca`: app-ca.logz.io
- `nl`: app-nl.logz.io
- `wa`: app-wa.logz.io
You can export the `LOGZIO_API_REGION` environment variable to not have to provide the `--region` flag on each run.
## License
[MIT](./LICENSE)
[1]: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html
[2]: https://docs.logz.io/user-guide/accounts/account-region.html