{"id":18686409,"url":"https://github.com/dloss/klp","last_synced_at":"2025-04-04T10:06:20.083Z","repository":{"id":218983364,"uuid":"747831070","full_name":"dloss/klp","owner":"dloss","description":"Lightweight CLI viewer for structured log files and streams (logfmt, JSONL and many others)","archived":false,"fork":false,"pushed_at":"2025-02-26T18:28:23.000Z","size":3076,"stargazers_count":125,"open_issues_count":1,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-28T09:07:35.411Z","etag":null,"topics":["cli","logfmt","logging","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dloss.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-24T18:16:47.000Z","updated_at":"2025-03-05T14:57:06.000Z","dependencies_parsed_at":"2024-02-19T20:29:23.425Z","dependency_job_id":"6bfa994a-5ef2-438d-97fc-32071160695c","html_url":"https://github.com/dloss/klp","commit_stats":null,"previous_names":["dloss/klp"],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dloss%2Fklp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dloss%2Fklp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dloss%2Fklp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dloss%2Fklp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dloss","download_url":"https://codeload.github.com/dloss/klp/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247157281,"owners_count":20893220,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","logfmt","logging","python3"],"created_at":"2024-11-07T10:27:27.039Z","updated_at":"2025-04-04T10:06:20.038Z","avatar_url":"https://github.com/dloss.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# klp\n\n_Kool Log Parser_\n\nklp is a lightweight, command-line interface (CLI) tool for analyzing and understanding structured logs in various formats.\nDesigned for software developers involved in debugging and troubleshooting.\n\nSupports [logfmt](https://brandur.org/logfmt), [JSON Lines](https://jsonlines.org),\n[CSV](https://en.wikipedia.org/wiki/Comma-separated_values), [TSV](https://en.wikipedia.org/wiki/Tab-separated_values), [SQLite](https://sqlite.org) and many other formats.\n\nSingle file Python script without dependencies apart from Python 3.7+ and its standard library. \n\n![](demo/basic.png)\n\n## Table of Contents\n\n- [Key Features](#key-features)\n- [Getting Started](#getting-started)\n  - [Installation](#installation)\n  - [Quick Start](#quick-start)\n  - [Basic Usage](#basic-usage)\n- [Core Features](#core-features)\n  - [Input Handling](#input-handling)\n  - [Error Handling](#error-handling)\n  - [Event Creation](#event-creation)\n  - [Overview and Statistics](#overview-and-statistics)\n  - [Pattern Normalization and Deduplication](#pattern-normalization-and-deduplication)\n  - [Time Management](#time-management)\n  - [Search and Filter](#search-and-filter)\n  - [Output Control](#output-control)\n  - [Configuration File](#configuration-file)\n- [Advanced Usage](#advanced-usage)\n  - [Complex Filtering with Python Expressions](#complex-filtering-with-python-expressions)\n  - [Advanced Input Transformations Using Python Code](#advanced-input-transformations-using-python-code)\n  - [Custom Output Formatting using Python](#custom-output-formatting-using-python)\n  - [Parsing Non-Line-Based Formats](#parsing-non-line-based-formats)\n- [Additional Resources](#additional-resources)\n  - [Complementary Tools](#complementary-tools)\n  - [Alternative Tools](#alternative-tools)\n\n## Key Features\n\n- **Focus on Essentials**: \nInstantly highlight key fields: timestamp, log level, and message.\n- **Advanced Time Analysis**:\nTrack time gaps in logs, fuse bursts of events, and filter by complex time ranges.\n- **Smart Filtering**:\nFilter by log level, time spans, or using Python expressions. \n- **Pattern Search**:\nGrep in specific fields and show context lines. \nUse a wide range of built-in patterns to search or extract URLs, IPs, paths, email addresses, errors, and more. \n- **Visual Insights**:\nGet statistics about the number of events and the occuring log levels, visually mark time gaps between events and see a succinct map of log level patterns.\n- **Enhanced Readability**:\nColor-coding, per-key display, expandable newlines and custom output templates using Python expressions. \n- **Format Conversion**:\nConvert between common formats like logfmt, JSON, JSONL, CSV, TSV and SQLite.\n- **Powerful Log Transformation**:  \nTransform unstructured logs into structured data using Python code and convenient helper methods.\nCreate simple ad-hoc parsers for your own custom formats, add proper field names and save the result in any of the supported output formats.\n\n## Getting Started\n\n### Installation\n\nInstall using pip:\n\n```bash\n$ pip install klp-logviewer\n```\n\nOr just download `klp.py` and run it using `python3`, without any installation.\n\n### Quick Start\n\nHere are some basic examples to get you started with klp:\n\n```bash\n# View logs in logfmt format\n$ klp mylog.logfmt\n\n# Specify other input formats:\n$ klp -f jsonl applog.jsonl\n$ klp -f csv mydata.csv\n$ klp -f line text.txt\n\n# Only show specific keys\n$ klp -k timestamp,level,message mylog.logfmt\n\n# Don't print the keys, only the values\n$ klp -p mylog.logfmt\n\n# Filter logs by log level:\n$ klp -l error,warning mylog.logfmt\n\n# Get an overview of your log data:\n$ klp -S mylog.logfmt\n\n# View only new log entries (useful with streaming logs):\n$ tail -f mylog.logfmt | klp -n\n```\n\n\u003e [!NOTE]\n\u003e klp defaults to the [logfmt](https://brandur.org/logfmt) input format and will extract any `key=value` patterns it finds, \n\u003e even if the file is in a different format. \n\u003e For non-logfmt logs, specify the format explicitly (e.g. `-f syslog`, or `-f line` for plain text).\n\n\n### Basic Usage\n\nklp can be used as a filter, reading from stdin:\n\n```bash\n$ tail -f log.txt | klp\n```\n\nOr you can read from a file:\n\n```bash\n$ klp log.txt\n```\n\nklp can directly process compressed log files:\n\n```bash\n$ klp app.log.gz\n$ klp logs.zip\n```\n\nOptions can be given before or after the filename.\nSee `klp --help` for the list of available options.\n\n\n#### Basic Filtering\n\n**Filter by Log Level:**\nUse `--loglevels`/`-l` to restrict the output to specific log levels:\n\n```bash\n$ cat mylog.txt | klp -l warning,error,fatal\n```\n\nExclude log levels with `--not-loglevel`/`-L`:\n```bash\n$ cat mylog.txt | klp -L trace,debug\n```\n\n**Filter by Time:**\nklp tries to parse timestamps automatically. \nThe following keys are treated as possibly holding timestamps: `timestamp`, `ts`, `time`, `at`.\n\n```bash\n# Show events from the last hour\n$ klp --since 1h app.log\n\n# Show events from both a start and end time\n$ klp --from \"2024-02-05T20:18:10Z\" --to \"2024-02-05T20:19:20Z\" app.log\n```\n\n**Filter by Regex:**\n\nUse `--grep`/`-g` for simple text search with regular expressions:\n\n```bash\n# Search for specific text\n$ klp -g \"error\" app.log\n\n# Case-insensitive search\n$ klp -g \"error\" -i app.log\n\n# Search in specific fields\n$ klp -g \"message~error\" app.log\n```\n\n## Core Features\n\n### Input Handling\n\n#### Supported input formats\n\nklp supports a wide range of input formats:\n\n- `logfmt`: Key-value pairs (**default**)\n- `jsonl`: JSON Lines (shortcut: `-j`)\n- `json`: JSON (only for complete files, not for streaming)\n- `csv`: Comma-separated values \n- `tsv`: Tab separated values \n- `psv`: Pipe separated values \n- `line`: lines of text (trailing whitespace removed)\n- `tsNm`: logs containing a timestamp (with N components) and a message. See below for examples. \n- `tsNlm`: logs containing a timestamp (with N components), a log level and a message\n- `log4j`: Apache Log4j format\n- `cef`: Common Event Format (CEF)\n- `unix`: Common format of Unix server logs (timestamp, hostname, service, optional pid, message)\n- `rfc5424`: Syslog as specified in RFC5424 (including a priority)\n- `syslog`: Syslog (either RFC 5424 or the classical Unix one without priority)\n- `clf`: NCSA Common Log Format\n- `combined`: Combined Log Format of Apache httpd\n- `sqlite`: SQLite database (use `--input-tablename` if more than one table is present)\n- `data`: Parse everything as one string \n\nUse the `--input-format` or `-f` option to specify the input format. For example:\n\n```bash\n$ klp -f jsonl input.log\n```\n\nThe `tsNm` formats allows parsing logs that consist of a timestamp, and a message. \n`N` indicates how many whitespace-separated components the timestamp consists of.\nUse `tsNlm` for logs that contain a log level between the timestamp and the message.\nHere are some examples:\n\n```bash\n# Log format: \u003ctimestamp\u003e \u003cmessage\u003e\n# Example:    2024-02-05T20:18:10.538Z Connection established to database\n$ klp -f ts1m server.log\n\n# Log format: \u003cdate\u003e \u003ctime\u003e \u003clevel\u003e \u003cmessage\u003e\n# Example:    2024-02-05 20:18:10 INFO Starting background worker\n$ klp -f ts2lm worker.log\n\n# Log format: \u003cdate\u003e \u003ctime\u003e \u003ctimezone\u003e \u003cmessage\u003e\n# Example:    2024-02-05 20:18:10 UTC Processing batch job #1234\n$ klp -f ts3m batch.log\n```\n\nSimilar formats exist up to `ts5m`/`ts5lm` for logs with more timestamp components.\nThe timestamp parts are joined and parsed using klp's standard timestamp parsing logic.\n\nMore complicated formats can often be parsed using the `line` format and creating or transforming events using Python code (`--input-exec`).\nSee the [Advanced input transformations using Python code](#advanced-input-transformations-using-python-code) section below.\nAlternatively, you can check out [jc](https://github.com/kellyjonbrazil/jc), a tool that converts numerous formats to JSON, which can be read by klp `-f json`.\n\n#### Input Options\n\nklp provides several options to control how input data is read and parsed:\n\n- `--input-encoding`: Specify the text encoding of the input data (default: utf-8)\n  ```bash\n  # Read a file with Latin-1 encoding\n  $ klp --input-encoding latin1 legacy.logfmt\n  \n  # Process Windows-1252 encoded logs\n  $ klp -f line --input-encoding cp1252 windows.log\n  ```\n\n- `--no-header`: For CSV, TSV, or PSV formats, indicates that the file has no header row.\n  This will auto-generate numbered column keys.\n  ```bash\n  # Process a headerless CSV file (columns will be named col0, col1, etc.)\n  $ klp -f csv --no-header data.csv\n  ```\n\n- `--input-delimiter`: Specify the delimiter for CSV, TSV, or PSV formats\n  ```bash\n  # Process a file with semicolon-separated values\n  $ klp -f csv --input-delimiter \";\" european.csv\n  \n  # Read a custom-delimited file\n  $ klp -f csv --input-delimiter \":\" custom.log\n  ```\n\n- `--skip`: Avoid parsing the first N lines (e.g. to skip comments or old data).\n  ```bash\n  # Skip the first 5 lines of a CSV file\n  $ klp -f csv --skip 5 data.csv\n  ```\n\n- `--logical-lines`: Handle log entries that span multiple physical lines by combining related lines into a single event. Two methods are supported:\n \n  1. *Indentation Continuation*: Lines starting with whitespace are joined with a space between them (indentation removed)\n\n  ```bash\n  $ cat multi_indented.log\n  Starting SQL query:\n      SELECT * FROM users\n      WHERE active = true\n  Query completed\n\n  $ klp -f line --logical-lines multi_indented.log\n  line=\"Starting SQL query: SELECT * FROM users WHERE active = true\"\n  line=\"Query completed\"\n  ```\n  2. *Backslash Continuation*: Lines ending with `\\` are concatenated with the next line (backslash removed)\n\n  ```bash\n  $ cat multi_backslash.log\n  Dec 16 10:15:30 shade app[123]: Starting a long process \\\n  with multiple arguments and lots of long configuration o\\\n  ptions\n  Dec 16 10:15:31 shade app[123]: Process completed\n\n  $ klp -f unix --logical-lines multi_backslash.log\n  timestamp=\"Dec 16 10:15:30\" hostname=shade service=app pid=123 message=\"Starting a long process with multiple arguments and lots of long configuration options\"\n  timestamp=\"Dec 16 10:15:31\" hostname=shade service=app pid=123 message=\"Process completed\"\n  ```\n\n  The `--logical-lines` option can be combined with any line-based input format and works with streaming input.\n  It processes the lines before they are parsed according to the input format, so it works transparently with all klp's features.\n\n#### Error Handling\n\nControl how klp handles errors with the `--errors` option:\n\n- `ignore` (default): Silently continue processing when errors occur. With `--debug`, errors will still be printed.\n- `print`: Print error messages to stderr but continue processing\n- `exit`: Print error messages to stderr and exit with status code 1\n\n```bash\n# Print parsing errors but continue processing\n$ klp --errors print malformed.log\n\n# Stop processing at first error\n$ klp --errors exit app.log\n\n# Process as much as possible, ignoring errors (default)\n$ klp --errors ignore mixed.log\n\n# Show errors even in ignore mode\n$ klp --errors ignore --debug mixed.log\n```\n\n### Event Creation\n\nklp parses each line of the input file (or stdin stream) into an *event*.\nIf a line cannot be parsed, it is ignored silently.\nAn event can have several *fields* (key/value pairs).\nThe events in a file or stream don't have to have the same fields.\n\nklp tries to automatically detect timestamps, log levels and log messages\nby looking for the following keys:\n\n* Timestamp keys: `ts`, `time`, `timestamp`, `t`, `at`, `_t`, `@t`\n* Log level keys: `log_level`, `loglevel`, `level`, `lvl`, `loglevel`, `severity`, `levelname`, `@l`\n* Message keys: `message`, `msg`, `@m` \n\nIf your timestamp or log level fields aren't in these defaults, use `--ts-key KEY` and `--loglevel-key KEY` to specify them.\nklp has special features to select events by timestamp or log level (see below).\n\n\n### Overview and Statistics\n\n#### Stats View\nUse `--stats-only`/`-S` to get some basic statistics about number of events in the log file,\nthe timespan, the names of the keys and the log levels:\n\n```bash\n$ klp -S mylog.logfmt\nEvents shown: 8043 (100% of 8043 lines seen)\nTime span shown: 2024-02-05T19:58:59.874Z to 2024-02-05T20:19:20.575Z (0:20:20.701000)\nKeys seen: timestamp,log_level,source,target,thread_id,message\nLog levels seen: DEBUG,TRACE,INFO (keys: log_level)\n```\n\nOr use `--stats`/`-s` to print those stats (to stderr), in addition to regular output.\n\n#### Level Map\nUse `--levelmap`/`-M` to print only the first character of the log level of each event.\nThe timestamp is for the first event shown in that line:\n\n```bash\n$ klp -M mylog.logfmt\n2024-02-05T20:18:10.538Z TTTTDTTTDITITTTTTTTTTTTTTTTDTTTDITITTTTTTTTTTTTTTTDTTTDITITTTTTTTTTTTTTTTDTTTDIT\n2024-02-05T20:18:30.546Z ITTTTTTTTTTTTTTTDDTTITITTTTTTTTTTTTTTITITTTTTTTTTTTTTTTTITITTTTTTTTTTTTTTTTITITT\n```\n\n#### Key Map\nVisualize patterns for any key:\n\n```bash\n$ klp --keymap status_code app.log\n```\n### Pattern Normalization and Deduplication\n\nUse `--normalize` to replace common patterns (like IP addresses, emails, timestamps) with placeholders and only show unique event patterns. This helps identify distinct message types by ignoring variable data.\n\n```bash\n# Original logs\n$ klp app.log\ntimestamp=\"2024-02-14T12:00:01Z\" message=\"Failed login from 192.168.1.100 for user@example.com\"\ntimestamp=\"2024-02-14T12:00:05Z\" message=\"Connection from 10.0.0.50 to https://api.example.com\"\ntimestamp=\"2024-02-14T12:00:10Z\" message=\"Failed login from 192.168.1.101 for admin@example.com\"\ntimestamp=\"2024-02-14T12:00:15Z\" message=\"Connection from 10.0.0.51 to https://api.example.com\"\n\n# With normalization - shows unique patterns only\n$ klp --normalize app.log\ntimestamp=\"\u003ctimestamp\u003e\" message=\"Failed login from \u003cipv4\u003e for \u003cemail\u003e\"\ntimestamp=\"\u003ctimestamp\u003e\" message=\"Connection from \u003cipv4\u003e to \u003curl\u003e\"\n```\n\nThe following patterns are detected and normalized:\n- Timestamps → `\u003ctimestamp\u003e` (only in timestamp fields)\n- Email addresses → `\u003cemail\u003e`\n- IPv4/IPv6 addresses → `\u003cipv4\u003e`, `\u003cipv6\u003e`\n- URLs → `\u003curl\u003e` (requires protocol prefix like http://)\n- Domain names → `\u003cfqdn\u003e`\n- MAC addresses → `\u003cmac\u003e` (colon or dot separated)\n- Unix path names → `\u003cpath\u003e`\n- Hash values:\n  - MD5 → `\u003cmd5\u003e` (32 hex chars)\n  - SHA1 → `\u003csha1\u003e` (40 hex chars)\n  - SHA256 → `\u003csha256\u003e` (64 hex chars)\n- UUIDs → `\u003cuuid\u003e`\n- Version strings → `\u003cversion\u003e` (starting with v/V)\n- Function calls → `\u003cfunction\u003e` (name with parentheses)\n- Hex color codes → `\u003chexcolor\u003e` (6 digits with # prefix)\n- OAuth tokens → `\u003coauth\u003e`\n\nThe normalization considers only the fields that would be displayed (affected by `--keys`/`-k` and `--keys-not`/`-K`). \nTimestamps are only normalized in fields recognized as timestamp fields (configurable with `--ts-key`).\n\n```bash\n# Only consider message patterns, ignore timestamp differences\n$ klp --normalize -k message app.log\nmessage=\"Failed login from \u003cipv4\u003e for \u003cemail\u003e\"\nmessage=\"Connection from \u003cipv4\u003e to \u003curl\u003e\"\n\n# Focus on specific event properties\n$ klp --normalize -k level,component,message app.log\nlevel=error component=auth message=\"Invalid credentials for \u003cemail\u003e\"\nlevel=warn component=network message=\"Timeout connecting to \u003curl\u003e\"\n```\n\nThis feature is particularly useful for:\n- Identifying distinct log patterns in large files\n- Finding unique error types by normalizing variable data\n- Analyzing log structure and message templates\n- Reducing noise from repetitive events with different parameters\n\n### Time Management\n\n#### Time Filtering\n\nklp tries to parse timestamps automatically.\nThe following keys are treated as possibly holding timestamps: `timestamp`, `ts`, `time`, `at`.\nYou can specify your own key with `--ts-key KEY`.\n\nklp includes a list of common time formats that will be tried for parsing.\nUse `--ts-format` to specify your own format, using [Python strptime() format codes](https://docs.python.org/3/library/datetime.html#format-codes) (e.g. `%a %d %b %Y, %I:%M%p` to parse `Mon 31 Jan 2022, 11:59PM`).\nA short reference of the format codes can be printed with `klp --help-time`.\n\nThere are multiple ways to specify the time range that you're interested in:\n\n* absolute time: `--from ISODATETIME` and `--to ISODATETIME` \n* relative from now: `--since INTERVAL` and `--until INTERVAL` \n* timespan, computed from program start: `--duration INTERVAL`\n* timespan, computed from first event shown: , `--timespan INTERVAL`\n\nTimespans can be specified with these units (for example: `--since 1w2h3.2s`):\n\n- `w` = weeks\n- `d` = days\n- `h` = hours\n- `m` = minutes\n- `s` = seconds\n- `ms` = milliseconds\n- `us` = microseconds\n\n#### Timezone Control\nBy default, klp preserves timestamps as they appear in the logs.\nUse the following options to control how timestamps are displayed:\n\n* `--localtime`/`-z`: Convert timestamps to your local timezone\n* `--utc`/`-Z`: Convert timestamps to UTC\n\n```bash\n# Show timestamps in local timezone\n$ klp --localtime -c mylog.logfmt\n2015-03-26T06:27:38.000+01:00 debug Started observing beach\n\n# Show timestamps in UTC\n$ klp --utc -c mylog.logfmt\n2015-03-26T05:27:38.000Z debug Started observing beach\n```\n\nNote that timezone conversion can slightly impact performance when processing large log files.\nOnly use these options when timezone conversion is specifically needed.\n\nThese options cannot be used together - choose either local time or UTC.\nThey work with all input formats and can be combined with other timestamp-related features like `--since`, `--until`, and `--timespan`.\n\n\n#### Time Gaps\nUse `--mark-gaps` to visually separate events that are far apart in time:\n\n```bash\n$ klp prometheus.logfmt -c --mark-gaps 1h\n2024-02-05T11:00:02.081Z info Deleting obsolete block\n_______________________________ time gap: 1:59:59.984000 ________________________________\n2024-02-05T13:00:02.065Z info write block\n2024-02-05T13:00:02.075Z info Head GC completed\n2024-02-05T13:00:02.076Z info Creating checkpoint\n2024-02-05T13:00:02.235Z info WAL checkpoint complete\n_______________________________ time gap: 1:59:59.666000 ________________________________\n2024-02-05T15:00:01.901Z info write block\n2024-02-05T15:00:01.905Z info Head GC completed\n_______________________________ time gap: 1:59:59.984000 ________________________________\n2024-02-05T17:00:01.889Z info write block\n2024-02-05T17:00:01.893Z info Head GC completed\n```\n\n#### Event Condensing\nThe `--fuse` option allows you to condense events that occur close together in time.\n\n```bash\n$ klp -c --fuse 1h prometheus.logfmt\n1 2024-02-05T15:00:01.901Z info write block\n3 2024-02-05T15:00:01.905Z info Head GC completed\n_______________________________ time gap: 1:59:59.984000 ________________________________\n1 2024-02-05T17:00:01.889Z info write block\n9 2024-02-05T17:00:02.111Z info Deleting obsolete block\n_______________________________ time gap: 1:59:59.782000 ________________________________\n1 2024-02-05T19:00:01.893Z info write block\n3 2024-02-05T19:00:01.897Z info Head GC completed\n```\n\nThis shows only the first and last events for each group of events that occur within 1 hour of each other.\nFrom the event number in the first column you can see how many events were left out.\n\n### Search and Filter\n\n#### Grep Capabilities\nUse `--grep`/`-g` to limit the processing to lines that match a given regular expression.\nWhen this flag is given multiple times, any of those regexes matching \nwill allow the line to be processed (logical OR).\n\n```bash\n# Search with multiple patterns (OR)\n$ klp -g \"error\" -g \"warning\" app.log\n\n# Search in specific fields\n$ klp -g \"message~error\" app.log\n```\n\nTo specify lines that should NOT be processed, use `--grep-not`/`-G`/`-v`.\n\nSearch is case-sensitive by default. \nUse `--ignore-case`/`-i` for case-insensitive matches.\n\nTo output only specific parts of the match, see the [extracting capture groups](#extract-capture-groups) section below.\n\n#### Context Lines\nLike with the original UNIX grep, klp can print context lines:\n- `-B N`: Show N lines before match\n- `-A N`: Show N lines after match\n- `-C N`: Show N lines before and after match\n\nEvents before the matching line are visually marked with `/`, lines after with `\\`.\n\n#### Built-in Patterns\nklp has several builtin regexes to match URLs, email addresses, common errors, path names, FQDN's or IPv4 addresses.\nUse `--grep-builtin`/`-r` to use them for matching lines or `--grep-builtin-not`/`-R` for ignoring them.\n\nUse `--extract`/`-x` to output only the matched portions:\n\n```bash\n# Extract email addresses\n$ klp -f line -x email mail.log\n\n# Extract all URLs from JSONL logs\n$ klp -f jsonl --extract url app.jsonl\n\n# Extract both IPs and error messages\n$ klp -x ipv4 -x err server.logfmt\n```\n\nUse `klp -x ?` to list all available extraction patterns.\nThey are also documented as `extract_*()` functions on the `klp --help-python` screen.\n\n### Select Blocks of Lines\nYou can define start and stop conditions to process specific blocks of logs:\n\n- `--start-after REGEX`: Start processing after a line matching REGEX\n- `--start-with REGEX`: Start processing when a line matches REGEX\n- `--stop-with REGEX`: Stop processing after a line matches REGEX\n- `--stop-before REGEX`: Stop processing when a line matches REGEX\n\n```bash\n$ klp --start-after \"Session started\" --stop-before \"Session ended\" app.log\n$ klp --start-with \"[Uu]ser \\w+ logged in\" --stop-with \"[Uu]ser \\w+ logged out\" app.log\n```\n\nUse `--num-blocks` to limit the number of start/stop blocks processed.\n\n\n### Output Control\n\n#### Default Output\n\nBy default, klp prints every event that was recognized and shows both the key and the value of each field.\nColors are used for syntax-highlighting (use `--no-color` or the `NOCOLOR` environment variable to turn this off or `--theme` to select different colors).\n\n\n#### Key Selection\nTo select specific fields, use `--keys`/`-k` and a comma-separated list of keys:\n\n```bash\n$ klp -k timestamp,message mylog.txt\ntimestamp=\"2022-09-05T06:25:27.465Z\" message=\"poll_read: waiting on response\"\n```\n\nOr specify fields to exclude using `--keys-not`/`-K`.\n\nUse `--plain`/`-p` to leave out the keys and only print the values:\n\n```bash\n$ cat mylog.txt | klp -k -p timestamp,message\n2022-09-05T06:25:27.465Z poll_read: waiting on response\n```\n\n#### Common Output Format\n\nOften the time stamp, log level and message are the most important fields.\nUse `--common`/`-c` to print them, without showing the names of the keys (only the values):\n\n```bash\n$ klp -c mylog.txt\n2023-01-17T07:55:48.233Z DEBUG preparing query s190578: select * from applications\n2023-01-17T07:55:48.234Z DEBUG executing statement s190578 with parameters: []\n2023-01-17T07:55:48.656Z DEBUG preparing query s190579: select * from oauthproviderconfig where id = 0\n2023-01-17T07:55:48.657Z DEBUG executing statement s190579 with parameters: []\n2023-01-17T07:55:48.662Z DEBUG preparing query s190580: select * from oauthproviderconfig where id = 0\n2023-01-17T07:55:48.663Z DEBUG executing statement s190580 with parameters: []\n```\n\n#### Synthetic Fields\nklp can add some additional, synthetic fields to the event.\nFor performance reasons this is only done if you explicitly list these keys using `--keys`/`-k`.\n\n* `_klp_timedelta`: time span between this event and the event before (hours:minutes:seconds.microseconds)\n* `_klp_ts`: timestamp showing when the event was shown by klp\n\n#### Output Formats\nklp supports multiple output formats:\n\n- `default`: Colored and formatted logfmt\n- `logfmt`: Plain logfmt\n- `jsonl`: JSON Lines (shortcut: `-J`)\n- `json`: JSON\n- `csv`: Comma separated values\n- `tsv`: Tab separated values \n- `psv`: Pipe separated values\n- `sqlite`: SQLite database (use `--output-tablename` to set table name. Default: `data`)\n- `extract`: Only output the grep matches\n\nSelect an output format with `--output-format` or `-F`:\n\n```bash\n$ klp -F jsonl app.log \u003e output.jsonl\n$ klp -F sqlite -o app.db app.log\n```\n\n#### Custom Formatting\nCustomize your output with:\n\n- `--header`: Add text before the first event\n- `--footer`: Add text after the last event\n- `--output-sep`: Specify the separator between fields\n- `--output-event-sep`: Specify the separator between events (default: newline)\n- `--output-delimiter`: CSV/TSV/PSV output delimiter\n- `--output-quoting`: CSV/TSV/PSV output quoting (`minimal`,`all`,`nonnumeric`,`none`)\n- `--indent`, `--no-indent`: Control indentation\n- `--expand`: Expand newlines\n- `--no-wrap`: Disable line wrapping\n- `--each-key`: Print each field of an event on a separate line, with proper indentation\n\nFor advanced formatting, see [Custom Output Formatting using Python](#custom-output-formatting-using-python).\n\n\n#### Extract Capture Groups\n\nUse `--grep` with capture groups and `-F extract` to extract matching text from your logs.\nUse `--extract-group` to specify which capture groups to output (can be used multiple times, group 0 is the entire match and the default):\n\n```bash\n$ echo 'Events #85-653-3234 and #85-653-3560 in file mylogs.txt.' | klp -f line -g '#([\\d-]+)' --extract-group 1 -F extract\n85-653-3234\n85-653-3560\n\n$ echo 'Events #85-653-3234 and #85-653-3560 in file mylogs.txt.' | klp -f line -g '#([\\d-]+).*file (\\w+)' --extract-group 1 --extract-group 2 -F extract\n85-653-3234\nmylogs\n```\n\nUse `key~pattern` to match in specific fields.\n\n#### Output File Control\nUse `--output-file`/`-o` to direct output to a file instead of stdout.\nThis can be used to convert the events to a different file format:\n\n```bash\n# Write formatted output to a file\n$ klp app.log -o processed.log\n\n# Convert to JSON for later processing\n$ klp app.log -F json -o events.json\n\n# Write events to file but show stats on screen\n$ klp app.log -s -F jsonl -o events.jsonl\n```\n\nNote: Using `--output-file` disables color output by default, unless explicitly requested with `--color`.\n\n### Configuration File\n\nklp can be configured using an INI-style configuration file. The configuration allows setting default options and defining aliases for commonly used argument combinations.\n\n#### File Locations\n\nThe configuration file is searched for in these locations (in order):\n1. `$XDG_CONFIG_HOME/klp/config.ini` (usually `~/.config/klp/config.ini`)\n2. `~/.klprc` (legacy compatibility)\n\nOn Windows:\n1. `%APPDATA%\\klp\\config.ini`\n2. `%USERPROFILE%\\.klprc`\n\nUse `klp --show-config` to see the current configuration and search locations.\n\n#### File Format\n\nThe configuration file uses INI format with two sections:\n\n```ini\n[defaults]\n# Default values for command line options\ninput-format = logfmt\ninput-encoding = utf-8\nerrors = ignore\n\n[aliases]\n# Define shortcuts for commonly used argument combinations.\n# Aliases can reference other aliases with -a.\n\n# Input format shortcuts\nfmt_spring = -f line -I 'ts,level,thread,cls,msg=line.split(\"|\",4)'\nfmt_java = -f line -I 'ts,level,thread,msg=line.cols(\"0\",\"1\",\"2\",\"3:\")'\n\n# Output format shortcuts\nstyle_brief = -k timestamp,level,message -p\nstyle_full = -k timestamp,level,source,thread,message\n\n# Combined format (references other aliases)\nspring_brief = -a fmt_spring -a style_brief\n```\n\nUse aliases with the `-a/--alias` option, which can be specified multiple times:\n\n```bash\n# Use a single alias\n$ klp -a fmt_spring app.log\n\n# Combine multiple aliases\n$ klp -a fmt_spring -a style_brief app.log\n\n# Mix with other options\n$ klp -a spring_brief --mark-gaps 1h app.log\n```\n\n\n#### Ignoring Configuration\n\nUse `--ignore-config` to skip reading the configuration file entirely,\nif you want to avoid applying your default options.\n\n```bash\n# Run with default settings, ignoring any config file\n$ klp --ignore-config app.log\n```\n\n## Advanced Usage\n\n### Complex Filtering with Python Expressions\n\nUse `--where EXPR` to only process lines where the given Python expression is True.\nAll fields of the event are available by their key name.\n\n```bash\n$ cat httplog.txt | klp --where \"len(message) \u003e 10 and int(response[:3] \u003e 499\"\n```\n\nThe whole event dict can be accessed as the underscore `_`.\n\nBy default, errors are ignored, because some keys might not be available in all lines. \nUse `--debug` to print exceptions.\n\nThis can be combined with other filters, such as `--grep` and grep context lines.\n\n### Advanced Input Transformations Using Python Code\n\nUse `--input-exec`/`-I` to transform events using Python code. The code is executed for each event, with all event fields available as Python variables. New fields can be created by assigning to variables, and any field can be modified or deleted.\n\n```bash\n# Create uppercase message, extract status code, and add severity level\n$ klp app.log -I 'msg_upper = message.upper(); status = int(response[:3]); severity = \"error\" if status \u003e= 500 else  \"info\"'\n\n# Multiple transformations can be chained using additional -I options\n$ klp app.log -I 'msg_upper = message.upper()' -I 'word_count = len(message.split())'\n\n# Process line-based log, extract fields, and clean up\n$ klp app.log -f line -I 'ts,level,msg = line.split(maxsplit=2); del line' -I 'level = level.strip(\"[]\")'\n```\n\nThis shows how to use Python code to transform events, create new fields through assignment, chain multiple transformations, and remove intermediate fields that are no longer needed. \nEach transformation has access to fields created by previous ones.\n\nTo read the code from a file, use a `@` prefix:\n\n```bash\n$ klp app.log -I @myscript.py\n```\n\n#### Special Variables\n\nWhen using `--input-exec`/`-I`, there are three special variables available for more complex transformations.\nEach has an equivalent short form using only underscores:\n\n* `_klp_event` or `_`: Provides read-access to the current event dictionary. This is useful for keys that aren't valid Python identifiers.\n* `_klp_event_add` or `__`: The contents of this dictionary will be merged into the current event.\n* `_klp_events` or `___`: A list of dictionaries, where each dictionary represents an event. This lets you create multiple events from a single input event.\n\n```bash\n# Using _ to access keys with special characters\n$ klp examples/qryn.jsonl -j -I \"method=_['req.method'].lower()\"\n\n# Using _klp_event_add to add or update fields\n$ klp app.log -I \"_klp_event_add={'new_field': 'value', 'updated_field': msg.upper()}\"\n\n# Using _klp_events to generate multiple events\n$ klp app.log -I \"_klp_events=[{'split': word} for word in msg.split()]\"\n```\n\n#### Text Column Processing with `cols()`\n\nWhen using `--input-exec`/`-I`, any string value in your event (whether from the original parsed event or created during processing) can be split into columns using the `cols()` method.\nThis is especially useful for processing fixed-format logs or extracting information from complex field values.\n\nBackground: This method is available because the event key variables are an enhanced string type, which includes additional methods, such as `.cols()`.\n\n```python\ncols(column_spec, sep=None, outsep=\" \")\n```\n\n- `column_spec`: One or more column selectors (see below)\n- `sep`: Optional separator between input columns (defaults to whitespace)\n- `outsep`: String used to join columns within each selector (defaults to space)\n\nColumn selectors can be:\n- Single index: `\"0\"` or `0` (first column)\n- Multiple indices: `\"0,2\"` (first and third columns, joined)\n- Slice: `\"1:3\"` (second and third columns, joined)\n- Negative indices: `\"-2\"` (second-to-last column)\n- Mix of these: `\"-2,2,4:\"` (combines columns in each argument)\n\nMultiple arguments return a list instead of a string.\nSo with Python's tuple unpacking, you can create multiple new keys from just one `cols()` call:\n\n```bash\n# Split log line into timestamp, level, and message (rest of the line)\n$ klp app.log -f line -I 'ts,level,msg=line.cols(\"0,1\", \"2\", \"3:\")'\n\n# Extract information from a complex message field\n$ klp myapp.logfmt -I 'user,action=message.cols(\"1,2\", \"3:\", sep=\"|\")'\n```\n\nExample inputs/outputs to explain the column spec syntax: \n\n```python\nline = \"alpha beta gamma delta epsilon\"\nline.cols(0)          # Returns \"alpha\"\nline.cols(\"0,2\")      # Returns \"alpha gamma\"\nline.cols(0, 2)       # Returns [\"alpha\", \"gamma\"]\nline.cols(\"1:3\")      # Returns \"beta gamma\"\nline.cols(1, \"-2,2,4:\", 3)  # Returns [\"beta\", \"delta gamma epsilon\", \"delta\"]\n```\n\nThis method can be called on any string field in your event. \nOut-of-range indices are silently ignored, returning an empty string for invalid selections.\n\nAs a shortcut, you can just call the field, which will call the `cols()` method: \n\n```python\nline(1, \"-2,2,4:\", 3)  # Returns [\"beta\", \"delta gamma epsilon\", \"delta\"]\n```\n\n#### String Extraction Methods\n\nBuilding a new field often requires extracting a specific portion of an existing field based on a known substring.\nThe following methods provide a simpler alternative to manual indexing or regular expressions,\nmaking field extraction more intuitive:\n\n`.before(text, n=1)`: Return the part of string before the nth occurrence of text\n```bash\n$ echo 'time=2024-02-08T15:04:05Z msg=\"Error: Database: Connection failed\"' | \\\n  klp -I 'error_type = msg.before(\":\")'\ntime=2024-02-08T15:04:05Z msg=\"Error: Database: Connection failed\" error_type=Error\n```\n\n`.after(text, n=1)`: Return the part of string after the nth occurrence of text\n```bash\n# Extract the error message after the second colon\n$ echo 'time=2024-02-08T15:04:05Z msg=\"Error: Database: Connection failed\"' | \\\n  klp -I 'error_details = msg.after(\":\", 2)'\ntime=2024-02-08T15:04:05Z msg=\"Error: Database: Connection failed\" error_details=\" Connection failed\"\n```\n\n`.between(start, end, n=1)`: Return the part of string between the nth occurrence of start and the following end\n```bash\n# Extract text between parentheses\n$ echo 'time=2024-02-08T15:04:05Z msg=\"User (id=12345) logged in\"' | \\\n  klp -I 'user_id = msg.between(\"(\", \")\")'\ntime=2024-02-08T15:04:05Z msg=\"User (id=12345) logged in\" user_id=\"id=12345\"\n```\n\n`.starting_with(text, n=1)`: Return the substring from the nth occurrence of text to the end\n```bash\n# Get everything from 'Error' onwards\n$ echo 'time=2024-02-08T15:04:05Z msg=\"[INFO] Error: Connection failed\"' | \\\n  klp -I 'error_msg = msg.starting_with(\"Error\")'\ntime=2024-02-08T15:04:05Z msg=\"[INFO] Error: Connection failed\" error_msg=\"Error: Connection failed\"\n```\n\n`.ending_with(text, n=1)`: Return the substring from start through the nth occurrence of text\n```bash\n# Get the message up to the first colon\n$ echo 'time=2024-02-08T15:04:05Z msg=\"[INFO] Error: Connection failed\"' | \\\n  klp -I 'prefix = msg.ending_with(\":\")'\ntime=2024-02-08T15:04:05Z msg=\"[INFO] Error: Connection failed\" prefix=\"[INFO] Error:\"\n```\n\nAll these methods:\n- Are available on all fields\n- Accept an optional occurrence number `n` (default 1)\n- Support negative indices to count from the end: `msg.after(\":\", -1)`\n- Can be chained with other methods: `msg.after(\":\").strip()`\n- Return empty string if the marker text isn't found\n- Preserve the original string if marker text is empty\n\n#### Helper Functions\n\nWhen using `--input-exec`, you have access to a variety of helper functions designed for common log processing tasks. The full list of available functions and modules can be viewed using `klp --help-python`.\nHere are some that need more explanation:\n\n##### `extract_regex()`\n\n`extract_regex(pattern, text, *groupargs)`: Extract text matching a regular expression pattern, with optional capture group selection\n\n  - `pattern`: Regular expression pattern to match\n  - `text`: String to search within\n  - `groupargs`: Optional indices of capture groups to extract\n\nReturns matched text, or `None` if no match. With `groupargs`, returns tuple of matched groups.\n\n```bash\n# Extract the first number from a message\n$ echo 'time=2024-02-08T15:04:05Z msg=\"Server CPU usage: 95.2%\"' | \\\n  klp -I 'cpu=extract_regex(r\"CPU usage: (\\d+\\.?\\d*)%\", msg, 1)'\ntime=2024-02-08T15:04:05Z msg=\"Server CPU usage: 95.2%\" cpu=95.2\n\n# Extract multiple parts from a complex message\n$ echo 'time=2024-02-08T15:04:05Z msg=\"User alice (role=admin) accessed /api/users\"' | \\\n  klp -I 'user,role = extract_regex(r\"User (\\w+) \\(role=(\\w+)\\)\", msg, 1, 2)'\ntime=2024-02-08T15:04:05Z msg=\"User alice (role=admin) accessed /api/users\" user=alice role=admin\n```\n\nLike `cols()` and other helper functions, this is available as a method on the fields as well: \n\n```bash\n# Use method syntax. Extract whole match if no groups specified \n$ echo 'time=2024-02-08T15:04:05Z msg=\"Request from 192.168.1.100\"' | \\\n  klp -I 'ip=msg.extract_regex(r\"\\d+\\.\\d+\\.\\d+\\.\\d+\")'\ntime=2024-02-08T15:04:05Z msg=\"Request from 192.168.1.100\" ip=192.168.1.100\n```\n\n##### `parse_kv()`\n\n`parse_kv(text, sep=None, kvsep=\"=\")`: Parse key-value pairs from a string and merge them into the current event\n  - `text`: Input string containing key-value pairs\n  - `sep`: Separator between different pairs (whitespace by default)\n  - `kvsep`: Separator between keys and values (defaults to \"=\")\n  - Commonly used with `__` to add the parsed fields to the current event\n\n```bash\n# Parse query parameters into separate fields\n$ echo 'time=2024-02-08T15:04:05Z msg=\"GET /search\" query=\"user=alice\u0026role=admin\"' | \\\n  klp -I '__=parse_kv(query, sep=\"\u0026\")'\ntime=\"2024-02-08T15:04:05Z\" msg=\"GET /search\" query=\"user=alice\u0026role=admin\" user=\"alice\" role=\"admin\"\n\n# Parse metrics with custom separator\n$ echo 'time=2024-02-08T15:04:05Z msg=\"System stats\" metrics=\"cpu:95.2 mem:87.5 disk:45.8\"' | \\\n  klp -I '__=parse_kv(metrics, kvsep=\":\")'\ntime=\"2024-02-08T15:04:05Z\" msg=\"System stats\" metrics=\"cpu:95.2 mem:87.5 disk:45.8\" cpu=\"95.2\" mem=\"87.5\" disk=\"45.8\"\n```\n\n##### `parse_jsonl()`\n\n`parse_jsonl(line)`: Returns a dictionary with key-value pairs representing the fields in the first JSON object found.\n- `line`: A string that includes JSON somewhere in it.\n\n```bash\n# Example log line with JSON content\n$ echo 'time=2024-02-08T15:04:05Z level=info msg=\"User details\" data={\"user_id\": \"12345\", \"action\": \"login\"}' |   klp -f line -I '__=parse_jsonl(line)'\nline=\"time=2024-02-08T15:04:05Z level=info msg=\\\"User details\\\" data={\\\"user_id\\\": \\\"12345\\\", \\\"action\\\": \\\"login\\\"}\" user_id=12345 action=login\n```\n\n##### `sh()`\n\n`sh(command, **kwargs)`: Execute a shell command and return its output\n  - `command`: The shell command to execute\n  - Returns the command's stdout as a string (stripped of trailing whitespace)\n  - Raises an exception if the command fails (disable with `check=False`)\n  - Can be customized with subprocess.run keyword arguments\n\n```bash\n# Add git commit info to each event\n$ echo 'time=2024-02-08T15:04:05Z level=info msg=\"Deployment started\"' | \\\n  klp -I '__={\"git_rev\": sh(\"git rev-parse HEAD\")[:7]}'\ntime=2024-02-08T15:04:05Z level=info msg=\"Deployment started\" git_rev=62efbb3\n\n# Add system load information\n$ echo 'time=2024-02-08T15:04:05Z level=info msg=\"Health check\"' | \\\n  klp -I '__={\"load\": sh(\"uptime\").split(\"load average:\")[-1].strip()}'\n```\n\nNote: Use the `sh()` function carefully, and only with trusted input to avoid command injection vulnerabilities.\n\n### Custom Output Formatting using Python\n\nFor advanced output formatting, use `--output-template` (with Python f-strings) or `--output-eval` which allows Python code:\n\n```bash\n$ klp --output-template \"{format_datetime(timestamp)} - {extract_ipv4(message) or 'no IP'}\" app.log\n$ klp --output-eval \"f'{ts} {pprint_json(data)}'\" app.log\n```\n\nSeveral Python modules can be used in output expressions.\nUse `--help-python` to show the list of available modules and functions.\n\nThe following additional functions are available:\n\n* `extract_json()`: return the first JSON object or array in a given string as a string\n* `extract_email()`: return the first email address in a given string\n* `extract_fqdn()`: return the first FQDN in a given string\n* `extract_ipv4()`: return the first IPv4 address in a given string\n* `extract_regex()`: return first part of the string matching a regex\n* `extract_url()`: return the first URL in a given string\n* `format_datetime()`: format given string according to ISO 8601 (with millisecond precision), guessing the datetime format  \n* `guess_datetime()`: convert a string into a Python datetime object\n* `pprint_json()`: pretty print JSON data\n\n\n### Parsing Non-Line-Based Formats\n\nThe `-f data` option reads the entire input as a single string into a field named `data`.\nThis is particularly useful for creating ad-hoc parsers for formats that aren't naturally line-based, using `--input-exec` to transform the content.\n\nFor example, to parse a simple Windows-style INI file:\n\n```bash\n$ cat config.ini\n[database]\nhost = localhost\nport = 5432\n\n[api]\nurl = https://api.example.com\nkey = secret123\n\n$ klp -f data config.ini -I '___ = [{\"section\": s.split(\"\\n\")[0].strip(\"[]\"),\n                                    **dict(l.split(\" = \") for l in s.split(\"\\n\")[1:] if \"=\" in l)}\n                                   for s in data.split(\"\\n[\")]'\nsection=database host=localhost port=5432\nsection=api url=https://api.example.com key=secret123\n```\n\nThe key advantage of `-f data` is maintaining the complete file context, which makes it easier to split the content on the correct boundaries and extract related information that spans multiple lines.\nCombined with Python's string manipulation functions and regular expressions, this allows quick creation of parsers for various structured formats.\n\n\n## Additional Resources\n\n### Complementary Tools\n\nThese tools aren't specialized for log files but are useful to post-process or view klp's output, particularly the JSONL or TSV formats:\n\n#### General Purpose Data Processing\n- [jq](https://jqlang.github.io/jq/): A lightweight and flexible command-line JSON processor\n- [jaq](https://github.com/01mf02/jaq): A jq clone focussed on correctness, speed, and simplicity\n- [yq](https://github.com/mikefarah/yq): A portable command-line YAML, JSON, XML, CSV, TOML and properties processor\n- [Miller](https://github.com/johnkerl/miller): Like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON \n- [dasel](https://github.com/TomWright/dasel): Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats\n- [rare](https://github.com/zix99/rare): Create console histograms, bar graphs, tables, heatmaps and more in realtime using regex and expressions\n- [spyql](https://github.com/dcmoura/spyql): Query data on the command line with SQL-like SELECTs powered by Python expressions\n- [rq](https://git.sr.ht/~charles/rq): CLI tool for evaluating Rego queries\n- [RecordStream](https://github.com/benbernard/RecordStream): A system for command-line analysis of data\n\n\n#### CSV Processing\n- [qsv](https://github.com/jqnatividad/qsv): CSVs sliced, diced \u0026 analyzed\n- [csvlens](https://github.com/YS-L/csvlens): Command line csv viewer\n\n#### Format Conversion and Viewing\n- [jc](https://github.com/kellyjonbrazil/jc): CLI tool and python library that converts the output of popular command-line tools, file-types, and common strings to JSON, YAML, or Dictionaries\n- [jtbl](https://github.com/kellyjonbrazil/jtbl): CLI tool to convert JSON and JSON Lines to terminal, CSV, HTTP, and markdown tables\n- [evtx2json](https://github.com/vavarachen/evtx2json): convert Windows evtx files (Windows Event Log Files) into JSON\n- [tailspin](https://github.com/bensadeh/tailspin): A log file highlighter\n- [ov](https://github.com/noborus/ov): Feature-rich terminal-based text viewer. It is a so-called terminal pager.\n- [Visidata](https://www.visidata.org): A terminal spreadsheet multitool for discovering and arranging data\n- [Benthos](https://github.com/benthosdev/benthos.git): High performance and resilient stream processor, able to connect various sources and sinks\n\n#### Processing TSV Output\nThe TSV format can be processed by spreadsheet software and common Unix tools:\n\n- [cut](https://man7.org/linux/man-pages/man1/cut.1.html): Uses tab separator by default\n- [AWK](https://man7.org/linux/man-pages/man1/awk.1p.html): `-F$'\\t'`\n- [sort](https://man7.org/linux/man-pages/man1/sort.1.html): `-t$'\\t'`\n- [column](https://man7.org/linux/man-pages/man1/column.1.html): `-s$'\\t' -N timestamp,log_level,message -J` (can convert to JSON)\n- [Vim](https://www.vim.org): `:set list    :set listchars=tab:\u003e-`\n- [Pandas](https://pandas.pydata.org): `pd.read_table()`\n- [SQLite](https://www.sqlite.org):\n```sql\nsqlite\u003e .mode tabs\nsqlite\u003e .import data.tsv data\nsqlite\u003e select date(timestamp) as day, count(*) from data group by day;\n```\n\n### Alternative Tools\n\nIf klp doesn't meet your needs, here are some alternatives:\n\n#### Logfmt CLI/TUI\n- [hutils](https://github.com/brandur/hutils): The original logfmt tools (Ruby)\n- [lfq](https://github.com/mattcontinisio/lfq): Command-line logfmt processor (Go). Quite similar to klp. Faster, but fewer features\n- [angle-grinder](https://github.com/rcoh/angle-grinder): Slice and dice logs on the command line (Rust). Flexible query syntax\n- [pq](https://github.com/iximiuz/pq): Parse and Query log files as time series\n- [Kelpie](https://github.com/Yord/klp): Small, fast, and magical command-line data processor similar to pxi, jq, mlr, and awk\n- [lnav](https://github.com/tstack/lnav): Very advanced log file viewer, lots of formats\n- [tailtales](https://github.com/davidmoreno/tailtales): Flexible log viewer for logfmt and other formats with filtering, filtering expressions, and real-time pipe following\n\n#### JSON CLI/TUI\n- [fblog](https://github.com/brocode/fblog): Small command-line JSON Log viewer (Rust). Lua filters.\n- [json-log-viewer](https://github.com/gistia/json-log-viewer): Powerful terminal based viewer for JSON logs using ncurses (JavaScript). TUI\n- [Toolong](https://github.com/Textualize/toolong): A terminal application to view, tail, merge, and search log files, plus JSONL (Python). TUI\n- [sumoshell](https://github.com/SumoLogic/sumoshell): Collection of utilities to improve analyzing log files (Go). Successor is angle-grinder.\n- [l'oGGo](https://github.com/aurc/loggo): A powerful terminal app for structured log streaming (Go). TUI\n- [hl](https://github.com/pamburus/hl): A log viewer that translates JSON logs into human-readable representation (Rust)\n- [logdissect](https://github.com/dogoncouch/logdissect.git): CLI utility and Python module for analyzing log files and other data (Python). Multiple input formats\n\n#### Unstructured Logs CLI/TUI\n- [tailspin](https://github.com/bensadeh/tailspin): A log file highlighter (Rust)\n- [GoAccess](https://goaccess.io): Real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser\n- [logmine](https://github.com/trungdq88/logmine): A log pattern analyzer CLI\n- [kl](https://github.com/robinovitch61/kl): An interactive Kubernetes log viewer for your terminal\n\n#### Web UI\n- [Klogg](https://github.com/variar/klogg): Multi-platform GUI application that helps browse and search through long and complex log files\n- [frontail](https://github.com/mthenw/frontail/): Streaming logs to the browser\n- [LogScreen](https://github.com/soorajshankar/logScreen): Loglines can be messy, read it better on a browser, `command | npx logscreen`\n- [Logdy](https://logdy.dev): Web based real-time log viewer. Stream ANY content to a web UI with autogenerated filters. Parse any format with TypeScript\n\n#### Large Scale Solutions\n- [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/): Fast, resource-efficient log database with full-text search, LogsQL querying, and seamless Unix tool integration.\n- [OpenObserve](https://github.com/openobserve/openobserve): Cloud-native observability platform built specifically for logs, metrics, traces, analytics, RUM, designed to work at petabyte scale\n- [Graylog](https://github.com/Graylog2/graylog2-server): Free and open log management\n- [Grafana Loki](https://grafana.com/docs/loki/latest/visualize/grafana/): Horizontally-scalable, highly-available, multi-tenant log aggregation system\n- [qryn](https://qryn.metrico.in/): All-in-one polyglot observability\n- [Kubernetes History Inspector (KHI)](https://github.com/GoogleCloudPlatform/khi): An interactive log visualization tool that transforms Kubernetes logs into comprehensive timelines for efficient troubleshooting and analysis","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdloss%2Fklp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdloss%2Fklp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdloss%2Fklp/lists"}