{"id":47982315,"url":"https://github.com/nlink-jp/lookup","last_synced_at":"2026-04-14T04:00:34.019Z","repository":{"id":311748612,"uuid":"1044838913","full_name":"nlink-jp/lookup","owner":"nlink-jp","description":"A powerful Go-based CLI tool to enrich JSON/JSONL data streams by looking up values in CSV or JSON files. Inspired by Splunk's lookup command.","archived":false,"fork":false,"pushed_at":"2026-03-29T06:16:43.000Z","size":71,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-04T11:40:02.329Z","etag":null,"topics":["cli","csv","data-enrichment","data-pipeline","devpos-tools","go","golang","json","jsonl","lookup","splunk"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nlink-jp.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-26T09:31:49.000Z","updated_at":"2026-03-29T06:16:46.000Z","dependencies_parsed_at":"2025-08-26T11:58:09.728Z","dependency_job_id":"ed0cad11-a5c2-4743-82c0-ad7800037253","html_url":"https://github.com/nlink-jp/lookup","commit_stats":null,"previous_names":["magifd2/lookup-go","nlink-jp/lookup"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/nlink-jp/lookup","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nlink-jp%2Flookup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nlink-jp%2Flookup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nlink-jp%2Flookup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nlink-jp%2Flookup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nlink-jp","download_url":"https://codeload.github.com/nlink-jp/lookup/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nlink-jp%2Flookup/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31781292,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-14T02:24:21.117Z","status":"ssl_error","status_checked_at":"2026-04-14T02:24:20.627Z","response_time":153,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","csv","data-enrichment","data-pipeline","devpos-tools","go","golang","json","jsonl","lookup","splunk"],"created_at":"2026-04-04T11:10:16.677Z","updated_at":"2026-04-14T04:00:34.014Z","avatar_url":"https://github.com/nlink-jp.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# lookup: A Powerful CLI Lookup Tool\n\n`lookup` is a command-line utility inspired by Splunk's powerful `lookup` command. It enriches JSON data streams by adding fields based on matching values in an external data source (like a CSV or JSON file). It's designed to be a flexible and high-performance tool for data enrichment pipelines.\n\nThe tool reads JSON objects (either as a JSON Array or as JSON Lines) from standard input, performs lookups based on sophisticated rules, and outputs the enriched JSON objects to standard output.\n\n---\n\n## Features\n\n-   **Multiple Data Sources**: Use either **CSV** or **JSON** files as your lookup table.\n-   **Advanced Matching Methods**:\n    -   `exact`: Case-sensitive or insensitive exact string matching.\n    -   `wildcard`: Glob-style wildcard matching (e.g., `bot-*`).\n    -   `regex`: Powerful matching using regular expressions.\n    -   `cidr`: Match IP addresses against CIDR blocks (e.g., `10.0.0.0/8`).\n-   **Flexible Configuration**: A central JSON configuration file separates lookup logic from your data, allowing for complex matching rules.\n-   **Built-in DNS Lookup**: Perform forward (`A` record) or reverse (`PTR` record) DNS lookups as a native feature.\n    -   Optionally specify a custom DNS server for queries.\n-   **Flexible Field Mapping**: Intuitive syntax (`\u003cconfig_ref_field\u003e as \u003cinput_field\u003e OUTPUT out1 as new1, ...`) to control which fields are matched and how new fields are named.\n-   **Handles Multiple Input Formats**: Automatically detects and processes both **JSON Array** and **JSON Lines (JSONL)** from stdin.\n-   **Cross-Platform**: Written in Go, it compiles to a single binary with no external dependencies, running on Linux, macOS, and Windows.\n\n---\n\n## Installation\n\nPre-compiled binaries for macOS, Windows, and Linux are available on the [Releases](https://github.com/nlink-jp/lookup/releases) page.\n\n---\n\n## Configuration\n\nThe configuration file defines where your data is and how to match against it.\n\n### Configuration File (`config.json`)\n\n```json\n{\n  \"data_source\": \"./path/to/your/data.csv\",\n  \"matchers\": [\n    {\n      \"input_field\": \"field_from_stdin\",\n      \"lookup_field\": \"column_in_data_source\",\n      \"method\": \"exact\",\n      \"case_sensitive\": false\n    },\n    {\n      \"input_field\": \"another_field_from_stdin\",\n      \"lookup_field\": \"another_column\",\n      \"method\": \"regex\"\n    }\n  ]\n}\n```\n\n-   **`data_source`**: (string) The relative or absolute path to your lookup data file (CSV or JSON).\n-   **`matchers`**: (array) A list of objects, where each object defines a specific matching rule.\n    -   **`input_field`**: A name for this lookup rule. This is what you refer to in the `-m` flag.\n    -   **`lookup_field`**: The column/key name in your `data_source` file to match against.\n    -   **`method`**: The matching algorithm to use. Supported values:\n        -   `\"exact\"` (default)\n        -   `\"wildcard\"`\n        -   `\"regex\"`\n        -   `\"cidr\"`\n    -   **`case_sensitive`**: (boolean, optional) If `true`, the match will be case-sensitive. Defaults to `false`. This applies to `exact`, `wildcard`, and `regex` methods.\n\n### Configuration Helper (`generate-config`)\n\nTo make setup easier, `lookup` provides a helper command to generate a configuration template from your data file. It scans your CSV, JSON, or JSONL file and creates a valid `config.json` structure based on the headers or keys it finds.\n\n```sh\n./lookup generate-config -file \u003cpath_to_your_data_file\u003e\n```\n\n-   **`-file \u003cpath\u003e`**: The path to your data source file (e.g., `users.csv` or `data.jsonl`).\n\n**Example:** Given a `users.csv` file with columns `username`, `department`, and `role`:\n\n```sh\n./lookup generate-config -file users.csv\n```\n\nOutput (save as `config.json` and edit as needed):\n\n```json\n{\n  \"data_source\": \"users.csv\",\n  \"matchers\": [\n    {\n      \"input_field\": \"username\",\n      \"lookup_field\": \"username\",\n      \"method\": \"exact\",\n      \"case_sensitive\": false\n    },\n    {\n      \"input_field\": \"department\",\n      \"lookup_field\": \"department\",\n      \"method\": \"exact\",\n      \"case_sensitive\": false\n    },\n    {\n      \"input_field\": \"role\",\n      \"lookup_field\": \"role\",\n      \"method\": \"exact\",\n      \"case_sensitive\": false\n    }\n  ]\n}\n```\n\n---\n\n## Usage\n\nThe basic command structure is:\n\n```sh\ncat input.json | ./lookup -c \u003cconfig.json\u003e -m \"\u003cmapping_rule\u003e\"\n```\n\n### Command-Line Flags\n\n| Flag           | Description                                                                                                                              | Required |\n| :------------- | :--------------------------------------------------------------------------------------------------------------------------------------- | :------- |\n| `-c \u003cpath\u003e`    | Path to the JSON configuration file that defines the data source and matching rules.                                                     | Yes      |\n| `-m \u003cstring\u003e`  | The mapping rule that specifies how to link input data to the lookup table. (See [Mapping Syntax](#mapping-syntax) below).               | Yes      |\n| `--dns`        | Enables DNS lookup mode. When used, the `-c` flag is ignored.                                                                            | No       |\n| `--dns-server` | (Optional) Specifies a custom DNS server for DNS lookups (e.g., `8.8.8.8` or `1.1.1.1:53`). If not set, the system's default resolver is used. | No       |\n\n### Mapping Syntax\n\nThe `-m` flag defines the link between the input stream and the lookup table, and controls the output.\n\n```\n\"CONFIG_REF_FIELD as INPUT_FIELD [OUTPUT original_name1 as new_name1, original_name2 as new_name2]\"\n```\n\n-   **`CONFIG_REF_FIELD as INPUT_FIELD`**: (Required)\n    -   `CONFIG_REF_FIELD` must match an `input_field` in one of your matchers in `config.json`.\n    -   `INPUT_FIELD` is the name of the field in the incoming JSON object to use for the lookup.\n-   **`OUTPUT ...`**: (Optional)\n    -   Controls which fields from the lookup file are added to the output and allows you to rename them.\n    -   If omitted, all columns from the matched row are added with their original names.\n\n### Examples\n\nLet's use the following files for our examples.\n\n**`users.csv`** (Data Source)\n```csv\nusername,department,role,building,ip_range\njdoe,Sales,Manager,A,192.168.1.10\nasmith,Engineering,Developer,B,192.168.1.25\nb-*,Engineering,QA,B,10.0.0.0/8\n^scanner-.*$,IT,Service,A,\n```\n\n**`lookup_config.json`** (Configuration)\n```json\n{\n  \"data_source\": \"./users.csv\",\n  \"matchers\": [\n    {\n      \"input_field\": \"user_lookup\",\n      \"lookup_field\": \"username\",\n      \"method\": \"exact\",\n      \"case_sensitive\": false\n    },\n    {\n      \"input_field\": \"host_lookup\",\n      \"lookup_field\": \"username\",\n      \"method\": \"wildcard\"\n    },\n    {\n      \"input_field\": \"process_lookup\",\n      \"lookup_field\": \"username\",\n      \"method\": \"regex\"\n    },\n    {\n      \"input_field\": \"ip_lookup\",\n      \"lookup_field\": \"ip_range\",\n      \"method\": \"cidr\"\n    }\n  ]\n}\n```\n\n**`input.jsonl`** (Input Data)\n```json\n{\"timestamp\": \"2023-10-28T11:00:00Z\", \"user\": \"JDOE\", \"event\": \"login\"}\n{\"timestamp\": \"2023-10-28T11:01:00Z\", \"hostname\": \"b-jones\", \"event\": \"connect\"}\n{\"timestamp\": \"2023-10-28T11:02:00Z\", \"process\": \"scanner-01\", \"event\": \"scan\"}\n{\"timestamp\": \"2023-10-28T11:03:00Z\", \"client_ip\": \"10.20.30.40\", \"event\": \"access\"}\n{\"timestamp\": \"2023-10-28T11:04:00Z\", \"client_ip\": \"8.8.8.8\", \"event\": \"external_access\"}\n```\n\n#### Example 1: Case-Insensitive `exact` Match\n\n```sh\ncat input.jsonl | ./lookup \\\n  -c lookup_config.json \\\n  -m \"user_lookup as user OUTPUT department as dept, role\"\n```\n\n**Output (for the first line):**\n```json\n{\"dept\":\"Sales\",\"event\":\"login\",\"role\":\"Manager\",\"timestamp\":\"2023-10-28T11:00:00Z\",\"user\":\"JDOE\"}\n```\n\n#### Example 2: `cidr` Match\n\n```sh\ncat input.jsonl | ./lookup \\\n  -c lookup_config.json \\\n  -m \"ip_lookup as client_ip\"\n```\n\n**Output (for the fourth line):**\n```json\n{\"building\":\"B\",\"client_ip\":\"10.20.30.40\",\"department\":\"Engineering\",\"event\":\"access\",\"ip_range\":\"10.0.0.0/8\",\"role\":\"QA\",\"timestamp\":\"2023-10-28T11:03:00Z\",\"username\":\"b-*\"}\n```\n\n#### Example 3: DNS Lookup\n\n**Command (using system resolver):**\n```sh\ncat input.jsonl | ./lookup \\\n  --dns \\\n  -m \"dns_reverse_lookup as client_ip OUTPUT hostname as resolved_host\"\n```\n\n**Command (using a custom DNS server):**\n```sh\ncat input.jsonl | ./lookup \\\n  --dns \\\n  --dns-server \"8.8.8.8\" \\\n  -m \"dns_reverse_lookup as client_ip OUTPUT hostname as resolved_host\"\n```\n\n**Output (for the last line, may vary):**\n```json\n{\"client_ip\":\"8.8.8.8\",\"event\":\"external_access\",\"resolved_host\":\"dns.google\",\"timestamp\":\"2023-10-28T11:04:00Z\"}\n```\n\n---\n\n## Building\n\nTo build from source, you need Go and Make installed.\n\n```sh\n# Build for the current platform\nmake build\n\n# Cross-compile for all platforms (Linux amd64/arm64, macOS amd64/arm64, Windows amd64)\nmake build-all\n\n# Build all binaries and create .zip archives in dist/\nmake package\n```\n\nThe build artifacts will be placed in the `dist/` directory.\n\n---\n\n## Documentation\n\n- [Architecture](docs/en/architecture.md) — Module structure, data flow, testing strategy\n- [アーキテクチャ](docs/ja/architecture.ja.md) — Japanese version\n- [日本語ドキュメント](README.ja.md)\n\n## License\n\nThis project is licensed under the [MIT License](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnlink-jp%2Flookup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnlink-jp%2Flookup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnlink-jp%2Flookup/lists"}