{"id":17793405,"url":"https://github.com/jaimemartinez/logxdb","last_synced_at":"2026-05-07T18:46:32.740Z","repository":{"id":259297547,"uuid":"876842628","full_name":"jaimemartinez/LogxDB","owner":"jaimemartinez","description":"LogxDB is a high-performance log parser designed to handle multiple log files concurrently with multiprocessing and auto-detect encoding to ensure compatibility with diverse formats. It stores parsed data in SQLite databases with support for custom regex patterns, table names, and column orders.","archived":false,"fork":false,"pushed_at":"2024-10-23T20:51:30.000Z","size":143,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-02T02:18:23.748Z","etag":null,"topics":["cli-tool","data-extraction","json","log-parser","logging","open-source","python3","regex-parsing","sqlite-database","yaml"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jaimemartinez.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-22T16:36:27.000Z","updated_at":"2024-11-09T01:32:02.000Z","dependencies_parsed_at":"2024-10-24T06:45:28.547Z","dependency_job_id":"eb1d31c6-b2d8-4a85-a387-8331f6f315dc","html_url":"https://github.com/jaimemartinez/LogxDB","commit_stats":null,"previous_names":["jaimemartinez/logxdb"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaimemartinez%2FLogxDB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaimemartinez%2FLogxDB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaimemartinez%2FLogxDB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaimemartinez%2FLogxDB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jaimemartinez","download_url":"https://codeload.github.com/jaimemartinez/LogxDB/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246741117,"owners_count":20826067,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli-tool","data-extraction","json","log-parser","logging","open-source","python3","regex-parsing","sqlite-database","yaml"],"created_at":"2024-10-27T11:08:27.206Z","updated_at":"2026-05-07T18:46:32.675Z","avatar_url":"https://github.com/jaimemartinez.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# LogxDB: Simple Log Parsing Library for Everyone  \n\n**LogxDB** is a high-performance log parser designed to handle multiple log files concurrently with multiprocessing and auto-detect encoding to ensure compatibility with diverse formats. It stores parsed data in SQLite databases with support for custom regex patterns, table names, and column orders.\n\n---\n\n## Project Structure\n\n```\nlogxdb/\n│\n├── parser.py              # Core logic: parsing logs, handling SQLite, multiprocessing\n├── parser_cli.py          # Command-line tool with regex REPL and configuration support\n├── test/\n│   ├── main.py                   # Example script to test and use the parser\n│   ├── example_log_500_lines.log # Example log file with timestamped entries\n│   ├── another_log.log           # Example log file with event-based entries\n├── LICENSE               # License file (MIT License)\n└── README.md             # Documentation and usage guide for the project\n```\n\n---\n### **Summary of Functionalities:**\n\n1. [**Logging Configuration**](#1-logging-configuration): Logs operations with different verbosity levels.\n2. [**Encoding Detection**](#2-detect-file-encoding): Detects encoding using `chardet`.\n3. [**Regex Matching**](#3-find-matching-regex-for-file): Identifies the appropriate regex for a log file.\n4. **Log Preprocessing**: Handles unmatched lines by appending them to the previous valid entry.\n5. **Multiple Match Handling**: Processes multiple regex matches within the same line.\n6. **SQLite Database Creation**: Creates tables dynamically if they don't exist.\n7. **Data Insertion**: Saves parsed data to the SQLite database.\n8. **Single File Processing**: Handles the entire process for a single log file.\n9. **Configuration File Loading**: Loads YAML/JSON configuration files.\n10. **Config-Based Parsing**: Uses configuration files for batch processing.\n11. **Multiprocessing**: Accelerates processing using multiple cores.\n\n---\n\n### **1. Logging Configuration**\n- **Description**: Configures logging to print to console and/or log files at different levels (DEBUG, INFO, WARNING, etc.).\n\n**Code Location**:\n```python\ndef configure_logging(self, log_level, log_file):\n    numeric_level = getattr(logging, log_level.upper(), logging.INFO)\n    handlers = [logging.StreamHandler()]\n    if log_file:\n        handlers.append(logging.FileHandler(log_file))\n    logging.basicConfig(\n        level=numeric_level,\n        format='%(asctime)s - %(levelname)s - %(message)s',\n        datefmt='%Y-%m-%d %H:%M:%S',\n        handlers=handlers\n    )\n```\n\n\n#### **How to Use**:\n```python\nfrom parser import LogParser\n\n# Initialize the parser with a log file and DEBUG level logging\nparser = LogParser(db_path='logs.db', log_level='DEBUG', log_file='parser.log')\n\n# This will configure logging to write to 'parser.log'\n```\n\n\n---\n\n### **2. Detect File Encoding**\n- **Description**: Detects the encoding of a given log file using **chardet** and provides a fallback if detection confidence is low.\n\n**Code Location**:\n```python\ndef detect_encoding(self, file_path):\n    try:\n        with open(file_path, 'rb') as f:\n            raw_data = f.read(4096)\n            result = chardet.detect(raw_data)\n            encoding = result.get('encoding', 'utf-8')\n            confidence = result.get('confidence', 0)\n            logging.info(f\"Detected encoding for {file_path}: {encoding} (Confidence: {confidence:.2f})\")\n            if confidence \u003c 0.7:\n                encoding = 'utf-8-sig'\n                logging.warning(f\"Low confidence in encoding detection. Defaulting to 'utf-8-sig'.\")\n            return encoding\n    except Exception as e:\n        logging.error(f\"Failed to detect encoding for {file_path}: {e}\")\n        return 'utf-8'\n```\n\n#### **How to Use**:\n```python\nencoding = parser.detect_encoding('example.log')\nprint(f\"Detected encoding: {encoding}\")\n```\n\n#### **Expected Output**:\n```\nDetected encoding: utf-8-sig\n```\n\n---\n\n### **3. Find Matching Regex for File**\n- **Description**: Iterates through the provided regex patterns to find the first matching one for a log file.\n\n**Code Location**:\n```python\ndef find_matching_regex(self, file_path, regexes):\n    encoding = self.detect_encoding(file_path)\n    with open(file_path, encoding=encoding, errors='replace') as f:\n        for line in f:\n            for regex in regexes:\n                if re.search(regex, line):\n                    logging.info(f\"Using regex: {regex} for file {file_path}\")\n                    return regex\n    logging.error(f\"No matching regex found for {file_path}.\")\n    raise LogParserError(f\"No matching regex found for {file_path}.\")\n```\n\n\n\n#### **How to Use**:\n```python\nregexes = [\n    r\"\\d{2}\\/\\d{2}\\/\\d{4}\",  # Example regex pattern\n    r\"\\w+\\s-\\s\\d+\"           # Another example pattern\n]\n\n# Attempt to find a matching regex\nmatching_regex = parser.find_matching_regex('example.log', regexes)\nprint(f\"Matching regex: {matching_regex}\")\n```\n\n---\n\n### **4. Preprocess Log Lines**\n- **Description**: Prepares log lines by appending unmatched lines to the previous valid entry to maintain structure.\n\n**Code Location**:\n```python\ndef preprocess_log(self, file_path, regex):\n    processed_lines = []\n    encoding = self.detect_encoding(file_path)\n    current_line = \"\"\n    with open(file_path, encoding=encoding, errors='replace') as f:\n        for line in f:\n            cleaned_line = line.strip()\n            if re.match(regex, cleaned_line):\n                if current_line:\n                    processed_lines.append(current_line)\n                    logging.debug(f\"Processed line: {current_line}\")\n                current_line = cleaned_line\n            else:\n                current_line += \"[new-line]\" + cleaned_line\n                logging.debug(f\"Appended unmatched line: {cleaned_line}\")\n        if current_line:\n            processed_lines.append(current_line)\n            logging.debug(f\"Added final processed line: {current_line}\")\n    logging.info(f\"Preprocessed {len(processed_lines)} lines from '{file_path}'.\")\n    return processed_lines\n```\n\n\n#### **How to Use**:\n```python\npreprocessed_lines = parser.preprocess_log('example.log', r\"(\\w+)\\s-\\s(\\d+)\")\nprint(preprocessed_lines)\n```\n\n#### **Expected Output**:\n```\n['Line 1 - 123', 'Continuation of Line 1[new-line]Line 2 - 456']\n```\n\n\n---\n\n### **5. Parse Lines with Multiple Matches**\n- **Description**: Uses `re.finditer()` to extract all non-overlapping matches in a line and parse them as separate entries.\n\n**Code Location**:\n```python\ndef parse_lines(self, lines, regex):\n    parsed_entries = []\n    for line in lines:\n        logging.debug(f\"Processing line: {line}\")\n        matches = list(re.finditer(regex, line))\n        if matches:\n            logging.debug(f\"Found {len(matches)} matches in line.\")\n            for match in matches:\n                entry = match.groupdict()\n                entry = {k: v.replace(\"[new-line]\", \"\\n\") if isinstance(v, str) else v for k, v in entry.items()}\n                logging.debug(f\"Parsed entry: {entry}\")\n                parsed_entries.append(entry)\n        else:\n            logging.warning(f\"No matches found for line: {line}\")\n    logging.info(f\"Parsed {len(parsed_entries)} entries.\")\n    return parsed_entries\n```\n\n#### **How to Use**:\n```python\nlines = [\n    \"(P3308-T5876)Info ( 247): Event 1.[new-line](P3308-T5876)Debug( 259): Event 2\"\n]\nregex = r\"(?P\u003cprocess\u003eP\\d+-T\\d+)\\s(?P\u003cseverity\u003e\\w+)\\s\\(\\s*\\d+\\):\\s(?P\u003clog\u003e.*?)(?=(\\(P\\d+-T\\d+\\)|$))\"\n\nparsed_entries = parser.parse_lines(lines, regex)\nprint(parsed_entries)\n```\n\n#### **Expected Output**:\n```\n[\n    {'process': 'P3308-T5876', 'severity': 'Info', 'log': 'Event 1.'},\n    {'process': 'P3308-T5876', 'severity': 'Debug', 'log': 'Event 2'}\n]\n```\n\n---\n\n### **6. Create SQLite Database Table**\n- **Description**: Creates a table in the SQLite database if it does not already exist.\n\n**Code Location**:\n```python\ndef create_table(self, conn, table_name, columns):\n    columns_definition = \", \".join([f\"{col} TEXT\" for col in columns])\n    query = f\"CREATE TABLE IF NOT EXISTS {table_name} (id INTEGER PRIMARY KEY AUTOINCREMENT, {columns_definition});\"\n    conn.execute(query)\n    logging.debug(f\"Created table '{table_name}' with columns {columns}\")\n```\n\n#### **How to Use**:\n```python\nconn = sqlite3.connect('logs.db')\ncolumns = ['process', 'severity', 'log']\nparser.create_table(conn, 'log_table', columns)\n```\n#### **Expected Output**:\n\nThe **SQLite table structure** after calling the `create_table` function:\n\n**Table Name:** `log_table`\n\n| **Column**   | **Type** |\n|--------------|-----------|\n| id           | INTEGER (Primary Key) |\n| process      | TEXT      |\n| severity     | TEXT      |\n| log          | TEXT      |\n\n\n\n---\n\n### **7. Save Parsed Data to Database**\n- **Description**: Inserts parsed log entries into the SQLite database.\n\n**Code Location**:\n```python\ndef save_to_db(self, table_name, data, column_order):\n    conn = sqlite3.connect(self.db_path)\n    try:\n        self.create_table(conn, table_name, column_order)\n        placeholders = \", \".join([\"?\"] * len(column_order))\n        query = f\"INSERT INTO {table_name} ({', '.join(column_order)}) VALUES ({placeholders})\"\n        for entry in data:\n            values = [entry.get(col) for col in column_order]\n            conn.execute(query, values)\n            logging.debug(f\"Inserted entry into table '{table_name}': {entry}\")\n        conn.commit()\n        logging.info(f\"Saved {len(data)} entries to table '{table_name}'.\")\n    except sqlite3.Error as e:\n        logging.error(f\"SQLite error: {e}\")\n        raise\n    finally:\n        conn.close()\n```\n\n#### **How to Use**:\n```python\ndata = [\n    {'process': 'P3308-T5876', 'severity': 'Info', 'log': 'Event 1'},\n    {'process': 'P3308-T5876', 'severity': 'Debug', 'log': 'Event 2'}\n]\n\nparser.save_to_db('log_table', data, ['process', 'severity', 'log'])\n```\n#### **Expected Output**:\n\nThe **SQLite table** after calling the `save_to_db` function:\n\n| **ID** | **Process**    | **Severity** | **Log**                                            |\n|--------|----------------|--------------|----------------------------------------------------|\n| 1      | P3308-T5876    | Info         | Event 1                                            |\n| 2      | P3308-T5876    | Debug        | Event 2                                            |\n---\n\n### **8. Process a Single File**\n- **Description**: Processes one log file according to its configuration, including regex matching and database insertion.\n\n**Code Location**:\n```python\ndef process_file(self, config):\n    try:\n        file_path, (table_name, regexes, column_order, stop_on_first_match) = config\n        if not os.path.isfile(file_path):\n            logging.error(f\"File not found: {file_path}\")\n            return\n        regex = self.find_matching_regex(file_path, regexes)\n        processed_lines = self.preprocess_log(file_path, regex)\n        parsed_data = self.parse_lines(processed_lines, regex)\n        self.save_to_db(table_name, parsed_data, column_order)\n    except Exception as e:\n        logging.error(f\"Error processing file '{file_path}': {e}\")\n```\n#### **How to Use**:\n```python\nconfig = ('example.log', ('log_table', [r\"\\w+\\s-\\s\\d+\"], ['process', 'severity', 'log'], True))\nparser.process_file(config)\n```\n\n---\n\n### **9. Load Configuration File**\n- **Description**: Loads configuration settings from a YAML or JSON file.\n\n**Code Location**:\n```python\ndef load_config(self, config_file):\n    config_path = Path(config_file)\n    if not config_path.exists():\n        raise LogParserError(f\"Config file not found: {config_file}\")\n    with open(config_file, 'r') as f:\n        if config_file.endswith(('.yaml', '.yml')):\n            return yaml.safe_load(f)\n        elif config_file.endswith('.json'):\n            return json.load(f)\n        else:\n            raise LogParserError(\"Unsupported config format.\")\n```\n\n#### **How to Use**:\n```yaml\n# config.yaml\nfiles:\n  - file: example.log\n    table: log_table\n    regexes:\n      - \"(P\\\\d+-T\\\\d+)\\\\s(\\\\w+)\\\\s\\\\(\\\\s*\\\\d+\\\\):\\\\s(.*)\"\n    columns:\n      - process\n      - severity\n      - log\n    stop_on_first_match: true\n```\n\n```python\nconfig = parser.load_config('config.yaml')\nprint(config)\n```\n\n---\n\n### **10. Parse Files Using Configuration File**\n- **Description**: Uses the configuration file to parse multiple log files with optional multiprocessing support.\n\n**Code Location**:\n```python\ndef parse_with_config_file(self, config_file, enable_multiprocessing=False):\n    config = self.load_config(config_file)\n    files_with_configs = {\n        entry['file']: (entry['table'], entry['regexes'], entry['columns'], entry.get('stop_on_first_match', True))\n        for entry in config['files']\n    }\n    self.parse_multiple_files(files_with_configs, enable_multiprocessing)\n```\n#### **How to Use**:\n```python\nparser.parse_with_config_file('config.yaml')\n```\n\n\n---\n\n### **11. Multiprocessing Support for Parsing Files**\n- **Description**: Enables faster processing by using multiple CPU cores.\n\n**Code Location**:\n```python\ndef parse_multiple_files(self, files_with_configs, enable_multiprocessing=False):\n    if enable_multiprocessing:\n        with Pool(processes=cpu_count()) as pool:\n            pool.map(self.process_file, files_with_configs.items())\n    else:\n        for config in files_with_configs.items():\n            self.process_file(config)\n```\n\n#### **How to Use**:\n```python\nparser.parse_with_config_file('config.yaml', enable_multiprocessing=True)\n```\n\n---\n\n\n### **Complete Example**\n\n#### **config.yaml**:\n```yaml\nfiles:\n  - file: example.log\n    table: log_table\n    regexes:\n      - \"(P\\\\d+-T\\\\d+)\\\\s(\\\\w+)\\\\s\\\\(\\\\s*\\\\d+\\\\):\\\\s(.*)\"\n    columns:\n      - process\n      - severity\n      - log\n    stop_on_first_match: true\n```\n\n#### **main.py**:\n```python\nfrom parser import LogParser\n\n# Initialize the parser\nparser = LogParser(db_path='logs.db', log_level='DEBUG')\n\n# Use config file for parsing\nparser.parse_with_config_file('config.yaml')\n```\n\n#### **Expected Output**:\n1. Log messages printed in the console/log file.\n2. Entries saved to the SQLite database.\n\n\n---\n## Documentation for `parser_cli.py`\n\n`parser_cli.py` is the command-line interface (CLI) for `LogxDB`, a log file parsing utility. This script allows users to process multiple log files, apply custom regular expressions (regexes), store parsed data in a SQLite database, and configure the parsing process via command-line arguments or configuration files (YAML/JSON).\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Features](#features)\n- [Installation](#installation)\n- [Usage](#usage)\n  - [1. Run Regex REPL](#1-run-regex-repl)\n  - [2. Parse Logs with Config Files](#2-parse-logs-with-config-files)\n  - [3. Parse Logs with CLI Arguments](#3-parse-logs-with-cli-arguments)\n  - [4. Logging and Error Handling](#4-logging-and-error-handling)\n  - [5. Multiprocessing Support](#5-multiprocessing-support)\n- [Code Examples](#code-examples)\n- [Commands Summary](#commands-summary)\n\n---\n\n## **Overview**\n\n`parser_cli.py` provides a flexible interface for parsing log files through the command line. It supports reading custom regular expressions from YAML/JSON configuration files or directly via command-line arguments. Additionally, it offers features like multiprocessing for faster log processing, logging to files, and an interactive regex testing mode.\n\n---\n\n## **Features**\n\n- **Interactive Regex REPL**: Test and evaluate regex patterns interactively.\n- **Configuration File Support**: Use YAML or JSON files to define log files, regexes, and table structure.\n- **Command-Line Parsing**: Pass log files, regex patterns, and database details directly via CLI.\n- **Multiprocessing Support**: Enable or disable multiprocessing to process logs faster using multiple CPU cores.\n- **Logging**: Customizable logging with multiple levels (DEBUG, INFO, WARNING, ERROR, CRITICAL) and file logging support.\n- **Error Handling**: Handles unexpected errors, missing files, and invalid configurations with meaningful messages.\n\n---\n\n## **Installation**\n\nEnsure you have Python 3.6+ installed on your machine. Install the required dependencies:\n\n```bash\npip install -r requirements.txt\n```\n\nThe required libraries include:\n- `argparse` for command-line argument parsing\n- `re` for regular expressions\n- `yaml` for YAML file handling\n- `json` for JSON file handling\n- `sqlite3` for database interaction\n- `logging` for error tracking and debugging\n\n---\n\n## **Usage**\n\n`parser_cli.py` can be used in several ways depending on the requirements. Below are usage examples for common scenarios.\n\n### **1. Run Regex REPL**\n\nThe Regex REPL allows you to interactively test regex patterns and evaluate their output on test strings.\n\n#### **Command:**\n```bash\npython parser_cli.py --repl\n```\n\n#### **Example Interaction:**\n```\nWelcome to the Regex Tester REPL! Type 'exit' to quit.\nEnter regex pattern: (\\d{4}-\\d{2}-\\d{2})\nEnter test string: 2024-10-23\nMatch found: {'0': '2024-10-23'}\n```\n\n---\n\n### **2. Parse Logs with Config Files**\n\nYou can configure the parsing process using YAML or JSON files. These config files allow you to specify the log files, regexes, table names, and columns to be stored in the SQLite database.\n\n#### **Command:**\n```bash\npython parser_cli.py --config config.yaml\n```\n\n#### **YAML Example:**\n```yaml\nfiles:\n  - file: \"example.log\"\n    table: \"example\"\n    regexes: \n      - \"(P\\\\d+-T\\\\d+).*?(Info|Debug).*\"\n    columns:\n      - process\n      - date\n      - time\n      - log\n```\n\n#### **JSON Example:**\n```json\n{\n  \"files\": [\n    {\n      \"file\": \"example.log\",\n      \"table\": \"example\",\n      \"regexes\": [\"(P\\\\d+-T\\\\d+).*?(Info|Debug).*\"],\n      \"columns\": [\"process\", \"date\", \"time\", \"log\"]\n    }\n  ]\n}\n```\n\n---\n\n### **3. Parse Logs with CLI Arguments**\n\nInstead of using config files, you can specify the log files, regex patterns, and database columns directly via CLI.\n\n#### **Command:**\n```bash\npython parser_cli.py --db-path logs.db \\\n  --files example.log \\\n  --regexes \"(P\\\\d+-T\\\\d+).*?(Info|Debug).*\" \\\n  --tables example \\\n  --columns \"process,date,time,log\" \\\n  --log-level DEBUG\n```\n\nThis command parses the `example.log` file using the provided regex pattern, stores the parsed data in the SQLite database at `logs.db`, and logs debug information.\n\n---\n\n### **4. Logging and Error Handling**\n\n`parser_cli.py` supports customizable logging levels and outputs logs to files.\n\n#### **Command:**\n```bash\npython parser_cli.py --db-path logs.db \\\n  --files example.log \\\n  --regexes \"(P\\\\d+-T\\\\d+).*?(Info|Debug).*\" \\\n  --tables example \\\n  --columns \"process,date,time,log\" \\\n  --log-level INFO \\\n  --log-file parser.log\n```\n\nThis example logs all information at `INFO` level and stores logs in `parser.log`. If an error occurs (e.g., missing files or incorrect regex), an appropriate error message will be logged and printed to the console.\n\n---\n\n### **5. Multiprocessing Support**\n\nTo speed up parsing for multiple files, enable multiprocessing by passing the `--multiprocessing` flag.\n\n#### **Command:**\n```bash\npython parser_cli.py --config config.yaml --multiprocessing\n```\n\nThis uses all available CPU cores to process multiple log files in parallel.\n\n---\n\n## **Code Examples**\n\n### **Regex REPL Example**\n```bash\npython parser_cli.py --repl\n```\nOutput:\n```\nWelcome to the Regex Tester REPL! Type 'exit' to quit.\nEnter regex pattern: (\\d{4}-\\d{2}-\\d{2})\nEnter test string: 2024-10-23\nMatch found: {'0': '2024-10-23'}\n```\n\n### **Parse Logs Using Config File**\n```bash\npython parser_cli.py --config config.yaml\n```\nThe `config.yaml` file specifies log files and regex patterns. Example:\n```yaml\nfiles:\n  - file: \"example.log\"\n    table: \"example\"\n    regexes: \n      - \"(P\\\\d+-T\\\\d+).*?(Info|Debug).*\"\n    columns:\n      - process\n      - date\n      - time\n      - log\n```\n\n### **Parse Logs Using Command-Line Arguments**\n```bash\npython parser_cli.py --db-path logs.db \\\n  --files example.log \\\n  --regexes \"(P\\\\d+-T\\\\d+).*?(Info|Debug).*\" \\\n  --tables example \\\n  --columns \"process,date,time,log\"\n```\n\n### **Multiprocessing Example**\n```bash\npython parser_cli.py --config config.yaml --multiprocessing\n```\n\n---\n\n## **Commands Summary**\n\n- `--repl`: Launch the interactive Regex Tester REPL.\n- `--config \u003cconfig_file\u003e`: Specify a YAML or JSON configuration file.\n- `--db-path \u003cdb_path\u003e`: Specify the path to the SQLite database.\n- `--files \u003cfile1\u003e \u003cfile2\u003e`: List the log files to parse.\n- `--regexes \u003cregex1\u003e \u003cregex2\u003e`: List the regex patterns for each file.\n- `--tables \u003ctable1\u003e \u003ctable2\u003e`: List the table names for storing parsed data.\n- `--columns \u003ccols1\u003e \u003ccols2\u003e`: List the columns for each table.\n- `--multiprocessing`: Enable multiprocessing for faster log parsing.\n- `--log-level \u003clevel\u003e`: Set the logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL).\n- `--log-file \u003clog_file\u003e`: Specify a log file to store the logs.\n\n---\n\n\n## Contributing\n\n1. **Fork the repository**.\n2. **Create a new branch**:\n   ```bash\n   git checkout -b feature-branch\n   ```\n3. **Commit your changes**:\n   ```bash\n   git commit -am 'Add new feature'\n   ```\n4. **Push to the branch**:\n   ```bash\n   git push origin feature-branch\n   ```\n5. **Open a pull request**.\n\n\n---\n\n## License\n\nThis project is licensed under the MIT License. See the `LICENSE` file for more details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaimemartinez%2Flogxdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjaimemartinez%2Flogxdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaimemartinez%2Flogxdb/lists"}