{"id":34862400,"url":"https://github.com/bokic/textparser","last_synced_at":"2026-05-15T16:33:44.469Z","repository":{"id":216518774,"uuid":"714368060","full_name":"bokic/textparser","owner":"bokic","description":"TextParser is a high-performance C library that parses text(CFML and JSON for now) into Abstract Syntax Trees using regex grammars, designed for building syntax highlighters, language servers, as well as other code related tools.","archived":false,"fork":false,"pushed_at":"2026-04-04T23:06:02.000Z","size":515,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-04-05T01:17:38.953Z","etag":null,"topics":["ast","ast-tree","json","parser","pcre2","tokenization"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bokic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-11-04T17:44:04.000Z","updated_at":"2026-04-04T23:06:07.000Z","dependencies_parsed_at":"2025-03-15T19:27:20.230Z","dependency_job_id":"c3c047a2-6bd6-4ab2-8d27-65400e5d7e84","html_url":"https://github.com/bokic/textparser","commit_stats":null,"previous_names":["bokic/textparser"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/bokic/textparser","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bokic%2Ftextparser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bokic%2Ftextparser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bokic%2Ftextparser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bokic%2Ftextparser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bokic","download_url":"https://codeload.github.com/bokic/textparser/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bokic%2Ftextparser/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33072447,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-15T11:35:32.926Z","status":"ssl_error","status_checked_at":"2026-05-15T11:35:31.362Z","response_time":103,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ast","ast-tree","json","parser","pcre2","tokenization"],"created_at":"2025-12-25T21:25:20.311Z","updated_at":"2026-05-15T16:33:44.464Z","avatar_url":"https://github.com/bokic.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TextParser [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/bokic/textparser)\n\nTextParser is a high-performance, extensible text parsing library written in C. It uses regular expressions to define language grammars and generates a hierarchical Abstract Syntax Tree (AST) for parsed documents.\n\nThe project currently provides robust support for CFML (ColdFusion Markup Language) and JSON, with a flexible architecture allows for easy addition of new language definitions.\n\n## Features\n\n- **High Performance**: Written in optimized C for fast parsing of large codebases.\n- **Small Footprint**: The library is designed to be small and easy to integrate into other projects.\n- **Minimal Dependencies**: The library has minimal dependencies (only crpe2 library for regex matching).\n- **Regex-Based Grammars**: Define language syntax using flexible regular expressions.\n- **Hierarchical AST**: Generates a structured tree of tokens (`textparser_token_item`) representing the code structure.\n- **Syntax Highlighting Support**: Tokens track metadata like color, background, and flags, making it suitable for building syntax highlighters and editors.\n- **Extensibility**: Language definitions are decoupled from the core parsing logic, constructed with JSON, and can be loaded at compile time (by generated header file) or at runtime (by loading JSON file).\n- **Python Tooling**: Includes Python scripts for: prototyping and validation of the core algorithm, generation of C header files, and other parser verification tools.\n\n## Project Structure\n\n- **`src/`**: Core C library implementation (`textparser.c`, `adv_regex.c`).\n- **`include/`**: Public header files (`textparser.h`).\n- **`cli/`**: Command-line tool mainly for testing and demonstrating the library.\n- **`definitions/`**: Language definitions (e.g., CFML, JSON).\n- **`python/`**: Python bindings, prototypes, and validation tools (`validate_cfml.py`).\n- **`tests/`**: Unit and integration tests.\n- **`ccat/`**: Utilities for text processing (e.g., color cat).\n\n## Build Instructions\n\n### Prerequisites\n\n- CMake (version 3.15 or higher)\n- Ninja build system\n- A C compiler (GCC/Clang)\n\n### Building\n\nYou can use the provided build script for a quick start:\n\n```bash\n./build.sh\n```\n\nAlternatively, you can build using standard CMake commands:\n\n```bash\ncmake -B build -G Ninja\ncmake --build build\n```\n\nArtifacts (libraries and executables) will be output to the `bin/` directory.\n\n## Installation\n\n### Arch Linux\n\n`textparser` is available on the Arch User Repository (AUR). You can install it using an AUR helper like `yay`:\n\n```bash\nyay -S textparser\n```\n\nOr view the package details at [https://aur.archlinux.org/packages/textparser](https://aur.archlinux.org/packages/textparser).\n\n## Usage\n\n### CLI Tool\n\nThe `textparser` CLI tool can be used to parse files and visualize the resulting token tree.\n\n```bash\nbin/textparser path/to/file.cfm\n```\n\n### C Library Integration\n\nTo use TextParser in your C project, include `textparser.h` and link against `libtextparser`.\n\n**Basic Example:**\n\n```c\n#include \u003ctextparser.h\u003e\n#include \u003cstdio.h\u003e\n\n// Assume 'my_lang_definition' is defined elsewhere\nextern const textparser_language_definition my_lang_definition;\n\nint main() {\n    textparser_defer(handle); // Auto-cleanup\n\n    // Open a file\n    int err = textparser_openfile(\"example.txt\", TEXTPARSER_ENCODING_LATIN1, \u0026handle);\n    if (err) {\n        fprintf(stderr, \"Failed to open file\\n\");\n        return 1;\n    }\n\n    // Parse using the language definition\n    err = textparser_parse(handle, \u0026my_lang_definition);\n    if (err) {\n        fprintf(stderr, \"Parse error\\n\");\n        return 1;\n    }\n\n    // Iterate through tokens\n    for (textparser_token_item *item = textparser_get_first_token(handle); item != NULL; item = item-\u003enext) {\n        // ... process item ...\n    }\n    \n    return 0;\n}\n```\n\n## Language Definition Example\n\nTextParser uses a JSON-based format to define language grammars. This allows for defining complex syntax rules using regular expressions and hierarchical token structures.\n\nHere is a simplified example of what a JSON definition might look like (based on `definitions/json_definition.json`):\n\n```json\n{\n  \"name\": \"json\",\n  \"version\": 1.0,\n  \"startTokens\": [\"Object\", \"Array\"],\n  \"tokens\": {\n    \"Object\": {\n      \"type\": \"StartStop\",\n      \"startRegex\": \"{\",\n      \"endRegex\": \"}\",\n      \"textColor\": \"0xffd700\",\n      \"nestedTokens\": [\"Key\", \"String\", \"Number\", \"ValueSeparator\"]\n    },\n    \"String\": {\n      \"type\": \"StartStop\",\n      \"startRegex\": \"\\\"\",\n      \"endRegex\": \"\\\"\",\n      \"textColor\": \"0xce9178\",\n      \"nestedTokens\": [\"StringEscape\"]\n    },\n    \"Number\": {\n      \"type\": \"SimpleToken\",\n      \"startRegex\": \"-?\\\\d+(?:\\\\.\\\\d+)?\",\n      \"textColor\": \"0xb5cea8\"\n    }\n  }\n}\n```\n\n## Development and Verification\n\nThe `python/` directory contains tools for verifying the parser's correctness, particularly for CFML.\n\n- **`validate_cfml.py`**: A robust validation script that compares the AST generated by this project against reference parsers (e.g., a Java-based CFML parser) to ensuring high fidelity and correctness.\n\n```bash\npython3 python/validate_cfml.py /path/to/cfml/files\n```\n\n## License\n\nSee `LICENSE` file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbokic%2Ftextparser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbokic%2Ftextparser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbokic%2Ftextparser/lists"}