Projects in Awesome Lists tagged with tokenizer-parser

https://github.com/risesoft-y9/data-labeling

数据标注是一款专门对文本数据进行处理和标注的工具，通过简化快捷的文本标注流程和动态的算法反馈，支持用户快速标注关键词并能通过算法持续减少人工标注的成本和时间。数据标注的过程先由人工标注构建基础，再由自动标注反哺人工标注，最后由人工标注进行纠偏，从而大幅度提高标注的精准度和高效性。数据标注需要依赖开源的数字底座进行人员岗位管控。

chinese data-annotation-tools data-annotations docker elasticsearch java nacos springboot2 tokenizer tokenizer-parser vue3

Last synced: 15 May 2025

https://github.com/ikskuh/parser-toolkit

A toolkit that makes it easier to write recursive-descent parsers in Zig.

compiler compiler-frontend parser recursive-descent-parser tokenizer tokenizer-parser zig zig-package ziglang

Last synced: 02 Sep 2025

https://github.com/ehwan/c-language-parser-in-rust

C language lexer & parser & virtual interpreter from scratch in Rust

abstract-syntax-tree c code-generation compiler interpreter lexer parser rust rust-lang tokenizer tokenizer-parser virtual-machine

Last synced: 16 Mar 2025

https://github.com/recp/xml

🔋 In-place lightweight XML parser

c-xml fast-xml header-only in-place-parser lightweight-xml optimized-xml parser tokenizer tokenizer-parser xml xml-loader xml-parser

Last synced: 11 Jun 2025

https://github.com/melchisedech333/antlr4-experiments

:wrench: My studies on context-free grammar, using ANTLR4 (C++) to generate the parser files. Some basics are developed, such as token processing, recursion, variable definition, array processing, Abstract Syntax Tree (AST) manipulation, UNICODE support, and error handling.

antlr-language-development antlr4 antlr4-grammar grammar-checker grammar-parser grammar-parser-generator grammar-rules grammars grammars-utility language-development language-processing semantic-analysis semantics syntax syntax-analysis syntax-analyzer syntax-tree tokenization tokenizer tokenizer-parser

Last synced: 11 Apr 2025

https://github.com/bytexenon/luaxen

Lua Compiler, (De)Obfuscator, Minifier, Beautifier, And more

assembler assembly assembly-language beautifier decompiler interpreter lexer lua lua51 minifier obfuscator parser programming-language pseudo-assembler pseudo-assembly tokenizer tokenizer-parser virtual-machine virtualmachine vm

Last synced: 14 May 2025

https://github.com/melchisedech333/lex-yacc-experiments

:wrench: My studies involving context-free grammar analysis. The analyzers were built using familiar tools such as YACC, Lex and Bison. Topics covered include token filtering, simple variable manipulation, and arrays.

bison bison-grammar bison-parser bison-yacc compiler-development language-development lex lex-example lex-yacc-parser lexical-analysis lexical-analyzer parser-generator parsing semantic-analysis syntactic-analysis tokenization tokenizer-parser yacc yacc-example yacc-lex

Last synced: 07 Feb 2026

https://github.com/rajspeaks/machine-learning-approach-to-bengali-corpus-tokenization-stemming-pos-tagging-using-bnltk

Machine Learning approach to Bengali Corpus POS Tagging using BNLTK. This is an experimenting project under the mentorship of Prof. Sandipan Ganguly, HIT-K.

bengali bengali-dataset bengali-language-processing bengali-natural-language-processing bengali-nlp english machine-learning natural-language-processing natural-language-understanding nlp nlp-library nlp-machine-learning postagger postagging rajdeep-das rajspeaks stemmer stemming tokenizer-parser

Last synced: 04 Apr 2025

https://github.com/mitchellgray100/critter-world-readme

A README for my private CS 2112 Critter World Project

game-development java javafx parser readme simulation-game threading tokenizer-parser

Last synced: 26 Mar 2025

https://github.com/vxern/text_expressions

A tiny and complete tool to supercharge static JSON strings with dynamic, user-defined expressions.

expression expression-evaluator language lexer lexer-parser parser tokenizer tokenizer-parser translation udf user-defined-expressions

Last synced: 11 Jan 2026

https://github.com/duffsdevice/tiny-parser

Write use-case specific parsers within minutes!

context-free-grammar parser parser-generator parser-library tokenizer tokenizer-parser

Last synced: 24 Aug 2025

https://github.com/daireto/odata-v4-query

A lightweight, simple and fast parser for OData V4 query options supporting standard query parameters. Provides helper functions to apply OData V4 query options to ORM/ODM queries such as SQLAlchemy and Beanie.

beanie-odm odata odata-query-parser odatav4 pymongo sqlalchemy tokenizer-parser

Last synced: 14 Jan 2026

https://github.com/melchisedech333/antlr4-demonstration-project

:wrench: Demonstration of using ANTLR4 (with runtime for C++) in projects for context-free grammar processing. The ANTLR4 (Java) package is included, and the project is configured to compile on Linux.

antlr antlr-development-environment antlr-grammar antlr4-grammar grammar-checker grammar-parser grammar-rules grammars grammars-utility language-development language-processors semantic-analysis semantics syntax syntax-analysis syntax-analyzer syntax-tree tokenization tokenizer tokenizer-parser