https://github.com/micromark/micromark

small, safe, and great commonmark (optionally gfm, mdx) compliant markdown parser

ast commonmark compile cst gfm markdown parse render tokenize unified

Last synced: 13 May 2025

https://github.com/winkjs/wink-nlp

Developer friendly Natural Language Processing ✨

bm25 chatbot custom-entity-detection hacktoberfest named-entity-extraction natural-language-processing negation-handling ner nlp pattern-matching pos-tagging sbd sentence-boundary-detection sentiment-analysis tokenize vectorizer visualization wink wink-nlp word-vectors

Last synced: 13 May 2025

https://github.com/wooorm/markdown-rs

CommonMark compliant markdown parser in Rust with ASTs and extensions

commonmark compiler gfm markdown parse render rust tokenize

Last synced: 13 May 2025

https://github.com/alasdairforsythe/tokenmonster

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript

text-tokenization tokenisation tokenization tokenize tokenizer tokenizing vocabulary vocabulary-builder vocabulary-generator

Last synced: 16 Jan 2026

https://github.com/syntax-tree/mdast-util-from-markdown

mdast utility to parse markdown

markdown mdast mdast-util parse tokenize unist

Last synced: 14 May 2025

https://github.com/here-be/snapdragon

snapdragon is an extremely pluggable, powerful and easy-to-use parser-renderer factory.

ast compile compiler javascript lex lexer node nodejs parse parser render source-map token tokenize

Last synced: 15 May 2025

https://github.com/winkjs/wink-nlp-utils

NLP Functions for amplifying negations, managing elisions, creating ngrams, stems, phonetic codes to tokens and more.

bag-of-words natural-language-processing ngrams nlp phonetize sentence-boundary-detection stem stop-words tokenize

Last synced: 06 Apr 2025

https://github.com/privateai/deid-examples

Examples scripts that showcase how to use Private AI Text to de-identify, redact, hash, tokenize, mask and synthesize PII in text.

anonymization cpra data-anonymization data-loss-prevention data-masking de-identification de-identify deidentification dlp gdpr hipaa masking privacy-tools redact redaction synthetic-data synthetic-data-generator synthetic-dataset-generation text-anonymization tokenize

Last synced: 07 Apr 2026

https://github.com/jonschlinkert/extract-comments

Extract JavaScript code comments from a string or glob of files.

code-comments comments esprima extract javascript jonschlinkert nodejs parse tokenize

Last synced: 07 May 2025

https://github.com/chayanforyou/bkash-pgwclient-demo-flutter

bKash payment gateway integration in flutter

bkash bkash-payment bkash-payment-gateway bkash-pgw flutter tokenize

Last synced: 31 Jul 2025

https://github.com/hexydec/htmldoc

A token based HTML Document parser and minifier written in PHP. Extract attribute values and text using CSS selectors.

html html-dom-parser html-parser html5 minification minify minify-html php simplehtmldom svg tokenize tokenizer

Last synced: 02 Apr 2026

https://github.com/begin/parsers-compilers

Lexers, tokenizers, parsers, compilers, renderers, stringifiers... What's the difference, and how do they work?

ast compiler guide lexer node parse parsers-compilers syntax-tree token token-stream tokenize

Last synced: 01 Apr 2025

https://github.com/jonschlinkert/babel-extract-comments

Uses babel to extract JavaScript code comments from a string. Returns an array of comment objects, with line, column, index, comment type and comment string.

babel comments extract javascript jonschlinkert nodejs parse tokenize

Last synced: 15 Apr 2025

https://github.com/jonschlinkert/tokenize-comment

Uses snapdragon to tokenize a single JavaScript block comment into an object, with description, tags, and code example sections that can be passed to any other comment parsers for further parsing.

code code-comment comment comment-parser gfm javadoc javascript jonschlinkert js jsdoc node nodejs parse parse-comments tokenize

Last synced: 06 May 2025

https://github.com/akb89/witokit

A Python toolkit to generate a tokenized dump of Wikipedia for NLP

dump multilingual nlp tokenize wikipedia wikipedia-dump

Last synced: 28 Apr 2025

https://github.com/asmeurer/brown-water-python

More detailed documentation for the Python tokenize module

documentation python python3 tokenize

Last synced: 01 Sep 2025

https://github.com/myint/untokenize

Transforms tokens into original source code (while preserving whitespace)

python tokenize untokennize

Last synced: 26 Apr 2025

https://github.com/here-be/snapdragon-scanner

Easily scan a string with an object of regex patterns to produce an array of tokens. ~100 sloc.

lex lexer parse scan scanner snapdragon string token tokenize tokenizer

Last synced: 12 Apr 2025

https://github.com/carlosplanchon/tokenizesentences

Python3 module to tokenize english sentences.

carlosplanchon opensource python python3 sentences tokenize

Last synced: 15 Sep 2025

https://github.com/paceaux/methodius

A utility for analyzing text on the web

bigram ngram parse split tokenize trigram

Last synced: 26 Jan 2026

https://github.com/queckezz/json-tokenize

Splits a JSON string into an annotated list of tokens

json lexer lexical-analysis tokenize tokenizer tokens

Last synced: 15 May 2025

https://github.com/here-be/snapdragon-stack

Snapdragon utility for creating a stack.

array nodes parse parser snapdragon stack tokenize tokens

Last synced: 03 Aug 2025

https://github.com/flex-development/mark-parser

finite state machine tokenizer

fsm parse state syntax tokenize

Last synced: 24 Jul 2026

https://github.com/linguistic-dev/n-gram-extractor

A PHP Library to extract n-grams from a text. Simple preprocessing tools (cleaning, tokenizing) included.

natural-language-processing ngram ngram-analysis ngrams nlp php php-library php7 tokenization tokenize tokenized-sentences tokenizer

Last synced: 22 Jul 2025

https://github.com/hamedzarei/nlp-simple-punctuation-correction

simple regex for correcting punctuations

nlp normalization punctuation python regex sentence-parser tokenize tokenizer

Last synced: 29 Jun 2026

https://github.com/bent10/stophtml

Extracts plain text from an HTML string. It's useful for Natural Language Processing (NLP) tasks.

html nlp plaintext strip text token tokenize

Last synced: 10 Apr 2025

https://github.com/bent10/attributes-parser

Tokenize and parse attributes string into meaningful tokens and key-value pairs.

attr attribute attributes attrs html markdown object prop properties property props serialize tokenize tokens

Last synced: 23 Jun 2025

https://github.com/here-be/snapdragon-token

Create a snapdragon token. Used by the snapdragon lexer, but can also be used by plugins.

compiler javascript lexer node nodejs parser snapdragon token tokenize tokenizer

Last synced: 15 Jul 2025

https://github.com/xpepermint/ngramablejs

String ngram splitter.

convert converter ngram parse parser search split tokenize tokenizer

Last synced: 29 Mar 2025

https://github.com/flex-development/fsm-tokenizer

finite state machine tokenizer

finite-state-machine fsm lexer parse state tokenize tokenizer

Last synced: 03 Apr 2025

https://github.com/hexacta/tokenizer

Split text into tokens.

browser hexacta-innovation-labs join nlp node split text tokenize tokenizer tokens words

Last synced: 16 Jan 2026

https://github.com/flex-development/vfile-tokenizer

vfile utility to tokenize a file

lexer parse tokenize vfile vfile-util

Last synced: 23 Mar 2025

https://github.com/basemax/go-lexer-token-simple

Simple Go lexer: Lex own syntax and read it's from file.

go golang lexer lexer-analyzer lexer-example lexers scanner token tokenization tokenize tokenizer tokens

Last synced: 16 Jul 2026

https://github.com/here-be/snapdragon-location

Adds a location object to snapdragon token or AST node.

ast javascript location node nodejs parse parser position snapdragon token tokenize tokenizer

Last synced: 12 Apr 2025

https://github.com/bent10/nomark

Transform hypertext strings (e.g., HTML, Markdown) into plain text for natural language processing (NLP) normalization

html markdown nlp normalize normalizer plaintext text token tokenize transform transformer

Last synced: 05 Feb 2026

https://github.com/seemueller-io/toak

instantly tokenize a git repository

cli code code-assistant dev markdown prompt repository tokenize

Last synced: 11 Jun 2025

https://github.com/lensvol/tokelor

Visualize Python token stream produced by tokenize module.

python tokenize

Last synced: 13 Jun 2026

https://github.com/openvoiceos/quebra_frases

chunks strings into byte sized pieces

chunk chunking sentence-chunking tokenization tokenize tokenized tokenizer word-tokenizing

Last synced: 12 Mar 2026

https://github.com/bent10/stopmarkdown

Extracts plain text from Markdown strings. It's useful for Natural Language Processing (NLP) tasks.

markdown nlp plaintext strip text token tokenize

Last synced: 10 Apr 2025

https://github.com/fcf-framework/fcf-framework-core

A package of basic functions and classes required for the framework to work

core fcf framework javascript nodejs tokenize

Last synced: 04 May 2026

https://github.com/icai/token-sort

Token(ize) Sort, also support weight sort

sort token tokenize

Last synced: 15 Jun 2025

https://github.com/flex-development/mark

a specification for building syntax-aware parsers, compilers, and text processing systems

compile fsm parse spec state syntax tokenize

Last synced: 24 Jul 2026

https://github.com/mugendi/wordize

consonant-blends natural-language-processing nlp tokenize words

Last synced: 19 Jul 2025

https://github.com/flex-development/docast-util-from-docs

docast utility to parse docblocks

comment docast docblock documentation jsdoc markdown mdast parse tokenize tsdoc typedoc typescript unist

Last synced: 02 Jan 2026

https://github.com/clitetailor/tokenize-monster

Yet another powerful tokenizer in js.

functional-programming tokenize tokenizer

Last synced: 02 Apr 2025

https://github.com/ussarmy/wink

A tool to Win the week

aws bm25 chatbot codechef hackerearth materializecss python sbd spoj stopstalk timus tokenize web2py wink-nlp

Last synced: 28 Mar 2025

https://github.com/kopcho/sales180

PKH Events

bitangels tokenize

Last synced: 08 Mar 2026

https://github.com/aminya/tokenizemeta.jl

Utilities for evaluated Tokens

cst cstparser eval julia parse parser token tokenize tokenizer

Last synced: 04 Apr 2025

https://github.com/rryi/tokens.jl

parse text into tokens, build memory-efficient token lists and trees,

lexer mutable-strings parser parsing string string-manipulation string-tree string-vector token tokenize

Last synced: 02 Mar 2026

https://github.com/siddiqsoft/string2map

Header only C++17 library to parse a string containing delimited key-value pairs into a map container including the feature to convert from std::string to std::wstring and vice-versa.

cpp http-header-parser key-value map parse string tokenize winhttp

Last synced: 23 Apr 2026

https://github.com/poyo46/jadoc

Tokenizes Japanese documents to enable CRUD operations.

conjugate crud inflect japanese mecab nlp tokenize

Last synced: 08 Apr 2026