{"id":32619176,"url":"https://github.com/fnl/tokenizer","last_synced_at":"2026-07-09T07:31:07.692Z","repository":{"id":12498130,"uuid":"15167707","full_name":"fnl/tokenizer","owner":"fnl","description":"a concurrent, deterministic finite state tokenizer (for letter-based scripts)","archived":false,"fork":false,"pushed_at":"2013-12-17T21:19:40.000Z","size":144,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-03-11T13:18:37.263Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fnl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-12-13T16:00:32.000Z","updated_at":"2018-10-06T03:20:28.000Z","dependencies_parsed_at":"2022-09-24T01:10:26.667Z","dependency_job_id":null,"html_url":"https://github.com/fnl/tokenizer","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"purl":"pkg:github/fnl/tokenizer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fnl%2Ftokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fnl%2Ftokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fnl%2Ftokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fnl%2Ftokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fnl","download_url":"https://codeload.github.com/fnl/tokenizer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fnl%2Ftokenizer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35291639,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-09T02:00:07.329Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-30T18:00:35.514Z","updated_at":"2026-07-09T07:31:07.687Z","avatar_url":"https://github.com/fnl.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tokenizer\n    import \"github.com/fnl/tokenizer\"\n\nA concurrent, deterministic finite state tokenizer\nfor Latin scripts that separates strings into words, numbers, and symbols.\nWords may be connected by dashes, underscores, and dots\n(but cannot start or end with those characters).\nNumbers may contain commas between digits,\nbut only before the comma-separator (a dot).\nAnything else not a linebreak, control character,\nor space is considered a symbol.\nSymbol tokens are always single (\"one-character\") runes.\n\nOptions:\n\n1. Normalization of quotes and dashes;\n2. Tokenization of spaces and/or linebreaks;\n3. Lowering the case of all words;\n4. Unescaping of HTML entities;\n5. Expansion of Greek letters.\n\nIn addition, a command-line tokenizer is provided as `fnltok`:\n\n\tgo install github.com/fnl/tokenizer/fnltok\n\nUsage:\n\n\tfnltok [options] [TEXTFILE ...]\n\n`fnltok` is a high-throughput, line-based command-line interface\nfor the tokenizer that writes the tokens to `STDOUT`.\nThis script is about 100 times faster than an equivalent Perl tokenizer\nusing regular expressions for the same task.\nIt can tokenize input based on lines and/or tab-separated values\n(while preserving the tabs).\nThe latter is useful to tokenize text in tabulated data files.\nBecause file I/O soon becomes the main bottleneck,\nhaving more than two or three parallel tokenizer processes (`$GOMAXPROCS`)\ndoes not improve its speed any further.\n\n## Synopsis\n\n\t// create an input channel for tokenization:\n\tin := make(chan string)\n\t\n\t// start the tokenizer;\n\t// returns an output channel of tokens:\n\tout := Lex(in, 100, AllOptions)\n\t\n\t// a semaphore to synchronize downstream results\n\tsemaphore := make(chan int)\n\t\n\t// somehow concurrently process the tokens...\n\tgo processTokens(out, semaphore)\n\t\n\t// send data to the tokenizer\n\tfor data := range myInput {\n\t  in \u003c- data\n\t}\n\t// and close the input stream once done\n\tclose(in)\n\t\n\t// wait for the output processing to complete\n\t\u003c-semaphore\n\n## func Lex\n\u003cpre\u003efunc Lex(input chan string, outputBufferSize int, options Option) chan Token\u003c/pre\u003e\nLex starts a scanner process to lex string input,\nreturning a Token output channel.\nThe outputBufferSize is the buffer size\nthat should be used to create the output channel.\n\nThe scanner waits for strings to lex on the input channel.\nAfter lexing, it sends the found tokens back via the output channel.\nAfter all tokens have been output for a given input, an EndToken is sent.\nIf the input channel is closed,\nthe output channel closes after the last (End) token has been emitted.\n\nPossible options; combine Option values by or-ing (\"|\"):\n\n\tSpaces:\n\t  emit space tokens.\n\tLinebreak:\n\t  emit tokens containing EOLMarkers.\n\tEntities:\n\t  resolve and replace HTML entities (/\u0026\\w+;/).\n\tQuotes:\n\t  replace two single with one double quote and\n\t  U+02BC (modifier apostrophe) with U+0027 (\"'\" - apostrophe).\n\tLowercase:\n\t  lower-case all words.\n\tGreek:\n\t  expand Greek letters to their upper-/lower-case Latin names.\n\tNoOptions:\n\t  use none of the options (the zero value default).\n\tAllOptions:\n\t  use all of the options.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffnl%2Ftokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffnl%2Ftokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffnl%2Ftokenizer/lists"}