{"id":20278525,"url":"https://github.com/wmentor/tokens","last_synced_at":"2025-03-04T01:44:37.804Z","repository":{"id":57544497,"uuid":"279112550","full_name":"wmentor/tokens","owner":"wmentor","description":"Text to tokes","archived":false,"fork":false,"pushed_at":"2023-12-26T18:58:45.000Z","size":28,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-14T06:54:13.023Z","etag":null,"topics":["go","golang","golang-library","text-processing","text-processor","tokenizer","tokens"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wmentor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-12T17:18:59.000Z","updated_at":"2022-08-22T19:46:38.000Z","dependencies_parsed_at":"2024-11-14T13:24:06.619Z","dependency_job_id":"b34ac909-99f0-4365-9108-f85a6f74a28e","html_url":"https://github.com/wmentor/tokens","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wmentor%2Ftokens","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wmentor%2Ftokens/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wmentor%2Ftokens/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wmentor%2Ftokens/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wmentor","download_url":"https://codeload.github.com/wmentor/tokens/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241768283,"owners_count":20017129,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","golang-library","text-processing","text-processor","tokenizer","tokens"],"created_at":"2024-11-14T13:23:53.444Z","updated_at":"2025-03-04T01:44:37.784Z","avatar_url":"https://github.com/wmentor.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tokens\n\n![test](https://github.com/wmentor/tokens/workflows/test/badge.svg)\n[![Coverage Status](https://coveralls.io/repos/github/wmentor/tokens/badge.svg?branch=master\u0026v=1.0.4)](https://coveralls.io/github/wmentor/tokens?branch=master)\n[![https://goreportcard.com/report/github.com/wmentor/tokens](https://goreportcard.com/badge/github.com/wmentor/tokens)](https://goreportcard.com/report/github.com/wmentor/tokens)\n[![https://pkg.go.dev/github.com/wmentor/tokens](https://pkg.go.dev/badge/github.com/wmentor/tokens.svg)](https://pkg.go.dev/github.com/wmentor/tokens)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\nText to tokens Go library.\n\n# Token get insensitive mode\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"strings\"\n\n\t\"github.com/wmentor/tokens\"\n)\n\nfunc main() {\n\n\ttxt := \"Hello, my little friend!\"\n\n\ttokenizer := tokens.New(strings.NewReader(txt))\n\n\tfor {\n\t\ttok, err := range tokenizer.Token()\n\t\tif err != nil { // io.EOF\n\t\t\tbreak\n\t\t}\n\t\tfmt.Println(tok)\n\t}\n}\n```\n\nResult:\n\n```\nhello\n,\nmy\nlittle\nfriend\n!\n```\n\nCase sensitive mode:\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"strings\"\n\n\t\"github.com/wmentor/tokens\"\n)\n\nfunc main() {\n\n\ttxt := \"Hello, my little friend!\"\n\n\ttokenizer := tokens.New(strings.NewReader(txt), tokens.WithCaseSensitive())\n\n\tfor {\n\t\ttok, err := range tokenizer.Token()\n\t\tif err != nil { // io.EOF\n\t\t\tbreak\n\t\t}\n\t\tfmt.Println(tok)\n\t}\n}\n```\n\nResult:\n\n```\nHello\n,\nmy\nliTTle\nfriend\n!\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwmentor%2Ftokens","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwmentor%2Ftokens","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwmentor%2Ftokens/lists"}