{"id":20669430,"url":"https://github.com/dluc/openai-tools","last_synced_at":"2025-04-19T18:12:36.775Z","repository":{"id":61732268,"uuid":"553041265","full_name":"dluc/openai-tools","owner":"dluc","description":"A collection of tools for working with OpenAI","archived":false,"fork":false,"pushed_at":"2023-05-04T10:33:35.000Z","size":572,"stargazers_count":98,"open_issues_count":6,"forks_count":15,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-03-24T00:49:29.707Z","etag":null,"topics":["gpt-3","gpt3","openai","tokenization","tokenizer"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dluc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"dluc"}},"created_at":"2022-10-17T16:10:13.000Z","updated_at":"2025-01-22T13:16:44.000Z","dependencies_parsed_at":"2024-06-21T17:50:47.393Z","dependency_job_id":"b236e689-73a3-4c54-aa84-10f5c11eb706","html_url":"https://github.com/dluc/openai-tools","commit_stats":{"total_commits":5,"total_committers":1,"mean_commits":5.0,"dds":0.0,"last_synced_commit":"6592c596d74d685f0e70d2aeacca3a9f5e258451"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dluc%2Fopenai-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dluc%2Fopenai-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dluc%2Fopenai-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dluc%2Fopenai-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dluc","download_url":"https://codeload.github.com/dluc/openai-tools/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249760126,"owners_count":21321843,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gpt-3","gpt3","openai","tokenization","tokenizer"],"created_at":"2024-11-16T20:14:18.439Z","updated_at":"2025-04-19T18:12:36.748Z","avatar_url":"https://github.com/dluc.png","language":"C#","funding_links":["https://github.com/sponsors/dluc"],"categories":[],"sub_categories":[],"readme":"# GPT Tokenizer\n\n## .NET / C#\n\nWhen using\n[OpenAI GPT](https://openai.com/blog/gpt-3-apps/),\nyou may need to know how many\n[tokens](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them)\nyour code is using for various purposes, such as estimating costs and improving\nresults. \n\nThe `GPT3Tokenizer` C# class can help you **count tokens** in your prompts and\nin the responses received.\n\n```csharp\nusing AI.Dev.OpenAI.GPT;\n\nstring text = \"January 1st, 2000\";\n\n// 5 tokens =\u003e [21339, 352, 301, 11, 4751]\nList\u003cint\u003e tokens = GPT3Tokenizer.Encode(text);\n```\n\nThe tokenizer uses a byte-pair encoding (BPE) algorithm to split words into\nsubwords based on frequency and merges rules. It can handle out-of-vocabulary\nwords, punctuation, and special tokens.\n\nThe result of this library is compatible with OpenAI GPT tokenizer that you can\nalso test\n[here](https://beta.openai.com/tokenizer).\n\n### Installation\n\nInstall [AI.Dev.OpenAI.GPT](https://www.nuget.org/packages/AI.Dev.OpenAI.GPT) NuGet package from nuget.org, e.g.:\n\n    dotnet add package AI.Dev.OpenAI.GPT --version 1.0.2\n\nor\n\n    NuGet\\Install-Package AI.Dev.OpenAI.GPT -Version 1.0.2\n\n## Python and Node.js\n\nIf you are looking for an equivalent solution in other languages:\n\n* [Python GPT tokenizer](https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2Tokenizer)\n* [Node.js GPT encoder](https://www.npmjs.com/package/gpt-3-encoder)\n\n# Licensing\n\nThis library is licensed CC0, in the public domain. You can use it for any\napplication, you can modify the code, and you can redistribute any part of it.\n\nI am not affiliated with OpenAI and this library is not endorsed by them. I just\nwork with several AI solutions and I share this code hoping to make technology\nmore accessible and easier to work with.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdluc%2Fopenai-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdluc%2Fopenai-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdluc%2Fopenai-tools/lists"}