{"id":16352419,"url":"https://github.com/maxim-saplin/tiktoken-bench","last_synced_at":"2026-04-10T16:40:35.963Z","repository":{"id":213101574,"uuid":"732732368","full_name":"maxim-saplin/tiktoken-bench","owner":"maxim-saplin","description":"Comparing OpenAI tokeniser (tiktoken) performance - stock Python/Rust vs JS/WASM","archived":false,"fork":false,"pushed_at":"2023-12-28T17:05:14.000Z","size":5634,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-12-28T18:55:21.968Z","etag":null,"topics":["ai","chatgpt","javascript","nlp","python","tiktoken"],"latest_commit_sha":null,"homepage":"","language":"Dart","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/maxim-saplin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-12-17T16:49:37.000Z","updated_at":"2023-12-19T13:47:58.000Z","dependencies_parsed_at":"2024-01-01T08:41:20.743Z","dependency_job_id":null,"html_url":"https://github.com/maxim-saplin/tiktoken-bench","commit_stats":{"total_commits":5,"total_committers":1,"mean_commits":5.0,"dds":0.0,"last_synced_commit":"b209aa6c3f997c7f6c9082cf32e9d53f4050e50e"},"previous_names":["maxim-saplin/tiktoken-bench"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxim-saplin%2Ftiktoken-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxim-saplin%2Ftiktoken-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxim-saplin%2Ftiktoken-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxim-saplin%2Ftiktoken-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/maxim-saplin","download_url":"https://codeload.github.com/maxim-saplin/tiktoken-bench/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239619573,"owners_count":19669447,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","chatgpt","javascript","nlp","python","tiktoken"],"created_at":"2024-10-11T01:26:13.443Z","updated_at":"2025-11-20T03:30:15.951Z","avatar_url":"https://github.com/maxim-saplin.png","language":"Dart","funding_links":[],"categories":[],"sub_categories":[],"readme":"Comparing OpenAI tokeniser (tiktoken) performance - stock Python/Rust vs JS/WASM. \n\nRunning tests on M1 MacBook Pro\n\n# Results\n\n- Small File (68 tokens)\n```\nPython/Rust (tiktoken 0.5.2)     ████ (0.04ms)\nPure JS (js-tiktoken 1.0.8)      █████ (0.05ms)\nJS/WASM (tiktoken 1.0.11)        ██████████ (0.11ms)\n@dqbd/WASM 1.0.7                 ██████████████████ (0.18ms)\n```\n\n- Medium File (1068 tokens)\n```\nPython/Rust (tiktoken 0.5.2)     ██████ (0.54ms)\nJS/WASM (tiktoken 1.0.11)        █████████ (0.78ms)\n@dqbd/WASM 1.0.7                 █████████ (0.80ms)\nPure JS (js-tiktoken 1.0.8)      ██████████ (0.96ms)\n```\n\n- Large File (923942 tokens)\n```\nPython/Rust (tiktoken 0.5.2)     ████████████████ (359.49ms)\n@dqbd/WASM 1.0.7                 ████████████████████ (421.71ms)\nJS/WASM (tiktoken 1.0.11)        ██████████████████████ (451.92ms)\nPure JS (js-tiktoken 1.0.8)      █████████████████████████████████████ (1005.69ms)\n```\n\n# Pyhton 3.11.6\n\n## tiktoken 0.5.2\n\nOpenAi implementation (using Rust behind the scenes)\n\n```\nFile: 0_small.txt (2000 - 68) - Avg Time: 0.04ms, StdDev: 29.47%\nFile: 1_medium.txt (200 - 1068) - Avg Time: 0.54ms, StdDev: 3.07%\nFile: 2_large.txt (20 - 923942) - Avg Time: 359.49ms, StdDev: 0.85%\n```\n\n# JS, Node 21.2.0\n\n```\nnpm install\nnpm start\n```\n\nPure JS and Web Assembly versions\n\n## tiktoken 1.0.11 (WASM)\n\n```\nFile: 0_small.txt (2000 - 68) - Avg Time: 0.11ms, StdDev: 55.14%\nFile: 1_medium.txt (200 - 1068) - Avg Time: 0.78ms, StdDev: 6.84%\nFile: 2_large.txt (20 - 923942) - Avg Time: 451.92ms, StdDev: 0.75%\n```\n\n## js-tiktoken 1.0.8\n\n```\nFile: 0_small.txt (2000 - 68) - Avg Time: 0.05ms, StdDev: 125.55%\nFile: 1_medium.txt (200 - 1068) - Avg Time: 0.96ms, StdDev: 29.51%\nFile: 2_large.txt (20 - 923942) - Avg Time: 1005.69ms, StdDev: 0.58%\n```\n\n## @dqbd/tiktoken 1.0.7 (WASM)\n\n```\nFile: 0_small.txt (2000 - 68) - Avg Time: 0.18ms, StdDev: 48.96%\nFile: 1_medium.txt (200 - 1068) - Avg Time: 0.80ms, StdDev: 11.21%\nFile: 2_large.txt (20 - 923942) - Avg Time: 421.71ms, StdDev: 1.30%\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxim-saplin%2Ftiktoken-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaxim-saplin%2Ftiktoken-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxim-saplin%2Ftiktoken-bench/lists"}