{"id":25841766,"url":"https://github.com/iconnect/tiktoken-hs","last_synced_at":"2025-07-29T00:09:22.822Z","repository":{"id":199864594,"uuid":"693585607","full_name":"iconnect/tiktoken-hs","owner":"iconnect","description":"Haskell bindings to an extremely limited subset of tiktoken-rs","archived":false,"fork":false,"pushed_at":"2024-02-15T14:33:43.000Z","size":39,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-08T18:48:22.936Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iconnect.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-19T10:13:54.000Z","updated_at":"2024-08-27T14:55:05.000Z","dependencies_parsed_at":"2023-11-22T11:51:13.052Z","dependency_job_id":null,"html_url":"https://github.com/iconnect/tiktoken-hs","commit_stats":null,"previous_names":["iconnect/tiktoken-hs"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/iconnect/tiktoken-hs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iconnect%2Ftiktoken-hs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iconnect%2Ftiktoken-hs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iconnect%2Ftiktoken-hs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iconnect%2Ftiktoken-hs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iconnect","download_url":"https://codeload.github.com/iconnect/tiktoken-hs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iconnect%2Ftiktoken-hs/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267606662,"owners_count":24114768,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-28T02:00:09.689Z","response_time":68,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-01T05:30:39.891Z","updated_at":"2025-07-29T00:09:22.802Z","avatar_url":"https://github.com/iconnect.png","language":"Haskell","funding_links":[],"categories":[],"sub_categories":[],"readme":"\ntiktoken.hs\n===========\n\nThis library is a binding to an _extremely_ (as in, one function) subset of the\n`tiktoken-rs` library. It exposes a function `countTokens :: Text -\u003e Word64` which\ncan be used to count tokens and return a result which should match the one returned\nby OpenAI itself (see for example [their online tool](https://platform.openai.com/tokenizer)).\n\n## Library design\n\nThis library uses the [haskell-foreign-rust](https://github.com/BeFunctional/haskell-foreign-rust)\nand [haskell-rust-ffi](https://github.com/BeFunctional/haskell-rust-ffi) to call into [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs)\nwhich is currently the industry-standard for tokenisation. Internally, this library is really composed\nby a Rust wrapper and a Haskell library, where the former is shipped alongside the latter, and we use\na Custom setup script to seamlessly build the Rust wrapper before building the Haskell library.\n\nFor more information see the blog post [Calling Purgatory from Heaven](https://well-typed.com/blog/2023/03/purgatory/).\n\n## Building the project\n\nThis project requires a `nighly` version of the Rust toolchain as well as the `cargo-c` applet. You can\ninstall both with:\n\n```\nrustup toolchain install nightly\ncargo install cargo-c\n```\n\nThen, you can build this project like any other Haskell library with `cabal v2-build`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ficonnect%2Ftiktoken-hs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ficonnect%2Ftiktoken-hs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ficonnect%2Ftiktoken-hs/lists"}