{"id":23391433,"url":"https://github.com/ahrefs/ocaml-sentencepiece","last_synced_at":"2025-06-17T00:36:50.801Z","repository":{"id":207118805,"uuid":"701320592","full_name":"ahrefs/ocaml-sentencepiece","owner":"ahrefs","description":"OCaml bindings to SentencePiece","archived":false,"fork":false,"pushed_at":"2023-11-22T09:39:10.000Z","size":145,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-12T05:44:39.711Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"OCaml","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ahrefs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-10-06T11:52:05.000Z","updated_at":"2023-10-06T13:01:23.000Z","dependencies_parsed_at":"2025-05-31T13:01:20.060Z","dependency_job_id":"11e8a8ea-0215-4baf-8259-17784be430c4","html_url":"https://github.com/ahrefs/ocaml-sentencepiece","commit_stats":null,"previous_names":["ahrefs/ocaml-sentencepiece"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ahrefs/ocaml-sentencepiece","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahrefs%2Focaml-sentencepiece","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahrefs%2Focaml-sentencepiece/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahrefs%2Focaml-sentencepiece/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahrefs%2Focaml-sentencepiece/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ahrefs","download_url":"https://codeload.github.com/ahrefs/ocaml-sentencepiece/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahrefs%2Focaml-sentencepiece/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260268632,"owners_count":22983601,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-22T04:17:27.691Z","updated_at":"2025-06-17T00:36:50.745Z","avatar_url":"https://github.com/ahrefs.png","language":"OCaml","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OCaml bindings to SentencePiece\n\n[SentencePiece](https://github.com/google/sentencepiece) is an unsupervised text tokenizer. \nTested on [v0.1.99](https://github.com/google/sentencepiece/releases/tag/v0.1.99).\n\n## Set up\n```\nsudo apt install libsentencepiece-dev\n```\n\nYou also need a sentencepiece model ([example](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2/resolve/main/sentencepiece.bpe.model)).\n\n\n## Example\n\n```ocaml\nutop # open Sentencepiece;;\nutop # #install_printer Processor.pp_int64_array1;;\nutop # let p = Processor.load_model \"sentencepiece.bpe.model\" |\u003e Result.get_ok;;\nval p : Processor.t = \u003cabstr\u003e\nutop # Processor.encode_int64_ids p \"Hey there!\";;\n- : (int64, Bigarray.int64_elt, Bigarray.c_layout) Bigarray.Array1.t =\n[|1; 28239; 2684; 37; 2; |]\nutop # Processor.encode_pieces p \"Hey there!\";;\n- : string list = [\"\u003cs\u003e\"; \"▁Hey\"; \"▁there\"; \"!\"; \"\u003c/s\u003e\"]\nutop # Processor.(decode_pieces p @@ encode_pieces p \"Hey there!\");;\n- : string = \"Hey there!\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahrefs%2Focaml-sentencepiece","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fahrefs%2Focaml-sentencepiece","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahrefs%2Focaml-sentencepiece/lists"}