{"id":21081756,"url":"https://github.com/openvoiceos/quebra_frases","last_synced_at":"2026-03-12T00:02:40.844Z","repository":{"id":57459322,"uuid":"358262483","full_name":"OpenVoiceOS/quebra_frases","owner":"OpenVoiceOS","description":"chunks strings into byte sized pieces","archived":false,"fork":false,"pushed_at":"2024-01-28T13:54:26.000Z","size":36,"stargazers_count":1,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-05-03T13:44:03.798Z","etag":null,"topics":["chunk","chunking","sentence-chunking","tokenization","tokenize","tokenized","tokenizer","word-tokenizing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenVoiceOS.png","metadata":{"files":{"readme":"readme.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-15T13:06:37.000Z","updated_at":"2023-06-07T18:51:09.000Z","dependencies_parsed_at":"2024-01-28T14:49:20.072Z","dependency_job_id":"824c9b05-ecb3-48e1-920c-43678b2727dc","html_url":"https://github.com/OpenVoiceOS/quebra_frases","commit_stats":null,"previous_names":["openvoiceos/quebra_frases","openjarbas/quebra_frases"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenVoiceOS%2Fquebra_frases","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenVoiceOS%2Fquebra_frases/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenVoiceOS%2Fquebra_frases/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenVoiceOS%2Fquebra_frases/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenVoiceOS","download_url":"https://codeload.github.com/OpenVoiceOS/quebra_frases/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254504719,"owners_count":22082075,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chunk","chunking","sentence-chunking","tokenization","tokenize","tokenized","tokenizer","word-tokenizing"],"created_at":"2024-11-19T20:10:49.704Z","updated_at":"2025-12-12T02:32:30.686Z","avatar_url":"https://github.com/OpenVoiceOS.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Quebra Frases\n\nquebra_frases chunks strings into byte sized pieces\n\n## Usage\n\nTokenization\n\n```python\nimport quebra_frases\n\nsentence = \"sometimes i develop stuff for mycroft, mycroft is FOSS!\"\nprint(quebra_frases.word_tokenize(sentence))\n# ['sometimes', 'i', 'develop', 'stuff', 'for', 'mycroft', ',', \n# 'mycroft', 'is', 'FOSS', '!']\n\nprint(quebra_frases.span_indexed_word_tokenize(sentence))\n# [(0, 9, 'sometimes'), (10, 11, 'i'), (12, 19, 'develop'), \n# (20, 25, 'stuff'), (26, 29, 'for'), (30, 37, 'mycroft'), \n# (37, 38, ','), (39, 46, 'mycroft'), (47, 49, 'is'), \n# (50, 54, 'FOSS'), (54, 55, '!')]\n\nprint(quebra_frases.sentence_tokenize(\n    \"Mr. Smith bought cheapsite.com for 1.5 million dollars, i.e. he paid a lot for it. Did he mind? Adam Jones Jr. thinks he didn't. In any case, this isn't true... Well, with a probability of .9 it isn't.\"))\n#['Mr. Smith bought cheapsite.com for 1.5 million dollars, i.e. he paid a lot for it.',\n#'Did he mind?',\n#\"Adam Jones Jr. thinks he didn't.\",\n#\"In any case, this isn't true...\",\n#\"Well, with a probability of .9 it isn't.\"]\n\nprint(quebra_frases.span_indexed_sentence_tokenize(\n    \"Mr. Smith bought cheapsite.com for 1.5 million dollars, i.e. he paid a lot for it. Did he mind? Adam Jones Jr. thinks he didn't. In any case, this isn't true... Well, with a probability of .9 it isn't.\"))\n#[(0, 82, 'Mr. Smith bought cheapsite.com for 1.5 million dollars, i.e. he paid a lot for it.'),\n#(83, 95, 'Did he mind?'),\n#(96, 128, \"Adam Jones Jr. thinks he didn't.\"),\n#(129, 160, \"In any case, this isn't true...\"),\n#(161, 201, \"Well, with a probability of .9 it isn't.\")]\n\nprint(quebra_frases.paragraph_tokenize('This is a paragraph!\\n\\t\\nThis is another '\n                                       'one.\\t\\n\\tUsing multiple lines\\t   \\n     '\n                                       '\\n\\tparagraph 3 says goodbye'))\n#['This is a paragraph!\\n\\t\\n',\n#'This is another one.\\t\\n\\tUsing multiple lines\\t   \\n     \\n',\n#'\\tparagraph 3 says goodbye']\n\nprint(quebra_frases.span_indexed_paragraph_tokenize('This is a paragraph!\\n\\t\\nThis is another '\n                                                    'one.\\t\\n\\tUsing multiple lines\\t   \\n     '\n                                                    '\\n\\tparagraph 3 says goodbye'))\n#[(0, 23, 'This is a paragraph!\\n\\t\\n'),\n#(23, 77, 'This is another one.\\t\\n\\tUsing multiple lines\\t   \\n     \\n'),\n#(77, 102, '\\tparagraph 3 says goodbye')]\n```\n\nchunking\n\n```python\nimport quebra_frases\n\ndelimiters = [\"mycroft\"]\nsentence = \"sometimes i develop stuff for mycroft, mycroft is FOSS!\"\nprint(quebra_frases.chunk(sentence, delimiters))\n# ['sometimes i develop stuff for', 'mycroft', ',', 'mycroft', 'is FOSS!']\n\n\nsamples = [\"tell me what do you dream about\",\n           \"tell me what did you dream about\",\n           \"tell me what are your dreams about\",\n           \"tell me what were your dreams about\"]\nprint(quebra_frases.get_common_chunks(samples))\n# {'tell me what', 'about'}\nprint(quebra_frases.get_uncommon_chunks(samples))\n# {'do you dream', 'did you dream', 'are your dreams', 'were your dreams'}\nprint(quebra_frases.get_exclusive_chunks(samples))\n# {'do', 'did', 'are', 'were'}\n\n\nsamples = [\"what is the speed of light\",\n           \"what is the maximum speed of a firetruck\",\n           \"why are fire trucks red\"]\nprint(quebra_frases.get_exclusive_chunks(samples))\n# {'light', 'maximum', 'a firetruck', 'why are fire trucks red'})\nprint(quebra_frases.get_exclusive_chunks(samples, squash=False))\n#[['light'],\n#['maximum', 'a firetruck'],\n#['why are fire trucks red']])\n```\n\n\n## Install\n\n```bash\npip install quebra_frases\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenvoiceos%2Fquebra_frases","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenvoiceos%2Fquebra_frases","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenvoiceos%2Fquebra_frases/lists"}