{"id":13493536,"url":"https://github.com/hansalemaos/charchef","last_synced_at":"2025-03-28T11:32:55.961Z","repository":{"id":66543383,"uuid":"603995821","full_name":"hansalemaos/charchef","owner":"hansalemaos","description":"3 functions to normalize strings, repair bad encoding, replace non-printable characters","archived":false,"fork":false,"pushed_at":"2023-02-28T20:23:42.000Z","size":233,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-21T20:07:35.008Z","etag":null,"topics":["ascii","normalize","numba","python","unicode"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/charchef/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hansalemaos.png","metadata":{"files":{"readme":"README.MD","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-02-20T05:29:47.000Z","updated_at":"2024-01-23T20:58:16.000Z","dependencies_parsed_at":"2024-01-24T03:15:01.491Z","dependency_job_id":"316598e6-10b2-4511-a448-a23500b5cc5c","html_url":"https://github.com/hansalemaos/charchef","commit_stats":{"total_commits":5,"total_committers":2,"mean_commits":2.5,"dds":0.4,"last_synced_commit":"e48a890f61a0fcd8f889716eb92d206c808d70c0"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hansalemaos%2Fcharchef","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hansalemaos%2Fcharchef/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hansalemaos%2Fcharchef/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hansalemaos%2Fcharchef/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hansalemaos","download_url":"https://codeload.github.com/hansalemaos/charchef/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245383176,"owners_count":20606268,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ascii","normalize","numba","python","unicode"],"created_at":"2024-07-31T19:01:16.234Z","updated_at":"2025-03-28T11:32:55.517Z","avatar_url":"https://github.com/hansalemaos.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"### 3 functions to normalize strings, repair bad encoding, replace non-printable characters \n\n\n#### The function use numba under the hood - that means the first run is very slow, (compile time), but then the speed-up is tremendous.\n\n\n##### pip install charchef\n\n```python\nfrom charchef import aa_convert_utf8_to_ascii_,aa_repair_bad_conversion_to_utf8,aa_replace_non_printable_chars\ntext = r\"\"\"ąćęłńóśźż ĄĆĘŁŃÓŚŹ\\x00Ż Junto à Estação de Carcavelos; Bragança Situado \nen el núcleo de Es Caló de Sant Agustí frente al Hostal Rafalet. Cartão MOBI.E R. \nConselheiro Emídio Navarro (frente ao ISEL) R. Conselheiro Emidio Navarro (frente ao ISEL)\n àáâãäåa èéêëe ìíîïi òóôõöo ùúûüu ýÿy Suzy \u0026amp; John \u0026quot; \u0026pound;682m \n \\u00FF\\u00FF\\u00F0\\u00f0\\x95\\xFF SmörgÃ¥s Non ti suscita niente la parola pietÃ\\xa0 RosŽ RUF MICH ZURÃœCK.\n aqu\\195\\173 09. Bát Nhã Tâm Kinh criança Koç University Technische Universität Dresden Universität \n für Musik und darstellende Kunst Wien Technische Universität Wien Ã\\x89cole Nationale Supérieure \n des Beaux-Arts Paris Universidad Simón BolÃ\\xadvar (USB) 240         Åland Islands   2014.0    \n MARIEHAMN   11437.0 1 240         Åland Islands   2010.0      MARIEHAMN   5829.5  1 240        \n Albania         2011.0      Durrës      113249.0 240         Albania         2011.0      TIRANA \n 418495.0 240         Albania         2011.0      Durrës      56511.0 \"Tutu Au Mic' – dumbéa\"\n    \"\"\".splitlines()\nbigc1 = aa_convert_utf8_to_ascii_(\n        str_=text,\n        preprocessing_functions=(\n            \"8x_3_lower_case_escaped\",\n            \"8x_3_upper_case_escaped\",\n            \"8u_4_upper_case_escaped\",\n            \"8u_4_lower_case_escaped\",\n            \"8x_69_upper_case_escaped\",\n            \"8x_69_lower_case_escaped\",\n            \"8n_escaped\",\n            \"8wrong_chars\",\n            \"8zerox_unescaped_lower\",\n            \"8zerox_unescaped_upper\",\n            \"8html_entity\",\n        ),\n        preprocessing_function_non_printable=(\n            \"substitute_allcontrols_s\",\n            \"substitute_allcontrols\",\n            \"substitute_allcontrols2\",\n            \"substitute_allcontrols2_s\",\n            \"substitute_allcontrols3\",\n            \"substitute_allcontrols3_s\",\n        ),\n        respect_german_letters=True,\n    )\n\nbigc2 = aa_repair_bad_conversion_to_utf8(\n        str_=text,\n        functions=(\n            \"8x_3_lower_case_escaped\",\n            \"8x_3_upper_case_escaped\",\n            \"8u_4_upper_case_escaped\",\n            \"8u_4_lower_case_escaped\",\n            \"8x_69_upper_case_escaped\",\n            \"8x_69_lower_case_escaped\",\n            \"8n_escaped\",\n            \"8wrong_chars\",\n            \"8zerox_unescaped_lower\",\n            \"8zerox_unescaped_upper\",\n            \"8html_entity\",\n        ),\n    )\n\nbigc3 = aa_replace_non_printable_chars(\n        str_=\"\\x00rsi\\\\x00d\\x00ad \\x0aSimón BolÃ\\xadvar\",\n        functions=(\n            \"substitute_allcontrols_s\",\n            \"substitute_allcontrols\",\n            \"substitute_allcontrols2\",\n            \"substitute_allcontrols2_s\",\n        ),\n        removex0a=False,\n    )\n\t\n\t\nbigc1 # replaces all accents, special characters ... \nOut[3]: \n['acelnoszz ACELNOSZZ Junto a Estacao de Carcavelos; Braganca Situado ',\n 'en el nucleo de Es Calo de Sant Agusti frente al Hostal Rafalet. Cartao MOBI.E R. ',\n 'Conselheiro Emidio Navarro (frente ao ISEL) R. Conselheiro Emidio Navarro (frente ao ISEL)',\n ' aaaaaeaa eeeee iiiii oooooeo uuuueu yyy Suzy \u0026 John \" PS682m ',\n ' yydd*y Smorgas Non ti suscita niente la parola pieti RosZ RUF MICH ZURUCK.',\n ' aqui 09. Bat Nha Tam Kinh crianca Koc University Technische Universitat Dresden Universitat ',\n ' fur Musik und darstellende Kunst Wien Technische Universitat Wien Ecole Nationale Superieure ',\n ' des Beaux-Arts Paris Universidad Simon Bolivar (USB) 240         Sland Islands   2014.0    ',\n ' MARIEHAMN   11437.0 1 240         Sland Islands   2010.0      MARIEHAMN   5829.5  1 240        ',\n ' Albania         2011.0      Durres      113249.0 240         Albania         2011.0      TIRANA ',\n ' 418495.0 240         Albania         2011.0      Durres      56511.0 \"Tutu Au Mic\\' - dumbea\"',\n '    ']\n \n \nbigc2 # Repairs messed up Unicode\nOut[4]: \n['ąćęłńóśźż ĄĆĘŁŃÓŚŹŻ Junto à Estação de Carcavelos; Bragança Situado ',\n 'en el núcleo de Es Caló de Sant Agustí frente al Hostal Rafalet. Cartão MOBI.E R. ',\n 'Conselheiro Emídio Navarro (frente ao ISEL) R. Conselheiro Emidio Navarro (frente ao ISEL)',\n ' àáâãäåa èéêëe ìíîïi òóôõöo ùúûüu ýÿy Suzy \u0026 John \" £682m ',\n ' ÿÿðð•ÿ Smörgås Non ti suscita niente la parola pietí RosŽ RUF MICH ZURÜCK.',\n ' aquí 09. Bát Nhã Tâm Kinh criança Koç University Technische Universität Dresden Universität ',\n ' für Musik und darstellende Kunst Wien Technische Universität Wien École Nationale Supérieure ',\n ' des Beaux-Arts Paris Universidad Simón Bolívar (USB) 240         Šland Islands   2014.0    ',\n ' MARIEHAMN   11437.0 1 240         Šland Islands   2010.0      MARIEHAMN   5829.5  1 240        ',\n ' Albania         2011.0      Durrës      113249.0 240         Albania         2011.0      TIRANA ',\n ' 418495.0 240         Albania         2011.0      Durrës      56511.0 \"Tutu Au Mic\\' – dumbéa\"',\n '    ']\n \n \nbigc3 # Removes non-printable characters\nOut[5]: ['rsidad Simón BolÃ\\xadvar']\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhansalemaos%2Fcharchef","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhansalemaos%2Fcharchef","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhansalemaos%2Fcharchef/lists"}