{"id":20664263,"url":"https://github.com/casualcodersprojects/tokenize_all_code","last_synced_at":"2026-04-17T14:33:24.293Z","repository":{"id":222495068,"uuid":"757474645","full_name":"CasualCodersProjects/tokenize_all_code","owner":"CasualCodersProjects","description":"Tokenize code snippets.","archived":false,"fork":false,"pushed_at":"2024-02-14T15:26:45.000Z","size":9,"stargazers_count":0,"open_issues_count":1,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-10-31T00:26:38.718Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://pypi.org/project/tokenize-all-code","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CasualCodersProjects.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-02-14T15:14:25.000Z","updated_at":"2024-02-14T15:23:10.000Z","dependencies_parsed_at":"2024-02-14T15:42:31.738Z","dependency_job_id":null,"html_url":"https://github.com/CasualCodersProjects/tokenize_all_code","commit_stats":null,"previous_names":["casualcodersprojects/tokenize_all_code"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/CasualCodersProjects/tokenize_all_code","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CasualCodersProjects%2Ftokenize_all_code","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CasualCodersProjects%2Ftokenize_all_code/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CasualCodersProjects%2Ftokenize_all_code/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CasualCodersProjects%2Ftokenize_all_code/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CasualCodersProjects","download_url":"https://codeload.github.com/CasualCodersProjects/tokenize_all_code/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CasualCodersProjects%2Ftokenize_all_code/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31933197,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-17T12:37:54.787Z","status":"ssl_error","status_checked_at":"2026-04-17T12:37:25.095Z","response_time":62,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T19:23:16.055Z","updated_at":"2026-04-17T14:33:24.272Z","avatar_url":"https://github.com/CasualCodersProjects.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tokenize All Code\r\n\r\nThis is a clone of the original code that was found [here](https://github.com/NicholasIapalucci/Tokenize-All). That repo has since disappeared, so I have cloned it here for archival and development purposes.\r\n\r\nTokenize blocks of code in Python. Used by `manim-code-blocks` to syntax highlight blocks of code. \r\n\r\n## Example Usage\r\n\r\n```python\r\nfrom tokenize_all import Java\r\n\r\ntokens = Java.tokenize(\r\n    \"\"\"\r\n    public class Main {\r\n        public static void main(String[] args) {\r\n            System.out.println(\"Hello world!\");\r\n        }\r\n    }\r\n    \"\"\"\r\n)\r\n\r\nfor token in tokens: print(token)\r\n```\r\n\r\nOutput:\r\n\r\n```\r\n\u003e\u003e Token[ type = keyword, value = public, start = 4, end = 10 ]\r\n\u003e\u003e Token[ type = keyword, value = class, start = 11, end = 16 ]\r\n\u003e\u003e Token[ type = class name, value = Main, start = 17, end = 21 ]\r\n\u003e\u003e Token[ type = left brace, value = {, start = 22, end = 23 ]\r\n\u003e\u003e Token[ type = keyword, value = public, start = 31, end = 37 ]\r\n\u003e\u003e Token[ type = keyword, value = static, start = 38, end = 44 ]\r\n\u003e\u003e Token[ type = keyword, value = void, start = 45, end = 49 ]\r\n\u003e\u003e Token[ type = function, value = main, start = 50, end = 54 ]\r\n\u003e\u003e Token[ type = left parentheses, value = (, start = 54, end = \u003e\u003e 55 ]\r\n\u003e\u003e Token[ type = class name, value = String, start = 55, end = \u003e\u003e 61 ]\r\n\u003e\u003e Token[ type = left bracket, value = [, start = 61, end = 62 ]\r\n\u003e\u003e Token[ type = right bracket, value = ], start = 62, end = 63 ]\r\n\u003e\u003e Token[ type = identifier, value = args, start = 64, end = 68 ]\r\n\u003e\u003e Token[ type = right parentheses, value = ), start = 68, end = \u003e\u003e 69 ]\r\n\u003e\u003e Token[ type = left brace, value = {, start = 70, end = 71 ]\r\n\u003e\u003e Token[ type = class name, value = System, start = 83, end = \u003e\u003e 89 ]\r\n\u003e\u003e Token[ type = dot, value = ., start = 89, end = 90 ]\r\n\u003e\u003e Token[ type = identifier, value = out, start = 90, end = 93 ]\r\n\u003e\u003e Token[ type = dot, value = ., start = 93, end = 94 ]\r\n\u003e\u003e Token[ type = function, value = println, start = 94, end = \u003e\u003e 101 ]\r\n\u003e\u003e Token[ type = left parentheses, value = (, start = 101, end = \u003e\u003e 102 ]\r\n\u003e\u003e Token[ type = string, value = \"Hello world!\", start = 102, \u003e\u003e end = 116 ]\r\n\u003e\u003e Token[ type = right parentheses, value = ), start = 116, end \u003e\u003e = 117 ]\r\n\u003e\u003e Token[ type = semicolon, value = ;, start = 117, end = 118 ]\r\n\u003e\u003e Token[ type = right brace, value = }, start = 126, end = 127 ]\r\n\u003e\u003e Token[ type = right brace, value = }, start = 131, end = 132 ]\r\n```\r\n\r\n# Building and Publishing\r\n\r\nRequires Python 3.9 or later.\r\n\r\n```sh\r\npip install twine build\r\npython -m build\r\ntwine upload dist/*\r\n```\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcasualcodersprojects%2Ftokenize_all_code","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcasualcodersprojects%2Ftokenize_all_code","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcasualcodersprojects%2Ftokenize_all_code/lists"}