{"id":19898950,"url":"https://github.com/mathematicator-core/tokenizer","last_synced_at":"2025-05-02T22:31:38.230Z","repository":{"id":52380223,"uuid":"198897227","full_name":"mathematicator-core/tokenizer","owner":"mathematicator-core","description":"Tokenizer that can convert string (user input / LaTeX) to numbers and operators.","archived":false,"fork":false,"pushed_at":"2024-07-10T04:57:48.000Z","size":125,"stargazers_count":10,"open_issues_count":4,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-07T07:51:11.957Z","etag":null,"topics":["binary-tree","calculator","latex","math","math-formulas","math-language","math-tokenizer","tokenizer"],"latest_commit_sha":null,"homepage":"https://baraja.cz","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mathematicator-core.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/funding.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":"janbarasek","custom":["https://baraja.cz","https://php.baraja.cz","https://vue.baraja.cz"]}},"created_at":"2019-07-25T20:28:59.000Z","updated_at":"2025-03-26T07:37:28.000Z","dependencies_parsed_at":"2022-08-21T19:20:08.902Z","dependency_job_id":null,"html_url":"https://github.com/mathematicator-core/tokenizer","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mathematicator-core%2Ftokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mathematicator-core%2Ftokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mathematicator-core%2Ftokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mathematicator-core%2Ftokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mathematicator-core","download_url":"https://codeload.github.com/mathematicator-core/tokenizer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252116360,"owners_count":21697364,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binary-tree","calculator","latex","math","math-formulas","math-language","math-tokenizer","tokenizer"],"created_at":"2024-11-12T20:06:33.622Z","updated_at":"2025-05-02T22:31:35.763Z","avatar_url":"https://github.com/mathematicator-core.png","language":"PHP","funding_links":["https://github.com/sponsors/janbarasek","https://baraja.cz","https://php.baraja.cz","https://vue.baraja.cz"],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n    PHP Math Tokenizer\n\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://mathematicator.com\" target=\"_blank\"\u003e\n        \u003cimg src=\"https://avatars3.githubusercontent.com/u/44620375?s=100\u0026v=4\"\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n[![Integrity check](https://github.com/mathematicator-core/tokenizer/workflows/Integrity%20check/badge.svg)](https://github.com/mathematicator-core/tokenizer/actions?query=workflow%3A%22Integrity+check%22)\n[![codecov](https://codecov.io/gh/mathematicator-core/tokenizer/branch/master/graph/badge.svg)](https://codecov.io/gh/mathematicator-core/tokenizer)\n[![Latest Stable Version](https://poser.pugx.org/mathematicator-core/tokenizer/v/stable)](https://packagist.org/packages/mathematicator-core/tokenizer)\n[![Latest Unstable Version](https://poser.pugx.org/mathematicator-core/tokenizer/v/unstable)](https://packagist.org/packages/mathematicator-core/tokenizer)\n[![License: MIT](https://img.shields.io/badge/License-MIT-brightgreen.svg)](./LICENSE)\n[![PHPStan Enabled](https://img.shields.io/badge/PHPStan-enabled%20L8-brightgreen.svg?style=flat)](https://phpstan.org/)\n\nMathematicator Tokenizer is a simple library for converting string math formulas to\na stream of tokens with LaTeX support.\n\n## Installation\n\n```\ncomposer require mathematicator-core/tokenizer\n```\n\n## Features\n\n- Convert all your math formulas to a stream of tokens\n- Convert user math input to LaTeX\n- Render the tokens tree map for debug\n\n## Usage\n\nInject the `Tokenizer` service via [DI](https://doc.nette.org/en/3.0/dependency-injection)\nand tokenize your query.\n\n```php\nuse \\Mathematicator\\Tokenizer\\Tokenizer;\n\n$tokenizer = new Tokenizer(/* some dependencies */);\n\n// Convert math formula to an array of tokens:\n$tokens = $tokenizer-\u003etokenize('(5+3)*(2/(7+3))');\n\n// Now you can convert tokens to a more useful format:\n$objectTokens = $tokenizer-\u003etokensToObject($tokens);\n\nvar_dump($objectTokens); // Return typed tokens with meta data\n\n// Render to LaTeX\necho $tokenizer-\u003etokensToLatex($objectTokens);\n\n// Render to debug tree (extremely fast):\necho $tokenizer-\u003erenderTokensTree($objectTokens);\n```\n\n### Configuration\n\nThe tokenizer uses automatic configuration based on DI.\nJust use the DI container and the service will be fully available.\n\n## Mathematicator Framework tools structure\n\nThe biggest advantage is that you can choose which layer best fits\nyour needs and start build on the top of it, immediately, without the need\nto create everything by yourself. Our tools are tested for bugs\nand tuned for performance, so you can save a significant amount\nof your time, money, and effort.\n\nFramework tend to be modular as much as possible, so you should be able\nto create an extension on each layer and its sublayers.\n\n**Mathematicator framework layers** ordered from the most concrete\none to the most abstract one:\n\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\n            \u003cb\u003e\n            \u003ca href=\"https://github.com/mathematicator-core/search\"\u003e\n                Search\n            \u003c/a\u003e\n            \u003c/b\u003e\n        \u003c/td\u003e\n        \u003ctd\u003e\n            Modular search engine layer that calls its sublayers\n            and creates user interface.\n        \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\n            \u003cb\u003e\n            \u003ca href=\"https://github.com/mathematicator-core/vizualizator\"\u003e\n                Vizualizator\n            \u003c/a\u003e\n            \u003c/b\u003e\n        \u003c/td\u003e\n        \u003ctd\u003e\n            Elegant graphic visualizer that can render to\n            SVG, PNG, JPG and Base64.\u003cbr /\u003e\n            \u003cu\u003eExtensions:\u003c/u\u003e\n            \u003cb\u003e\n            \u003ca href=\"https://github.com/mathematicator-core/mandelbrot-set\"\u003e\n                Mandelbrot set generator\n            \u003c/a\u003e\n            \u003c/b\u003e\n        \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\n            \u003cb\u003e\n            \u003ca href=\"https://github.com/mathematicator-core/calculator\"\u003e\n                Calculator\n            \u003c/a\u003e\n            \u003c/b\u003e\n        \u003c/td\u003e\n        \u003ctd\u003e\n            Modular advance calculations layer.\n            \u003cbr /\u003e\n            \u003cu\u003eExtensions:\u003c/u\u003e\n            \u003cb\u003e\n            \u003ca href=\"https://github.com/mathematicator-core/integral-solver\"\u003e\n                Integral Solver\n            \u003c/a\u003e,\n            \u003ca href=\"https://github.com/mathematicator-core/statistic\"\u003e\n                Statistics\n            \u003c/a\u003e\n            \u003c/b\u003e\n        \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\n            \u003cb\u003e\n            \u003ca href=\"https://github.com/mathematicator-core/engine\"\u003e\n                Engine\n            \u003c/a\u003e\n            \u003c/b\u003e\n        \u003c/td\u003e\n        \u003ctd\u003e\n            Core logic layer that maintains basic controllers,\n            DAOs, translator, common exceptions, routing etc.\n        \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\n            \u003cb\u003e\n            \u003ca href=\"https://github.com/mathematicator-core/tokenizer\"\u003e\n                Tokenizer\n            \u003c/a\u003e\n            \u003c/b\u003e\n        \u003c/td\u003e\n        \u003ctd\u003e\n            Tokenizer that can convert string (user input / LaTeX) to numbers\n            and operators.\n        \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\n            \u003cb\u003e\n            \u003ca href=\"https://github.com/mathematicator-core/numbers\"\u003e\n                Numbers\n            \u003c/a\u003e\n            \u003c/b\u003e\n        \u003c/td\u003e\n        \u003ctd\u003e\n            Fast \u0026 secure storage for numbers with arbitrary precision.\n            It supports Human string and LaTeX output and basic conversions.\n        \u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\n**Third-party packages:**\n\n⚠️ Not guaranteed!\n\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\n            \u003cb\u003e\n            \u003ca href=\"https://github.com/cothema/math-php-api\"\u003e\n                REST API\n            \u003c/a\u003e\n            \u003c/b\u003e\n        \u003c/td\u003e\n        \u003ctd\u003e\n            Install the whole pack as a REST API service\n            on your server (Docker ready) or\n            access it via public cloud REST API.\n        \u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\n## Contribution\n\n\u003e Please help to improve this documentation by sending a Pull request.\n\n### Tests\n\nAll new contributions should have its unit tests in `/tests` directory.\n\nBefore you send a PR, please, check all tests pass.\n\nThis package uses [Nette Tester](https://tester.nette.org/).\nYou can run tests via command:\n```bash\ncomposer test\n````\n\nFor benchmarking, we use [phpbench](https://github.com/phpbench/phpbench).\nYou can run benchmarks this way:\n```bash\ncomposer global require phpbench/phpbench @dev # only the first time\nphpbench run\n````\n\nBefore PR, please run complete code check via command:\n```bash\ncomposer cs:install # only first time\ncomposer fix # otherwise pre-commit hook can fail\n````\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmathematicator-core%2Ftokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmathematicator-core%2Ftokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmathematicator-core%2Ftokenizer/lists"}