{"id":13961514,"url":"https://github.com/tokay-lang/tokay","last_synced_at":"2025-12-12T10:58:01.348Z","repository":{"id":37057991,"uuid":"214232373","full_name":"tokay-lang/tokay","owner":"tokay-lang","description":"Tokay programming language, designed for ad-hoc parsing and syntax-directed development.","archived":false,"fork":false,"pushed_at":"2025-10-26T21:30:20.000Z","size":1960,"stargazers_count":248,"open_issues_count":50,"forks_count":7,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-10-26T23:16:00.481Z","etag":null,"topics":["ad-hoc","awk","compiler","language","packrat","parsing","programming-language"],"latest_commit_sha":null,"homepage":"https://tokay.dev","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tokay-lang.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2019-10-10T16:29:01.000Z","updated_at":"2025-10-26T21:30:25.000Z","dependencies_parsed_at":"2023-11-08T23:25:08.215Z","dependency_job_id":"a2439577-00ae-4d04-a492-7ea6e830a004","html_url":"https://github.com/tokay-lang/tokay","commit_stats":{"total_commits":601,"total_committers":2,"mean_commits":300.5,"dds":"0.0033277870216306127","last_synced_commit":"157e2045a5ade75a80da23d38a742b3b2da2d955"},"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"purl":"pkg:github/tokay-lang/tokay","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokay-lang%2Ftokay","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokay-lang%2Ftokay/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokay-lang%2Ftokay/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokay-lang%2Ftokay/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tokay-lang","download_url":"https://codeload.github.com/tokay-lang/tokay/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokay-lang%2Ftokay/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27681502,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-12T02:00:06.775Z","response_time":129,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ad-hoc","awk","compiler","language","packrat","parsing","programming-language"],"created_at":"2024-08-08T17:01:12.782Z","updated_at":"2025-12-12T10:58:01.341Z","avatar_url":"https://github.com/tokay-lang.png","language":"Rust","funding_links":[],"categories":["Rust","Uncategorized"],"sub_categories":["Uncategorized"],"readme":"\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/tokay-lang/tokay-artwork/main/banner-with-darkmode-styling.svg\" alt=\"The Tokay Logo\" title=\"Tokay Programming Language\" style=\"width: 80%\"\u003e\u003cbr\u003e\n    \u003ca href=\"https://github.com/tokay-lang/tokay/actions/workflows/main.yml\"\u003e\n        \u003cimg src=\"https://github.com/tokay-lang/tokay/actions/workflows/main.yml/badge.svg\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://crates.io/crates/tokay\"\u003e\n        \u003cimg src=\"https://img.shields.io/crates/v/tokay\" alt=\"Version badge for crates.io\" title=\"crates.io/tokay\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://docs.rs/tokay/latest/tokay\"\u003e\n        \u003cimg src=\"https://img.shields.io/docsrs/tokay\" alt=\"Badge of docs.rs\" title=\"docs.rs/tokay\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://tokay.dev/\"\u003e\n        \u003cimg src=\"https://img.shields.io/website?down_color=red\u0026down_message=offline\u0026up_color=green\u0026up_message=online\u0026url=https%3A%2F%2Ftokay.dev%2F\" alt=\"Indicator for tokay.dev Website availability\" title=\"tokay.dev\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://opensource.org/licenses/MIT\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/License-MIT-green.svg\" alt=\"MIT License badge\" title=\"MIT License\"\u003e\n    \u003c/a\u003e\n\u003c/div\u003e\n\n# Tokay\n\nTokay is a programming language, designed for ad-hoc parsing and syntax-driven development.\n\n\n\u003e [!IMPORTANT]\n\u003e Tokay is under development and not considered for production use yet.\n\u003e Be part of Tokay's ongoing development, and [contribute](CONTRIBUTING.md)!\n\u003e\n\n## About\n\nTokay is a programming language that can be used to quickly implement solutions for text processing problems. This can involve either simple data extractions, but also the parsing of syntactical structures or parts thereof and the conversion of information into structured parse trees or abstract syntax trees for additional processing.\n\nTherefore, Tokay is not only a practical tool for simple one-liners like matchers or recognizers, but also a language that can be used to implement code analyzers, refactoring tools, interpreters, compilers or transpilers. Actually [Tokay's own language parser](src/compiler/Tokay.tok) is implemented in Tokay itself.\n\nTokay is inspired by [awk](https://en.wikipedia.org/wiki/AWK), has syntactic and semantic flavours from [Python](https://www.python.org/) and [Rust](https://www.rust-lang.org/), but also follows its own philosophy, ideas and design principles. Thus, Tokay isn't directly compareable to other languages or projects, and stands on its own. It's an entirely new programming language in its own niche.\n\nTokay is still a very young project and gains much potential. Currently, development is going towards generic parselets (a speciality of Tokay) and a [WebAssembly-based development environment](https://github.com/tokay-lang/tokay-wasm).\n\n[Volunteers are welcome!](CONTRIBUTING.md)\n\n## Highlights\n\n- Interpreted, procedural and imperative scripting language\n- Concise and easy to learn syntax and object system\n- Stream-based input processing\n- Automatic parse tree construction and synthesis\n- Left-recursive parsing structures (\"parselets\") supported\n- Implements a memoizing packrat parsing algorithm internally\n- Robust and fast, as it is written entirely in safe [Rust](https://rust-lang.org)\n- Enabling awk-style one-liners in combination with other tools\n- Generic parselets and functions (*partly implemented)\n- Import system to create modularized programs (*coming soon)\n- Embedded interoperability with other programs (*coming soon)\n\n## Installation\n\nBy using Rusts dependency manager and build-tool `cargo`, simply install Tokay with\n\n```bash\n$ cargo install tokay\n```\n\nFor Arch Linux-based distros, there is also a [`tokay`](https://aur.archlinux.org/packages/tokay) and [`tokay-git`](https://aur.archlinux.org/packages/tokay-git) package in the [Arch Linux AUR](https://aur.archlinux.org/).\n\n## Examples\n\nTokay's version of \"Hello World\" is quite obvious.\n\n```tokay\nprint(\"Hello World\")\n```\n\n\u003e ```\n\u003e $ tokay 'print(\"Hello World\")'\n\u003e Hello World\n\u003e ```\n\nTokay can also greet any wor(l)ds that are being fed to it. The next program prints \"Hello Venus\", \"Hello Earth\" or \"Hello\" followed by any other name previously parsed by the builtin `Word`-token. Any other input than a word is automatically omitted.\n\n```tokay\nprint(\"Hello\", Word)\n```\n\n\u003e ```\n\u003e $ tokay 'print(\"Hello\", Word)' -- \"World 1337 Venus Mars 42 Max\"\n\u003e Hello World\n\u003e Hello Venus\n\u003e Hello Mars\n\u003e Hello Max\n\u003e ```\n\nA simple program for counting words and numbers and printing a total afterwards can be implemented like this:\n\n```tokay\nWord words += 1\nNumber numbers += 1\nend print(words || 0, \"words,\", numbers || 0, \"numbers\")\n```\n\n\u003e ```\n\u003e $ tokay 'Word words += 1; Number numbers += 1; end print(words || 0, \"words,\", numbers || 0, \"numbers\")' -- \"this is just the 1st stage of 42.5 or .1 others\"\n\u003e 9 words, 3 numbers\n\u003e ```\n\nBy design, Tokay constructs syntax trees from consumed information automatically.\n\nThe next program implements a parser and interpreter for simple mathematical expressions, like `1 + 2 + 3` or `7 * (8 + 2) / 5`. The result of each expression is printed afterwards.\n\nProcessing direct and indirect left-recursions without ending in infinite loops is one of Tokay's core features.\n\n```tokay\n_ : Char\u003c \\t\u003e+            # redefine whitespace to just tab and space\n\nFactor : @{\n    Int _                 # built-in 64-bit signed integer token\n    '(' _ Expr ')' _\n}\n\nTerm : @{\n    Term '*' _ Factor     $1 * $4\n    Term '/' _ Factor     $1 / $4\n    Factor\n}\n\nExpr : @{\n    Expr '+' _ Term       $1 + $4\n    Expr '-' _ Term       $1 - $4\n    Term\n}\n\nExpr _ print(\"= \" + $1)   # gives some neat result output\n```\n\n\u003e ```\n\u003e $ tokay examples/expr_from_readme.tok\n\u003e 1 + 2 + 3\n\u003e = 6\n\u003e 7 * (8 + 2) / 5\n\u003e = 14\n\u003e 7*(3-9)\n\u003e = -42\n\u003e ...\n\u003e ```\n\nCalculate the fibonacci numbers from parsed integers:\n\n```tokay\nfibonacci : @n {\n    if n \u003c= 1 n else fibonacci(n - 1) + fibonacci(n - 2)\n}\n\nInt print($1, \"=\u003e\", fibonacci($1))\n```\n\n\u003e ```\n\u003e $ tokay examples/fibonacci2.tok\n\u003e 0\n\u003e 0 =\u003e 0\n\u003e 1\n\u003e 1 =\u003e 1\n\u003e 2\n\u003e 2 =\u003e 1\n\u003e 3\n\u003e 3 =\u003e 2\n\u003e 4\n\u003e 4 =\u003e 3\n\u003e 5\n\u003e 5 =\u003e 5\n\u003e 6\n\u003e 6 =\u003e 8\n\u003e 7\n\u003e 7 =\u003e 13\n\u003e 8\n\u003e 8 =\u003e 21\n\u003e 9\n\u003e 9 =\u003e 34\n\u003e 10\n\u003e 10 =\u003e 55\n\u003e ```\n\n## Documentation\n\nThe Tokay homepage [tokay.dev](https://tokay.dev) provides links to a quick start and documentation. The documentation source code is maintained in a [separate repository](https://github.com/tokay-lang/tokay-docs).\n\n## Building\n\nTo build Tokay, use Rust's standard build tool `cargo`:\n\n```bash\n# for debugging\n$ cargo run -- PARAMS... -- INPUTS...\n\n# for debug build\n$ cargo build\n\n# for optimized release-build\n$ cargo build --release\n```\n\nAdditionally, parts of Tokay's source code are generated using a `Makefile`-based tooling, which is described below.\n\n### Build features\n\nTokay can be built with the following features. Features marked with `(*)` are default features.\n\n- `cbor` (*): Serialize and deserialize VM-programs into and from CBOR binaries (includes `serde`)\n- `cli` (*): Command-line interface (main.rs)\n- `serde`: General `serde` features\n- `static_expression_evaluation` (*): Evaluates static expressions like `1 + 2 + 3` directly to static value 6 during compile-time to reduce amount of resulting operations\n- `use_cbor_parser` (*): Compiles Tokay with integrated binary-encoded CBOR parser for faster compile times (includes `serde` and `cbor`)\n\n### `make builtins` - update Tokay's builtin registry from the Rust sources\n\nGenerate the file `src/_builtins.rs` using `src/_builtins.tok`.\n\n- `make builtins` updates the content of `src/_builtins.rs` using `src/_builtins.tok` from the Rust sources.\n- `make show-builtins` dumps what would be generated by `make builtins`\n- `make reset-builtins` resets `src/_builtins.rs` from git, if something went wrong.\n\n### `make parser-cbor` / `make parser-ast` - update Tokay's parser from `Tokay.tok`\n\nTokay uses a program written in itself (`src/compiler/Tokay.tok`) to generate its own language parser (used by `src/compiler/parser.rs`) incrementally.\n\nThere are currently two parser implementations, the CBOR-based pre-compiled Tokay parser is used as the default.\n\n- `make parser-cbor` updates the content of `src/compiler/_tokay.cbor` from `src/compiler/Tokay.tok`.\n- `make reset-parser-cbor` resets `src/compiler/_tokay.cbor` from git, if something went wrong.\n\nThe old AST-based parser that is compiled internally into a Tokay VM program is still be supported.\u003cbr\u003e\nIn the end, both parsers implement exactly the same program.\n\n- `make parser-ast` updates the content of `src/compiler/_tokay.rs` from `src/compiler/Tokay.tok`.\n- `make show-parser-ast` dumps what would be generated by `make parser-ast`\n- `make reset-parser-ast` resets `src/compiler/_tokay.rs` from git, if something went wrong.\n\n### `make prelude` - update Tokay's prelude from `prelude.tok`\n\nTokay needs the prelude to provide general language features and defaults.\n\n- `make prelude` updates the content of `src/compiler/prelude.rs` from `src/prelude.tok`.\n- `make show-prelude` dumps what would be generated by `make prelude`\n- `make reset-prelude` resets `src/compiler/prelude.rs` from git, if something went wrong.\n\n## Debugging\n\nFor debugging, there are two methods to use.\n\n### Tracing using the `log`-crate\n\nFor Rust standard trace, use the [`env_logger` facilities](https://docs.rs/env_logger/latest/env_logger/). Full trace is only compiled into debug executables, the release version only provides warning level and upwards.\n\n```\n$ RUST_LOG=tokay=debug tokay\n```\n\n### Built-in AST and VM debugger using `TOKAY_DEBUG` and `TOKAY_PARSER_DEBUG`\n\nSet `TOKAY_DEBUG` to a debug level between 1-6. Additionally, `TOKAY_INSPECT` can be set to one or a list of parselet name (-prefixes) which should be inspected in VM step-by-step trace (`TOKAY_DEBUG=6`).\n\n| Level | Mode                              |\n| ----- | --------------------------------- |\n| 0     | No debug                          |\n| 1     | Print constructed AST             |\n| 2     | Print final intermediate program  |\n| 3     | Print compiled VM program         |\n| 4     | Print VM execution trace          |\n| 5     | Print VM stack contents           |\n| 6     | VM opcode debugger                |\n\nView the parsed AST of a program in debug-level 1:\n\n\u003e ```\n\u003e `$ TOKAY_DEBUG=1 cargo run -q -- 'x = 42 print(\"Hello World \" + x)'`\n\u003e main [start 1:1, end 1:33]\n\u003e  sequence [start 1:1, end 1:33]\n\u003e   assign_drop [start 1:1, end 1:8]\n\u003e    lvalue [start 1:1, end 1:3]\n\u003e     identifier [start 1:1, end 1:2] =\u003e \"x\"\n\u003e    value_integer [start 1:5, end 1:7] =\u003e 42\n\u003e   call [start 1:8, end 1:33]\n\u003e    identifier [start 1:8, end 1:13] =\u003e \"print\"\n\u003e    callarg [start 1:14, end 1:32]\n\u003e     op_binary_add [start 1:14, end 1:32]\n\u003e      value_string [start 1:14, end 1:28] =\u003e \"Hello World \"\n\u003e      identifier [start 1:31, end 1:32] =\u003e \"x\"\n\u003e Hello World 42\n\u003e ```\n\n`TOKAY_PARSER_DEBUG` sets the specific debug level for the parser, which is implemented in Tokay itself and is part of the compiler. Only levels \u003e 2 can be recognized here, as the AST of the parser is built into the code.\n\nHere's the VM debugger in action running the simple \"Hello World\"-program:\n\n\u003e ```\n\u003e `$ TOKAY_INSPECT=\"__main__\" cargo run -q -- 'print(\"Hello World\")'`\n\u003e __main__      --- Code ---\n\u003e __main__       000 Offset(Offset { offset: 6, row: 1, col: 7 })\n\u003e __main__      \u003e001 LoadStatic(1)\n\u003e __main__       002 Offset(Offset { offset: 0, row: 1, col: 1 })\n\u003e __main__       003 CallStaticArg((2, 1))\n\u003e __main__      --- Reader ---\n\u003e __main__       offset=Offset { offset: 0, row: 1, col: 1 }\n\u003e __main__       eof=false\n\u003e __main__      --- Globals ---\n\u003e __main__      --- Stack ---\n\u003e __main__      --- Frames ---\n\u003e __main__       000 capture: 0, reader: 0, fuse: None\n\u003e\n\u003e __main__      ip = 1 state = Ok(Push([59d29e639f88] \"Hello World\" (10)))\n\u003e __main__      --- Code ---\n\u003e __main__       000 Offset(Offset { offset: 6, row: 1, col: 7 })\n\u003e __main__       001 LoadStatic(1)\n\u003e __main__       002 Offset(Offset { offset: 0, row: 1, col: 1 })\n\u003e __main__      \u003e003 CallStaticArg((2, 1))\n\u003e __main__      --- Reader ---\n\u003e __main__       offset=Offset { offset: 0, row: 1, col: 1 }\n\u003e __main__       eof=false\n\u003e __main__      --- Globals ---\n\u003e __main__      --- Stack ---\n\u003e __main__       000 [59d29e639f88] \"Hello World\" (10)\n\u003e __main__      --- Frames ---\n\u003e __main__       000 capture: 0, reader: 0, fuse: None\n\u003e\n\u003e Hello World\n\u003e __main__      ip = 3 state = Ok(Push([59d29e498fd8] void (10)))\n\u003e __main__      exit state = Ok(Push([59d29e498fd8] void (10)))\n\u003e ```\n\n## Logo\n\nThe Tokay programming language is named after the [Tokay gecko (Gekko gecko)](https://en.wikipedia.org/wiki/Tokay_gecko) from Asia, shouting out \"token\" in the night.\n\nThe Tokay logo and icon was thankfully designed by [Timmytiefkuehl](https://github.com/timmytiefkuehl).\u003cbr\u003e\nCheck out the [tokay-artwork](https://github.com/tokay-lang/tokay-artwork) repository for different versions of the logo as well.\n\n## License\n\nCopyright © 2025 by Jan Max Meyer, Phorward Software Technologies.\n\nTokay is free software under the MIT license.\u003cbr\u003e\nPlease see the LICENSE file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftokay-lang%2Ftokay","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftokay-lang%2Ftokay","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftokay-lang%2Ftokay/lists"}