{"id":13578228,"url":"https://github.com/kyren/piccolo","last_synced_at":"2025-05-14T14:08:13.543Z","repository":{"id":64238390,"uuid":"128157796","full_name":"kyren/piccolo","owner":"kyren","description":"An experimental stackless Lua VM implemented in pure Rust","archived":false,"fork":false,"pushed_at":"2025-04-28T02:22:40.000Z","size":2911,"stargazers_count":1899,"open_issues_count":29,"forks_count":74,"subscribers_count":43,"default_branch":"master","last_synced_at":"2025-05-08T19:03:49.573Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kyren.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE-CC0","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-04-05T04:11:40.000Z","updated_at":"2025-05-08T04:54:51.000Z","dependencies_parsed_at":"2024-04-25T00:28:40.272Z","dependency_job_id":"7b8cc405-68a7-4ead-a86e-0192dce905a3","html_url":"https://github.com/kyren/piccolo","commit_stats":null,"previous_names":["kyren/piccolo","kyren/luster"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyren%2Fpiccolo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyren%2Fpiccolo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyren%2Fpiccolo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kyren%2Fpiccolo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kyren","download_url":"https://codeload.github.com/kyren/piccolo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254159689,"owners_count":22024564,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T15:01:28.588Z","updated_at":"2025-05-14T14:08:13.524Z","avatar_url":"https://github.com/kyren.png","language":"Rust","funding_links":[],"categories":["Rust","Libraries"],"sub_categories":["Scripting"],"readme":"[![crates.io](https://img.shields.io/crates/v/piccolo)](https://crates.io/crates/piccolo)\n[![docs.rs](https://docs.rs/piccolo/badge.svg)](https://docs.rs/piccolo)\n[![Build Status](https://img.shields.io/circleci/project/github/kyren/piccolo.svg)](https://circleci.com/gh/kyren/piccolo)\n[![Chat](https://img.shields.io/discord/865004050357682246)](https://discord.gg/CSJCVTvgNB)\n\n## piccolo - An experimental stackless Lua VM implemented in pure Rust\n\n**(After *four* years, now UN-paused!)**\n\nProject Goals, in roughly descending priority:\n  * Be an arguably working, useful Lua interpreter.\n  * Be an easy way to *confidently* sandbox untrusted Lua scripts.\n  * Be resilient against DoS from untrusted scripts (scripts should not be able\n    to cause the interpreter to panic or use an unbounded amount of memory and\n    should be guaranteed to return control to the caller in some bounded amount\n    of time).\n  * Be an easy way to bind Rust APIs to Lua safely, with a bindings system that is\n    resilient against weirdness and edge cases, and with user types that can\n    safely participate in runtime garbage collection.\n  * Be pragmatically compatible with some version(s) of PUC-Rio Lua.\n  * Don't be obnoxiously slow (for example, avoid abstractions that would make\n    the interpreter fundamentally slower than PUC-Rio Lua).\n\nYou read more about the design of `piccolo` (and try it out a live REPL!) in\n[this blog post](https://kyju.org/blog/piccolo-a-stackless-lua-interpreter/).\n\n## API Instability\n\nExpect *frequent* pre-1.0 API breakage, this crate is still very experimental.\nAll API incompatible changes will be accompanied by minor version bumps, but\nthese will be very common.\n\n## Safety\n\nThe goal with `piccolo` is to have the majority of it written in safe Rust.\nCurrently, there are a few sources of unsafety, but crucially these sources\nof unsafety are *isolated*. `piccolo` will avoid at all costs relying on\nabstractions which *leak* unsafety, it should always be possible to interact\nwith even low level details of `piccolo` without using `unsafe`.\n\nThe current primary sources of unsafety:\n  * The particularly weird requirements of Lua tables require using hashbrown's\n    low level RawTable API.\n  * Userdata requires unsafety to allow for downcasting non-'static userdata\n    with a safe interface.\n  * The implementation of async `Sequence`s require unsafety to \"tunnel\" the\n    normal `Sequence` method parameters into the future (this is completely\n    hidden from the user behind a safe interface).\n  * Unsafe code is required to avoid fat pointers in several Lua types, to keep\n    `Value` as small as possible and allow potential future smaller `Value`\n    representations.\n\n*(`piccolo` makes no attempt yet to guard against side channel attacks like\nspectre, so even if the VM is memory safe, running untrusted scripts may carry\nadditional risk. With no JIT or callback API to accurately measure time, this\nmight be practically impossible anwyay.)*\n\n## A unique system for Rust \u003c-\u003e GC interaction\n\n*The garbage collector system for `piccolo` is now in its [own repo](\nhttps://github.com/kyren/gc-arena), and also on crates.io. See the README in the\nlinked repo for more detail about the GC design.*\n\n`piccolo` has a real, cycle detecting, incremental garbage collector with\nzero-cost `Gc` pointers (they are machine pointer sized and implement `Copy`)\nthat are usable from safe Rust. It achieves this by combining two things:\n\n1) An unsafe `Collect` trait which allows tracing through garbage collected\n   types that, despite being unsafe, can be implemented safely using procedural\n   macros.\n2) Branding `Gc` pointers by unique, invariant \"generative\" lifetimes to ensure\n   that such pointers are isolated to a single root object, and to guarantee\n   that, outside an active call to `mutate`, all such pointers are either\n   reachable from the root object or are safe to collect.\n   \n## Stackless VM\n\nThe `mutate` based GC API means that long running calls to `mutate` can be\nproblematic. No garbage collection can take place during a call to `mutate`, so\nwe have to make sure to regularly return from the `mutate` call to allow garbage\ncollection to take place.\n\nThe VM in `piccolo` is thus written in what is sometimes called \"stackless\"\nor \"trampoline\" style. It does not rely on the rust stack for Lua -\u003e Rust and\nRust -\u003e Lua nesting, instead callbacks can either have some kind of immediate\nresult (return values, yield values from a coroutine, resume a thread, error),\nor they can produce a `Sequence`. A `Sequence` is a bit like a `Future` in\nthat it is a multi-step operation that the parent `Executor` will drive to\ncompletion. `Executor` will repeatedly call `Sequence::poll` until the sequence\nis complete, and the `Sequence` can yield values and call arbitrary Lua\nfunctions while it is being polled.\n\nAs an example, it is of course possible for Lua to call a Rust callback, which\nthen in turn creates a new Lua coroutine and runs it. In order to do so, a\ncallback would take a Lua function as a parameter, then create a new coroutine\n`Thread` from it and return `SequencePoll:Resume` to run it. The outer main\n`Executor` will run the created `Thread`, and when it is finished it will\n\"return\" via `Sequence::poll` (or `Sequence::error`). This is exactly how the\n`coroutine.resume` Lua stdlib function is implemented.\n\nAs another example, `pcall` is easy to implement here, a callback can call the\nprovided function with a `Sequence` underneath it, and the sequence can catch\nthe error and return the error status.\n\nYet another example, imagine Rust code calling a Lua coroutine thread which\ncalls a Rust `Sequence` which calls yet more Lua code which then yields. Our\nstack will look something like this:\n\n```\n[Rust] -\u003e [Lua Coroutine] -\u003e [Rust Sequence] -\u003e [Lua code that yields]\n```\n\nThis is no problem with this VM style, the inner Rust callback is paused as a\n`Sequence`, and the inner yield will return the value all the way to the top\nlevel Rust code. When the coroutine thread is resumed and eventually returns,\nthe Rust `Sequence` will be resumed.\n\nWith any number of nested Lua threads and `Sequence`s, control will always\ncontinuously return outside the GC arena and to the outer Rust code driving\neverything. This is the \"trampoline\" here, when using this interpreter,\nsomewhere there is a loop that is continuously calling `Arena::mutate` and\n`Executor::step`, and it can stop or pause or change tasks at any time, not\nrequiring unwinding the Rust stack.\n\nThis \"stackless\" style has many benefits, it allows for concurrency patterns\nthat are difficult in some other VMs (like tasklets), and makes the VM much more\nresilient against untrusted script DoS.\n\n## Async `Sequence`s\n\nThe downside of the \"stackless\" style is that writing things as a `Sequence`\nimplementation is much more difficult than writing in normal, straight control\nflow. This is identical to the problem Rust had before proper `async` support,\nwhere it required implementing `Future` manually or using difficult to use\ncombinators. Ideally, if we could somehow implement `Collect` for the generated\nstate machine for a rust `async` block, then we could use rust `async` (or more\ndirectly, unstable Rust coroutines) to implement our `Sequence` state machines.\n\nUnfortunately, implementing a trait like this for a Rust async (coroutine) state\nmachine is not currently possible. HOWEVER, `piccolo` is currently still able to\nprovide a safe way to implement `Sequence` using async blocks by using a clever\ntrick: a shadow stack.\n\nThe `async_sequence` function can create a `Sequence` impl from an `async`\nblock, and the generated `Future` tells the outer sequence what actions to\ntake on its behalf. Since the Rust future cannot (safely) hold GC pointers\n(since it cannot possibly implement `Collect` in today's Rust), we instead\nallow it to hold proxy \"stashed\" values, and these \"stashed\" values point to\na \"shadow stack\" held inside the outer sequence which allows them to be traced\nand collected properly! We provide a `Locals` object inside async sequences\nand this is the future's \"shadow stack\"; it can be used to stash / fetch any\nGC value and any values stashed using this object are treated as owned by the\nouter `Sequence`. In this way, we end up with a Rust future that can store GC\nvalues safely, both in the sense of being sound and not leading to dangling\n`Gc` pointers, but also in a way that cannot possibly lead to things like\nuncollectable cycles. It is slightly more inconvenient than if Rust async blocks\ncould implement `Collect` directly (it requires entering and exiting the GC\ncontext manually and stashing / unstashing GC values), but it is MUCH easier\nthan manually implementing a custom `Sequence` state machine!\n\nUsing this, it is easy to write very complex Rust callbacks that can themselves\ncall into Lua or resume threads or yield values back to Lua (or simply return\ncontrol to the outermost Rust code), while also maintaining complex internal\nstate. In addition, these running callbacks are themselves *proper* garbage\ncollected values, and all of the GC values they hold will be collected if they\nare (for example) forgotten as part of a suspended Lua coroutine. Without async\nsequences, this would require writing complex state machines by hand, so this is\n*critical* for very complex uses of `piccolo`.\n\n## Executor \"fuel\" and VM memory tracking\n\nThe stackless VM style \"periodically\" returns control to the outer Rust code\ndriving everything, and how often this happens can be controlled using the\n\"fuel\" system.\n\nLua and Lua driven callback code *always* happens within some call to\n`Executor::step`. This method takes a `fuel` parameter which controls how long\nthe VM should run before pausing, with fuel measured (roughly) in units of VM\ninstructions.\n\nDifferent amounts of fuel provided to `Executor::step` bound the amount of Lua\nexecution that can occur, bounding both the CPU time used and also the amount of\nmemory allocation that can occur within a single `Executor::step` call (assuming\ncertain rules are followed w.r.t. provided callbacks).\n\nThe VM also now accurately tracks all memory allocated within its inner\n`gc-arena::Arena` using `gc-arena` memory tracking features. This can extend\nto userdata and userdata APIs, and assuming the correct rules are followed in\nexposed userdata and callbacks, allows for accurate memory reporting and memory\nlimits.\n\n*Assuming* that both of these mechanisms work correctly, and *assuming* that all\ncallback / userdata APIs also follow the same rules, this allows for completely\nsandboxing untrusted scripts not only in memory safety and API access but also\nin CPU and RAM usage. These are big assumptions though, and `piccolo` is still\nvery much WIP, so ensuring this is done correctly is an ongoing effort.\n\n## What currently works\n\n* An actual cycle detecting, incremental GC similar to the incremental collector\n  in PUC-Rio Lua 5.3 / 5.4\n* Lua source code is compiled to a VM bytecode similar to PUC-Rio Lua's, and\n  there are a complete set of VM instructions implemented\n* Almost all of the core Lua language works. Some tricky Lua features that\n  currently actually work:\n  * Real closures with proper upvalue handling\n  * Proper tail calls\n  * Variable arguments and returns and generally proper vararg (`...`) handling\n  * Coroutines, including yielding that is transparent to Rust callbacks\n  * Gotos with label handling that matches Lua 5.3 / 5.4\n  * Proper _ENV handling\n  * Metatables and metamethods, including fully recursive metamethods that\n    trigger other metamethods (Not every metamethod is implemented yet,\n    particularly `__gc` finalizers).\n* A robust Rust callback system with sequencing callbacks that don't block\n  the interpreter and allow calling into and returning from Lua without using\n  the Rust stack, and a way to integrate Rust async so that implementing these\n  callbacks is not wildly painful.\n* Garbage collected \"userdata\" with safe downcasting.\n* Some of the stdlib (almost all of the core, fundamental parts of the stdlib\n  are implemented, e.g. things like the `coroutine` library, `pcall`, `error`,\n  most everything that exposes some fundamental runtime feature is implemented).\n* A simple REPL (try it with `cargo run --example interpreter`)\n\n## What currently doesn't work\n\n* A large amount of the stdlib is not implemented yet. Most \"peripheral\" parts\n  of the stdlib are this way, the `io`, `file`, `os`, `package`, `string`,\n  `table`, and `utf8` libs are either missing or very sparsely implemented.\n* There is no support yet for finalization. `gc-arena` supports finalization in\n  such a way now that it should be possible to implement `__gc` metamethods with\n  resurrection and tables with weak keys / values and ephemeron tables fully,\n  but it has not been done yet. Currently, the `__gc` metamethod has no effect.\n* The compiled VM bytecode is in several ways worse than what PUC-Rio Lua will\n  generate and the VM sorely needs optimization (very little effort has gone\n  here so far).\n* Error messages that don't make you want to cry\n* Stack traces\n* Debugger\n* Aggressive optimization and *real* effort towards matching or beating (or\n  even just being within a respectable distance of) PUC-Rio Lua's performance\n  generally.\n* Probably much more I've forgotten about\n\n## What will probably never be implemented\n\nThis is not an exhaustive list, but these are some things which I currently\nconsider *almost definite* non-goals.\n\n* An API compatible with the PUC-Rio Lua C API. It would be amazingly difficult\n  to implement and would be very slow, and some of it would be basically\n  impossible (longjmp error handling and adjacent behavior).\n* Perfect compatibility with certain classes of behavior in PUC-Rio Lua:\n  * PUC-Rio Lua behaves differently on systems depending on the OS, environment,\n    compilation settings, system locale, etc. (In certain versions of PUC-Rio Lua,\n    even the behavior of the *lexer* changes depending on the system locale!)\n    `piccolo` is more or less aiming to emulate PUC-Rio Lua behavior with the\n    \"C\" locale set with the default settings in `luaconf.h` on 64-bit Linux.\n  * The specific format of error messages.\n  * The specific iteration order of tables, and the specific behavior of the\n    length operator (the length operator currently functions correctly and will\n    always return a table \"border\", but for tables that are not sequences,\n    the choice of border that is returned may differ).\n* The `debug` library is unimplemented and much of it will probably never be\n  implemented due to fundamental VM differences.\n* Compatibility with PUC-Rio Lua bytecode\n* `os.setlocale` and other weirdness inherited from C\n* `package.loadlib` and all functionality which allows loading C libraries.\n* Perfectly matching all of the (sometimes quite exotic) garbage collector\n  corner case behavior in PUC-Rio Lua.\n\n## Why is it called 'piccolo'?\n\nIt's a cute little \"pico\" Lua, get it?\n\nIt's not really all that \"pico\" anymore, but it's still a cute little instrument\nyou can safely carry with you anywhere!\n\n## Wasn't this project called something else? Luster? Deimos?\n\nThere was an embarassing naming kerfluffle where I somehow ended up with other\npeople's project names *twice*. They're all the same project. I promise I'm done\nrenaming it.\n\n## License\n\n`piccolo` is licensed under either of:\n\n* MIT license [LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT\n* Creative Commons CC0 1.0 Universal Public Domain Dedication\n  [LICENSE-CC0](LICENSE-CC0) or\n  https://creativecommons.org/publicdomain/zero/1.0/\n\nat your option.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyren%2Fpiccolo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkyren%2Fpiccolo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkyren%2Fpiccolo/lists"}