{"id":16792292,"url":"https://github.com/chrivers/transwarp","last_synced_at":"2025-04-05T09:13:28.835Z","repository":{"id":140207798,"uuid":"69769813","full_name":"chrivers/transwarp","owner":"chrivers","description":"Transwarp compiler - a python3 implementation of a Simple Type Format parser and renderer","archived":false,"fork":false,"pushed_at":"2023-03-07T20:42:32.000Z","size":156,"stargazers_count":2,"open_issues_count":2,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-10T18:12:19.656Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chrivers.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-10-01T23:47:56.000Z","updated_at":"2023-03-05T21:20:17.000Z","dependencies_parsed_at":"2024-07-23T01:50:35.089Z","dependency_job_id":null,"html_url":"https://github.com/chrivers/transwarp","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrivers%2Ftranswarp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrivers%2Ftranswarp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrivers%2Ftranswarp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrivers%2Ftranswarp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chrivers","download_url":"https://codeload.github.com/chrivers/transwarp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247312094,"owners_count":20918344,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T08:45:19.747Z","updated_at":"2025-04-05T09:13:28.808Z","avatar_url":"https://github.com/chrivers.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Transwarp #\n\nTranswarp is a user-friendly compiler, that takes specification (in\nthe form of `*.stf` files) as input, and helps you generate code,\ndocumentation, test cases, or whatever you like! Your documentation\nwill never be outdated again!\n\nYou can use `stf` files to describe protocols, data format, parsers,\nor pretty much anything that deals with structured data. The output\ncan be source code, documentation, or any other user- or\nmachine-readable thing you write a template for.\n\nThe input data format is simple, and very easily extensible. It only\nenforces the most basic structure - you decide the rest!\n\n## Overview ##\n\n```\n+--------------------------+                                +-------------------------+\n|                          |  Templates can load plugins    |                         |\n|  Template files (*.tpl)  \u003c--------------------------------+ Compiler plugins (*.py) |\n|                          |                                |                         |\n+-+------------------------+                                +-------------------------+\n  |\n  | Compiler loads templates  +----------------------+\n  +---------------------------\u003e                      |  Compiler writes output\n                              |  Transwarp Compiler  +-------------------------+\n  +---------------------------\u003e                      |                         |\n  | Compiler loads stf files  +----------------------+                         |\n  |                                                                            |\n+-+------------------------+                                +------------------v------+\n|                          |                                |                         |\n|    Source files (*.stf)  |                                |    Generated content    |\n|                          |                                |                         |\n+--------------------------+                                +-------------------------+\n```\n\nTo use transwarp, you need a specification, and a corresponding\ntemplate (or set of templates). The transwarp compiler parses all\ninput files, and generates the output from your templates.\n\nAfter all input files are parsed, the data is put into a tree\nstructure, which is made available for the template code to inspect\nand transform. Since the templates are completely in charge of the\noutput, there are no requirements for how (or how much) you have to\nuse the available data. More importantly, you can start by making your\nexisting source files into templates, and slowly build conversion\nlogic into them!\n\nThe widely used, well-tested and well-documented [Mako\ntemplate](http://www.makotemplates.org/) system is used for templating.\n\n## Quick start example ##\n\nOkay, so you want to try out transwarp! Let's take a look at\ngenerating parser code and templates for the Artemis Space Bridge\nSimulator game.\n\nHere's what you need:\n\n 0. python 3\n 0. The transwarp compiler (https://github.com/chrivers/transwarp)\n 0. A protocol specification (for example, https://github.com/chrivers/isolinear-chips)\n 0. Templates for that protocol (for example, https://github.com/chrivers/duranium-templates)\n\n```bash\n# download and install transwarp\n$ git clone https://github.com/chrivers/transwarp\n$ cd transwarp\n$ ./setup.py -q install # use sudo if you want to install system-wide\n$ cd ..\n\n# as an example, get artemis protocol spec\n$ git clone https://github.com/chrivers/isolinear-chips isolinear\n\n# as an example, get rust code templates\n$ git clone https://github.com/chrivers/duranium-templates duranium\n\n# to see the complete set of options:\n$ transwarp --help\n```\n\nIf you get a list of commend line arguments, transwarp is installed\ncorrectly. Otherwise, please open an issue so we can fix the problem.\n\nNow we are ready to compile!\n```bash\n# let's see a summary of what transwarp would like to compile\n$ transwarp -D isolinear/protocol -I duranium/templates -L duranium/lib -O output-dir -s\n[*] Will create:\n    output-dir/client/mod.rs\n    output-dir/client/reader.rs\n    output-dir/client/writer.rs\n    output-dir/enums.rs\n    output-dir/maps.rs\n    output-dir/server/mod.rs\n    output-dir/server/object.rs\n    output-dir/server/reader.rs\n    output-dir/server/update.rs\n    output-dir/server/writer.rs\n    output-dir/structs.rs\n    ...\n```\n\nLet's do a quick breakdown of the arguments here:\n\n```-D isolinear-chips/protocol```\n\nThis points transwarp to the protocol specification. From this\ndirectory, all `*.stf` files are loaded. The files are always loaded\nin alphabetical order, for consistency between runs. The templates\nwill have access to each file, as a variable with the filename\n(without extension), prefixed by underscore.\n\nExample: `protocol.stf`, will be available as `_protocol` in the\ntemplates.\n\n```-I duranium-templates/templates```\n\nHere we point transwarp to the templates. This directory is\n*recursively* scanned for templates, to be generated. The default\ntemplate file extension is `.tpl`, but this can be changed with the\n--extension (-e) option. For each input template, an output file will\nbe generated in the output dir, except for the `.tpl` extension.\n\nExample: `server/writer.rs.tpl` will be output to `server/writer.rs`.\n\n```-L duranium-templates/lib```\n\nTemplates can load python code into the templates, which advanced\ntemplates can use to do more serious data processing. The duranium\ntemplates (which generate rust code), use a single such plugin\nmodule. This is not strictly required, but makes the template code\ncleaner, since some repeated functionality can easily be shared in one\nplace.\n\nSimpler templates, such as documentation generators, wouldn't have to\nuse any plugins at all. It is very reasonable to write self-contained\ntemplates.\n\n```-O output-dir```\n\nHere we set the output directory. The relative path of the input file\nfrom the input-dir is used as the output file name.\n\n```-s (summarize)```\n\nTranswarp defaults to showing a diff (like `git diff`), but since the\nfiles have not been initially generated yet, this would produce a very\nlarge amount of output. To avoid this, we use `-s` to produce only a\nlist of changes.\n\n\nIf we agree with the summary, we can write all the files by appending\n`-u` (update mode):\n\n```\n$ transwarp -D isolinear/protocol -I duranium/templates -L duranium/lib -O output-dir -u\n[*] Updated output-dir/server/mod.rs\n[*] Updated output-dir/structs.rs\n[*] Updated output-dir/server/object.rs\n[*] Updated output-dir/server/writer.rs\n[*] Updated output-dir/client/writer.rs\n[*] Updated output-dir/client/reader.rs\n...\n```\n\nTranswarp looks at the modification time of both stf files and\ntemplate files, and tries to only update changed files, so if we run\nit again, we can see we are synchronized:\n\n```\n$ transwarp -D isolinear/protocol -I duranium/templates -L duranium/lib -O output-dir -u\n[*] All templates up-to-date\n```\n\n## Command examples ##\n\nRender templates from \"foo\" into \"bar\":\n\n```\n$ transwarp -I foo -O bar -u\n```\n\nSummarize changes caused by rendering templates from \"src\" into current dir:\n\n```\n$ transwarp -D someproto -I src -s\n```\n\nShow diff caused by rendering templates from current dir (recursively) into \"target\":\n\n```\n$ transwarp -D myspec -O target\n```\n\nEnable verbose mode to explain what is going on behind the scenes:\n\n```\n$ transwarp -D data -I template_dir -O output_dir -v\n```\n\nEnable quiet mode to compile from build scripts without unnecessary output:\n\n```\n$ transwarp -D specfiles -I ../templates -O src/protocol-parser/ -q\n```\n\n## STF specifications ##\n\nThe input data format, the Simple Type Framework (.stf), is\na user-friendly text-based format, designed to be easily\nhuman-readable while still being parsable.\n\nThere are *no keywords* in stf, only a few different types of\nstructure. The stf files describe a data structure, which is then used\nby the template in whatever manner they wish. This leaves you in\ncharge of how to structure and organize your specification data.\n\nAny line that has `##` as the first non-whitespace contents, is\nregarded as a comment line, and will be completely ignored.\n\nThere are exactly 4 kinds of \"things\" in stf files:\n\n - blocks\n - fields\n - types\n - constants\n\nThere are no reserved words, and no identifiers have special\nmeanings. Let's discuss each of them in more detail.\n\n### Constants ###\n\nA constant is simple to define. The syntax is the following:\n\n `name = 0x{hexvalue}`\n\nThe name can consist of all alphanumeric characters, and the integer\nmust be in hex format. Here's an example of a list of constants:\n\n```ruby\nInfusionPCoils        = 0x00\nHydrogenRam           = 0x01\nTauronFocusers        = 0x02\nCarapactionCoils      = 0x03\nPolyphasicCapacitors  = 0x04\nCetrociteCrystals     = 0x05\n```\n\nThat's all you need to know about constants.\n\n### Fields ###\n\nIf you need to store something that isn't an integer constant, you'll\nwant to use a field. The syntax is:\n\n `name : type`\n\nNotice that we use `:` instead of `=`. That's what separates fields\nfrom constants! In the next section, we will take a look at how types\nwork, but here are some examples of fields for now:\n\n```ruby\nindex: u32\nvessel_type_id: u32\nx_coordinate: f32\ny_coordinate: f32\nz_coordinate: f32\npitch: f32\n```\n\nAs you can probably surmise, \"u32\" and \"f32\" are valid types. But\nremember, there *are no* built-in types, keywords, or reserved\nidentifiers in stf! In fact, all simple strings are valid type\nnames. It is up to you (in the templates) to give these names meaning!\n\nBut types can do more, when you need them to. We'll take a closer look\nin the next section.\n\n### Types ###\n\nAs we saw in the previous section, any string of alphanumeric\ncharacters is a valid type: However, types have a powerful feature:\n*type arguments*. Any type can have type argument, and they are\nwritten in angle brackets, like so:\n\n`type\u003carg1, arg2, ...\u003e`\n\nAll type arguments are, themselves, types! This means that your type\narguments can have type arguments, which can have.. you get the\nidea. Types are really just a compact way to describe a tree structure\nwith named nodes.\n\nIf a type is not followed by angle brackets, it has 0 type\narguments. Here are some examples of valid types:\n\n```\nf32\nu32\nstring\nGameMasterButton\u003cu8\u003e\nDriveType\u003cu8\u003e\nmap\u003cShipSystem, f32\u003e\nmap\u003cTubeIndex, TubeStatus\u003cu8\u003e\u003e\n```\n\nAgain, *none* of these examples assume anything about the names given,\nand it is completely up to you to pick the names you would like to\nuse. It is syntactically valid to construct a type called\n`u32\u003cbool\u003cstring\u003cwith_milk\u003e\u003e\u003e`, but it might be harder to come up with\na reasonable explanation for what it *means*. But that's your job ;-)\n\nNow that we have the basic kinds of content explained, let's see how\nwe can group it together to make it more useful.\n\n### Blocks ###\n\nA block is, in essence, a namespace for a group of things. Think of it\nas a container with a name you can refer to.\n\nHere's an example of the block syntax:\n\n```ruby\nexpr name\n    const1 = 0x01\n    const2 = 0x02\n    const3 = 0x03\n    ...\n\n    field1: type1\n    field2: type2\u003cfoo\u003e\n    field3: type3\u003ca, b, c\u003e\n    ...\n```\n\nThere's a few things going on here, so let's take a look at them one at a time.\n\nFirst of all, you start a block by writing 2 names on a line, with any\namount of whitespace in between. The `name` part is used to refer to\nthe block from the templates, and `expr` is any kind of descriptive\nword you like, to give your templates something some more\ninformation. For instance, you might like to refer to a block as an\n`enum`, a `struct` a `class`, or something like that. But you could\njust as well have a `document`, a `protocol` or a `spaceship`.\n\nSecond, we can see that blocks contain any number of constants and\nfields. These follow the exact syntax described earlier, so there\nshould be no surprises here. It is customary (but not enforced) to put\nconstants first, and fields later, although having blocks with both\nconstants and fields is a uncommon in practice.\n\nThird, you need to put *exactly 4 spaces* in front of all content that\ngoes in a block, to mark it as being part of that block. This is a\nlittle rigid, and could maybe be relaxed in a future stf\nspecification, but for now it's always 4 spaces.\n\nFinally, you should know that only the header (expr + name) is\nmandatory - blocks do not have to have any constants or fields at\nall. Also, you can put blocks inside blocks (.. inside blocks,\netc). As you have probably guessed by now, stf is not going to tell\nyou how to structure your data. Of course, you add 4 more spaces for\neach additional level of nesting.\n\nThat was a bit of a mouthful, so let's take a look at some valid blocks:\n\n```ruby\nenum MediaCommand\n    Delete   = 0x00\n    Play     = 0x01\n    Create   = 0x02\n```\n\nHere we have a block with the name `MediaCommand` and the expr\n`enum`. It contains 3 constants, named `Delete`, `Play` and `Create`.\n\nLet's see a slightly more complex example:\n\n```ruby\nstruct PlayerShipUpgrades\n    mask_bytes = 11\n\n    active: map\u003cUpgradeType, bool8\u003e\n    count: map\u003cUpgradeType, i8\u003e\n    time: map\u003cUpgradeType, u16\u003e\n```\n\nThis is a block with the name `PlayerShipUpgrades` and expr\n`struct`. It contains 1 constant, `mask_bytes` and 3 fields, `active`,\n`count` and `time`.\n\nFinally, let's take a look at an example of nested blocks:\n\n```ruby\npacket ServerPacket\n\n    struct AllShipSettings\n        ships: map\u003cShipIndex, ShipV240\u003e\n\n    struct BeamFired\n        id: i32\n```\n\nIn this example, our `ServerPacket` block contains 2 blocks, named\n`AllShipSettings` (which contains the field `ships`) and `BeamFired`\n(which contains the field `id`), respectively.\n\nNow that you know how blocks, fields, types and constants work, you\nare free to combine them in any way you like!\n\n## Advanced features ##\n\nThere are a few advanced features available, designed to make the\nformat easier and more powerful to work with. Let's go through all\nthree of them here, and afterwards you will know everything there is\nto know about the format!\n\n### Doc comments ###\n\nMaking comments in the stf files is not a bad idea, but what if you'd\nlike to use access your descriptions and examples from inside the\ntemplates? No problem, you simply need to mark a comment with `#` (doc\ncomment) instead of `##` (regular comment). Lines marked when `#` will\nNOT be ignored by the compiler, but will instead have their text\ncontents appended to the next item. Here are some examples:\n\n```ruby\n## this line is completely ignored by the compiler\n\n## the next line is a doc comment (but this line is not)\n# this enum is super important, because reasons\nenum VeryImportant\n  Wow = 0x10\n  NotWow = 0x20\n\n# this struct is less important than the enum\nstruct LessImportant\n\n  ## You can also add docs for constants and fields, of course!\n  ## this is demonstrated below\n\n  # compared to VeryImportant, this is definitely not as good\n  KindOfWow: bool\n\n  # remember to do that thing here, because of...\n  ## you can inject normal comments in the middle of doc comments too\n  ## and it will work as expected (be ignored)\n  # ..the reasons for doing it\n  NoBigDeal: u32\n```\n\nTo summarize: Like normal comments, doc comments do nothing on their\nown, but unlike normal comments, they are available to the templates,\nmeaning you can use them to provide descriptions for your items, for\nexample by converting them into comments in whatever target language\nyour templates generate.\n\n## Licence ##\n\nThis project is licences under the GNU [General Public\nLicence](https://www.gnu.org/licenses/gpl-3.0.txt) (GPL) Version 3.\n\n## Etymology ##\n\nIn the Star Trek universe, *Transwarp* refers to speeds faster than\nWarp 10. This is the theoretical warp speed limit, much the like the\nspeed of light is limit in our universe.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrivers%2Ftranswarp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchrivers%2Ftranswarp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrivers%2Ftranswarp/lists"}