Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/imatix/gsl

iMatix GSL code generator
https://github.com/imatix/gsl

Last synced: about 1 month ago
JSON representation

iMatix GSL code generator

Awesome Lists containing this project

README 

        

# GSL/4.1 - a Universal Code Generator



## Contents



**[Overview](#overview)**

*  [Contributing](#contributing)

*  [Scope and Goals](#scope-and-goals)

*  [Ownership and License](#ownership-and-license)

 [Ownership and License of generated sources](#ownership-and-license-of-generated-sources)

*  [Building and installing](#building-and-installing)

 [Building on FreeBSD 10](#building-on-freebsd-10)

 [Building on Cygwin](#building-on-cygwin)

*  [This Document](#this-document)



**[Starting with GSL](#starting-with-gsl)**

*  [Hello World](#hello-world)

*  [Templates and Scripts](#templates-and-scripts)

*  [Modeling a Web Site](#modeling-a-web-site)

*  [First Draft](#first-draft)

*  [Inserting Variables](#inserting-variables)

*  [Looping through Trees](#looping-through-trees)

*  [Building the Output](#building-the-output)

*  [Putting it All Together](#putting-it-all-together)

*  [Exercise for the Reader](#exercise-for-the-reader)

*  [Extending the Model](#extending-the-model)



**[Model-Oriented Programming](#model-oriented-programming)**

*  [Becoming a Very Good Programmer](#becoming-a-very-good-programmer)

*  [Tools that Write Software](#tools-that-write-software)

*  [Abstractions and Modeling Languages](#abstractions-and-modeling-languages)

*  [Leverage to Move Mountains](#leverage-to-move-mountains)

*  [Case Study - OpenAMQ](#case-study---openamq)

*  [Other Model-Driven Architectures](#other-model-driven-architectures)

*  [Why use MOP?](#why-use-mop)

*  [A Short History of Code Generation](#a-short-history-of-code-generation)

*  [Myths about Code Generation](#myths-about-code-generation)

*  [The Correctness of Generated Code](#the-correctness-of-generated-code)



**[GSL/4.1 Reference Manual](#gsl41-reference-manual)**

*  [Command-line Syntax](#command-line-syntax)

*  [Concepts](#concepts)

 [Scalar Data Types](#scalar-data-types)

 [Structured Data Types](#structured-data-types)

 [Constants](#constants)

 [Scopes](#scopes)

 [Data Specifiers](#data-specifiers)

 [Expressions](#expressions)

*  [Internals](#internals)

 [Internal Variables](#internal-variables)

 [Template and Script Modes](#template-and-script-modes)

 [Template Lines](#template-lines)

 [Script Lines](#script-lines)

 [Comments](#comments)

 [Ignorecase](#ignorecase)

 [Shuffle](#shuffle)

 [COBOL](#cobol)

 [Line Terminators](#line-terminators)

 [Escape Symbol](#escape-symbol)

 [Substitute Symbol](#substitute-symbol)

 [Arguments](#arguments)

 [Predefined Identifiers](#predefined-identifiers)

*  [Built-In Functions](#built-in-functions)

 [Global Functions](#global-functions)

 [conv](#conv)

 [diag](#diag)

 [environment](#environment)

 [fileio](#fileio)

 [Directory Iteration](#directory-iteration)

 [gsl control](#gsl-control)

 [math](#math)

 [regexp](#regexp)

 [process management](#process-management)

 [script](#script)

 [socket](#socket)

 [string](#string)

 [symb](#symb)

 [thread](#thread)

 [time](#time)

 [XML](#xml)

*  [Script Commands](#script-commands)

 [Output File Manipulation](#output-file-manipulation)

 [Control Structures](#control-structures)

 [Scope Manipulation](#scope-manipulation)

 [Symbol Definition](#symbol-definition)

 [Structured Data Manipulation](#structured-data-manipulation)

 [Script Manipulation](#script-manipulation)

 [Macros and Functions](#macros-and-functions)

 [Miscellaneous](#miscellaneous)

 [Examples](#examples)



## Overview



### Contributing



We use the C4.1 process, see: https://rfc.zeromq.org/spec:22.



### Scope and Goals



GSL/4.1 is a code construction tool.  It will generate code in all languages and for all purposes.  If this sounds too good to be true, welcome to 1996, when we invented these techniques.  Magic is simply technology that is twenty years ahead of its time. In addition to code construction, GSL has been used to generate database schema definitions, user interfaces, reports, system administration tools and much more.



This is the fourth major version of GSL, now considered a stable product, repackaged together with its dependencies for easy building from git.



### Ownership and License



GSL was actively developed by [iMatix Corporation](http://www.imatix.com) from 1995-2005 and is copyright © 1991-2010 iMatix Corporation.  Version 4 was developed as part of the technical infrastructure for the [OpenAMQ](http://www.openamq.org) messaging product.



The authors grant you free use of this software under the terms of the GNU General Public License version 3 or, at your choice, any later version. (GPLv3+). For details see the files `COPYING` in this directory.



#### Ownership and License of generated sources



The copyright of the output of GSL is by default the property of the user or whomever writes the template(s).



### Building and installing



Dependencies:



* pcre package (e.g. libpcre3-dev)



To build from git on a UNIX-like box, and install into `/usr/local/bin`:



    git clone git://github.com/imatix/gsl

    cd gsl/src

    make

    sudo make install



To install it to another location, change the last command to:



    sudo make install DESTDIR=/my/custom/prefix



To show command-line help:



    ./gsl



#### Building on FreeBSD 10



Install GNU Make and GNU Compiler. For example, with `pkg`, `pkg install gmake gcc`. Then edit `src/Makefile` and add "-lm" to `src/Makefile` where you see CCLIBS configured. It may look similar to:



    export CCLIBS = -lpcre



You want to add the math library:



    export CCLIBS = -lpcre -lm



Cd to `src` and run:



    CCNAME=gcc47 gmake

    gmake install



#### Building on Cygwin



Install apt-cyg, a cygwin package manager:



    lynx -source rawgit.com/transcode-open/apt-cyg/master/apt-cyg > apt-cyg

    install apt-cyg /bin



Install git:



    apt-cyg install git



Install gcc's dependencies:



    apt-cyg install wget gcc-g++ make diffutils libmpfr-devel libgmp-devel libmpc-devel libpcre-devel libcrypt-devel



Download, Build and Install gcc:



    wget http://ftpmirror.gnu.org/gcc/gcc-4.9.2/gcc-4.9.2.tar.gz

    tar xf gcc-4.9.2.tar.gz

    mkdir build-gcc && cd build-gcc

    ../gcc-4.9.2/configure --program-suffix=-4.9.2 --enable-languages=c,c++ --disable-bootstrap --disable-shared

    make -j4

    make install



Finally build gsl:



    git clone git://github.com/imatix/gsl

    cd gsl/src

    make

    make install



### This Document



This document was written by Pieter Hintjens in October 2010 based on two 2005 articles on 'model oriented programming', and the GSL reference manual.  This text is originally at README.txt and is built using [gitdown](http://github.com/imatix/gitdown). The text was updated by Gyepi Sam in January 2013 to port documentation from earlier versions and to include more examples.



## Starting with GSL



GSL is an acronym for Generator Scripting Language. And that is what it does. You write scripts in gsl, feed it some data from some XML files and it generates nicely formatted text files for you. These files can be source code, a web site, a recipe book or whatever you like.



Read on to get you started with code generation!



1



### Hello World



Our first step is to make a "hello world" program in GSL. It's quite simple. Make a file called `hello.gsl` that contains one line:



    echo "hello world"



To run this, use the following command:



    gsl hello



GSL is a simple language and you'll not have any difficulty understanding its syntax, except in a few places where it does specialised work. It will take you a little longer to understand what you can do with GSL, but that is the real point of these articles. GSL is not as rich as other scripting languages. It is a code generator scripting language, not a programming tool. It lacks some control structures, and it runs a little slowly.



Initially, GSL looks like any other scripting language. I can write little scripts like this:



    amount = 1000

    year = 2006

    while year < 2026

        amount = amount * 1.05

        year = year + 1

    endwhile

    echo amount



Which calculates the value of my savings account if I were to leave it untouched for twenty years, and the interest rate were steady at five percent. Note these syntax aspects:



* `variable = expression` - Assign a value to a variable

* `while condition... endwhile` - Repeat a block while the condition is true



To run the above program, assuming it was saved in a file called `interest.gsl`, I type this command:



    gsl interest



This executes the script and tells me that if I am really patient, I'll be rich one day. Now I'm going to change this little program to make the same kind of calculation for different amounts, rates, and years. Where do I put these different terms and rates? The answer is, in an XML file. The file is called `deposits.xml`:



    

    

        

        

        

    



We change our script to give the result below.



    .template 0

     for deposit

        year = 1

        accumulated = amount

        while year < years

            accumulated = accumulated * (rate / 100 + 1)

            year = year + 1

        endwhile

        echo "Original amount:" + amount + " becomes: " + accumulated

     endfor

    .endtemplate



Note these syntax aspects:



* `.template 0` - Start script (non-template) block

* `for ` - Repeat block for all instances of child item called `childname`



We will run the new interest calculation script using this command:



    gsl deposits.xml



Note the change of command syntax. We first ran the GSL script. Now we're running the XML file. This is one of GSL's features - you can run XML files as if they were scripts. It's the `script =` setting that does the trick, working much like the hash-bang `#!` command in Linux.



Any GSL script, no matter how simple, works with an XML document loaded into GSL's memory as a data tree. In our first `interest.gsl` script, the data tree contains just this:



    



GSL automatically creates this data tree when we ask it to execute a GSL script.



If, on the other hand, we ask GSL to execute an XML file, it loads this XML file into its data tree. Assuming we also asked for it, it will then execute a GSL script against that XML tree. Technically speaking, GSL searches the root item - which can have any name - for an attribute called "script". We can put attributes into the root item in several ways. One is to simply add them to the XML file, as we did. The other is to place them on the command line, like this:



    gsl -script:interest deposits.xml



All variables that we define and use are stored in the data tree, somewhere. This is the only data structure that GSL scripts work with, and it can get very complex. For many people, understanding this complexity is the most difficult thing about using GSL - hierarchies of data are one of those things most human brains do not handle very well. We use abstractions like XNF to make this simpler, but that is something I'll discuss later.



### Templates and Scripts



GSL uses the term "template" to describe text that is output as generated code. GSL works in two modes - script mode, and template mode. When you execute a GSL script directly, as we did in the first example, GSL starts in script mode. When you execute a GSL script indirectly, through an XML file, as we did in the second example, GSL starts in template mode. Try removing the `.template 0` and `.endtemplate` lines and you'll see what I mean. The script just gets copied to the output stream, the console, by default.



In template mode, GSL commands start with a dot in the first column. In script mode, all lines are assumed to be GSL commands unless they start with `>` (output) in the first column, in which case they are handled as template lines.



Script mode is useful when you are doing a lot of GSL scripting work. Often you need to prepare data, check the XML tree, and so on, before you can start to generate code. Template mode is useful when you want to output a lot of data, or actually want to generate code.



You can mix GSL commands and template code by putting a dot at the start of

lines with GSL commands. Like this:



    .while year < years

    .   accumulated = accumulated * (rate / 100 + 1)

    .   year = year + 1

    .endwhile



I'm now going to generate a little HTML report of the different calculations. The listing below shows the third version of `interest.gsl`:



    .output "deposits.html"

    

    

    So You Want To Be A Millionaire?

    

    

    
So You Want To Be A Millionaire?


    

    Original amount

    Interest rate

    Term, years

    Final amount

    

    .for deposit

    .   year = 1

    .   accumulated = amount

    .   while year < years

    .       accumulated = accumulated * (rate / 100 + 1)

    .       year = year + 1

    .   endwhile

    $(amount)

    $(rate)%

    $(years)

    $(accumulated)

    

    .endfor

    

    

    



Note these syntax aspects:



* `output ` - Start sending output to the filename specified

* $(name) - Insert value of attribute in output text



To produce the HTML report run the same command as before:



    gsl deposits.xml



And then load deposits.html into your browser to see what it looks like.



If you're a web developer with any experience, you will see right away what's

happening. We're generating a web page dynamically, just like a hundred other

web tools. But there are significant differences:



Unlike a dynamic web page, here we explicitly specify the output file

ourselves, using the "output" command. We can output zero, one, or a hundred

different files if we want to.



We're working off a data tree that can be as complex as we want. Each "for"

loop opens a new scope, acting on a set of child entities. A dynamic web page

works off some flat data, coming from the browser or a database. You can make

web pages that work on a hierarchical data set, but it's extra work.



GSL lets you load and navigate XML data so easily that you don't even realize

you're busy. The combination of an explicit script language like GSL plus a

hierarchical XML data tree works well.



### Modeling a Web Site



I'm going to propose a simple abstract model for a web site, as an example. When you understand this example, you'll have a much better idea of how we design new models, so that you can design your own.



To start with, I'll explain how I design a new model, and then I'll take you through the steps of building a code generator that brings it to life.



Our model lets us build simple web sites. A web site is a mixture of different types of document, for instance:



  * HTML pages for the content.

  * JavaScript for menus.

  * CSS style sheets for look and feel.

  * Images for icons and for cosmetics.



And so on. When we make a new model, it's worth asking the question, "how would I make a thousand of these?" I.E., a thousand web sites. Well, we'd have lots of content, which would be different for each web site, possibly with some common parts. The content could definitely be based on standard templates - it's unlikely we'd make each of a thousand sites entirely from scratch.



If we used JavaScript menus, we'd presumably use the same code in each site, changing only the menu content to match the structure of the site.



Most likely we'd use a unique CSS stylesheet for each site, to give each site a unique look and feel, but they could also be based on a standard template.



Finally, the images and icons would be a mixture of standard graphics and customised graphics, depending on how pretty we want each site to look.



Our model is going to be the basis for code generation, that is, the mass production of as much of the above as is reasonable. To do this, we need to make a compact and efficient statement of exactly what is needed to produce each web site.



It's like constructing a thousand houses. It's expensive to design and build each house as a unique thing. It's much cheaper to make a single common plan, and then for each house, state the differences. So one house might have a different roof shape, while another has larger windows, but all houses share the same materials, wall and floor construction, and so on.



When we mass produce something, we're clearly aiming for low cost and consistent , and hopefully high, quality. It's the same with code generation. So, let's get to our web site model. What information do we actually need to specify?



* First, we need to know all the pages in the web site, so that we can build menus.



* Second, we need basic information for each page. Typically, I like to define a title and subtitle, an image (for pretty marketing purposes), and a block of content (which can be raw HTML).



* Third, we some information for all pages - for example, a logo and a copyright statement.



The next step is to sketch a model that can hold this information in a useful way. Remember that we use XML as a modeling language. So, we invent an XML syntax for our model. For each page, I'd like to write something like this:



    

    

    Content HTML goes here

    

    



When I design new XML languages like the above, I use entity attributes to hold single-line properties, and child entities to hold multi-line properties or properties that can occur more than once. It just seems more elegant than putting properties in child entities, since this implies those properties can occur many times. It does not make sense for a page to have more than one name, title, subtitle, or image in our model, so we define these as attributes of the page entity. The iMatix MOP tools use this style very heavily.



Once we've defined a set of pages, how do we tie these together into a web site? Let's use a second model for the overall web site:



    

    

         ...

    ...

    



I've defined a `` tag that breaks the pages into groups. Now let's jump right in and make ourselves a web site. There's no better way to test a model than to try using it. As an example, I'll make a new web site for my local grocer, who has decided, finally, to go on-line.



### First Draft



We'll make the web site as several XML files. This is a design choice. We could also make the site as a single large XML file. It's a trade-off between ease of use (a single file is easier in smaller cases) and scalability (it's not practical to edit a large site with hundreds of pages as a single file).



To start with, we'll define the overall site like this:



    

    

    

        

    

    

        

        

    

    



Note the first line, which defines the file as XML, and the `script` tag, which tells GSL what script to run to process the data. We've defined three pages. Let's write very a simple version of each of these:



Next, we will write three more short XML files as shown below. First the index page:



    

    

    
Close to you


    
We're just around the corner, if you live near by.


    
Always open


    
And if we're closed, just come back tomorrow.


    
Cheap and convenient


    
Much cheaper and easier than growing your own vegetables and fruit.


    

    



Next, the fruit page:



    

    

    
Always fresh


    
Just like it was plucked from the tree last month.


    
Special deal


    
Any five pieces of fruit, for the price of ten!


    
Money back if not satisfied


    
We'll give you your money back if we're not satisfied with it!


    

    



and last the vegetable page:



    

    

    
100% organic vegetables


    
All vegetables made from cardon, oxygen, and hydrogen molecules

    with trace elements.


    
Country fresh style


    
We don't know what that means, but it sounded nice!


    
Unique take-away concept


    
Now you can consume your vegetables in the comfort of your own home.


    

    



Finally, here is the first draft of the web generation script. It does not produce anything, it simply loads the web site data into an XML tree and then saves this (in a file called `root.xml`) that we can look at to see what live data the script is actually working with:



    .###  Since we run the script off the XML file, it starts in

    .###  template mode.

    .template 0

     for section

        for page

            ###  Load XML  data

            xml to section from "$(page.name).xml"

            ###  Delete old  tag

            delete page

        endfor

     endfor

     save root

    .endtemplate



Let's look at what this script does. First, it switches off template mode so we can write ordinary GSL without starting each line with a dot. GSL starts scripts in template mode if they are launched from the XML file. It's useful in many cases but not here. So, we wrap the whole script in `.template 0` and `.endtemplate`.



Second, the script works through each section and page, and loads the XML data for that page. It does this using two commands, `xml` and `delete`. The first loads XML data from a file into the specified scope (``, in this case), and the second deletes the current page (since the loaded data also contains a `` tag).



Finally, the script saves the whole XML tree to a file. If you want to try the next steps you must have installed GSL, as I described in the last article. Run the script like this:



    gsl site



GSL looks for the file called `site.xml`. When the script has run, take a look at `root.xml`. This shows you what we're going to work with to generate the real HTML.



### Inserting Variables



When we generate output, we insert variable values into the generated text. This is very much like using shell variables.



GSL does automatic case conversion on output variable. This is very useful when we generate programming languages. For example, the $(name) form outputs a variable in lower case:



    output "$(filename).c"



The $(NAME) form outputs the same value in uppercase:



    #if defined ($(FILENAME)_INCLUDED)



And the $(Name) form outputs the variable in title case, i.e. the first letter is capitalised:



    ###################  $(Filename)   #################



One side-effect of automatic case conversion is that we'll often get variables converted to lower case simply because we used the $(name) form. If we don't want a variable to be automatically case converted, we use this form: $(name:). This is also called the 'empty modifier'.



A second side-effect of automatic case conversion is that variable names are not case sensitive. By default GSL ignores the case of variable names so that $(me) and $(ME) refer to the same variable.



But putting empty modifiers in every variable expansion gets tiresome, and GSL

lets us switch off automatic case conversion, using this instruction:



    ignorecase = 0



This tells GSL, "variable names are case sensitive, and do not convert variable values on output".



### Looping through Trees



In our first draft we loaded each page into the XML tree and deleted the original page definition. That was this text:



    for section

        for page

            xml to section from "$(page.name).xml"

            delete page

        endfor

    endfor



To generate output for each page, we're going to iterate through the sections one more time. Since we're deleting old `` entities and loading new ones from the XML definitions, we need to iterate through the sections and pages over again. This is the code that generates the output for each page:



    for section

        for page

            include "template.gsl"

        endfor

    endfor



The include command executes GSL code in another file. We're going to do all the hard work in a separate file, which I've called `template.gsl`, so that it's easy to change the HTML generation independently from the top-level GSL code. This is good practice for several reasons:



It's nice, in larger projects, that each big code generation task sits in its own file where it can be owned by a single person.



We can add more templates - to produce other types of output - for the same model very easily and safely.



And you'll see in later examples that we tend to write a single GSL file for each output we want to produce. In XNF - the tool we use for larger-scale code generation projects - these scripts are called "targets".



### Building the Output



The HTML template looks like this:



    .template 1

    .echo "Generating $(page.name) page..."

    .output "$(page.name).html"

    

    

       ...

    

    .endtemplate



Most of it is fairly straight-forward, though you do need to understand how XHTML and CSS work (and I'm not going to explain that here).



* The echo command tells the user what's going on. It's polite to do this, although in realistic cases we'll also let the user suppress such reports using a 'quiet' option.



* The output command creates the HTML page.



* The text `` to `