https://github.com/tyoung3/sw
StreamWork Go Code Generation and Flow-Based-Programming Engine
https://github.com/tyoung3/sw
abstract-syntax-tree bnfc code-generation dataflow dataflow-compiler fbp go graph-editor graph-editor-gui graph-visualization graphviz prototype prototyping
Last synced: about 2 months ago
JSON representation
StreamWork Go Code Generation and Flow-Based-Programming Engine
- Host: GitHub
- URL: https://github.com/tyoung3/sw
- Owner: tyoung3
- License: apache-2.0
- Created: 2019-05-20T11:02:59.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2022-08-24T11:11:08.000Z (over 3 years ago)
- Last Synced: 2024-06-21T11:19:44.487Z (over 1 year ago)
- Topics: abstract-syntax-tree, bnfc, code-generation, dataflow, dataflow-compiler, fbp, go, graph-editor, graph-editor-gui, graph-visualization, graphviz, prototype, prototyping
- Language: C
- Homepage:
- Size: 61.8 MB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE.TXT
- Security: SECURITY.md
Awesome Lists containing this project
README
# STREAMWORK -- SW-0.28.8
Copyright (C) 2019,2020,2021,2022 Thomas W. Young, sw@twyoung.com
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file or its derivitaves except in compliance with the License.
You may obtain a copy of the License at
[Apache License](http://www.apache.org/licenses/LICENSE-2.0)
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
All files in this repo are included in this license.
## StreamWork
Go language code generator.
[](https://github.com/tyoung3/sw/actions/workflows/codeql-analysis.yml)
[](https://goreportcard.com/report/github.com/tyoung3/sw)
[](https://opensource.org/licenses/Apache-2.0)
## Contents
* [Description](#description)
* [Synopsis](#synopsis)
* [Return code](#returncode)
* [**_TestSWL_** program](#testswl)
* [Network Definition](#networkdefinition)
* [Network Definition Files](#networkfiles)
* [QuickStart on Linux](#quickstart)
* [Requirements](#requirements)
* [Installation](#installation)
* [Post Install](#postinstall)
* [Foo Bar test](#foobartest)
* [Building on Linux](#buildingonlinux)
* [StreamWork Language(SWL) Notes](#swlnotes)
* [Stream types and component arguments](#streamtypes)
* [Variables](#variables)
* [Semi-colon commentary](#semicolons)
* [Left Arrow comments](#leftarrow)
* [Collate/Merge Example](#collatemerge)
* [Bash Scripts](#bashscripts)
* [Hello World](#helloworld)
* [**_swgo_** script](#swgo)
* [**_swgraph_** Script](#swgraph)
* [**_gofbp_ script**](#gofbpscript)
* [gofbp Example](#gofbpexample)
* [LOADBAL.SW](#loadbal)
* [Generate a project](#generateaproject)
* [**_sw.sh_** Synopsis](#synopsis)
* [BUGS](#bugs)
* [Home Page](#homepage)
* [Release Notes](#releasenotes)
* [Author](#author)
## Description
StreamWork is a simple system for building complex programs.
StreamWork can generate and test a complete Golang project from a simple
text file. It can also create a nice graph showing the components and
their connections.
By default, StreamWork reads, parses, and analyzes; a simple
SWL language StreamWork network definition(ND) file,
then generates a main Go program.
Options allow you to produce an abstract syntax tree, a GraphViz dot file, or a goFBP main prgram instead.
**_sw_** [-m MODE] [-cfg CONFIGURATION_FILE] [-d DEFAULT_PATH] [ SW_FILE ]
**_sw_** -v
**_sw_** --help
If SW_FILE is omitted, **_sw_** will read from stdin.
```
MODE=
0-GOMODE . Generate a main SW Go program[DEFAULT].
1-ASTMODE . Display the Abstract Syntax Tree
2-GENTREE . Display the Linearized tree
3-GRAPHMODE . Generate Graphviz .dot file code
4-JAVAFBP . Incomplete and untested. Looking for help.
5-PROJECT . Generate **_swgen.sh_** code for a project tree.
7-CMODE . Incomplete and untested. Looking for help.
8-GENGOFBP . Generate main go code for J. P. Morrison's gofbp.
```
## Return code
**_sw_** returns SUCCESS unless it encounters an error, in which case **_sw_** returns FAILURE.
See package documention for component return codes.
## **_TestSWL_** program
**_TestSWL_** syntax checks stdin or a SteamWork network definition file or a SteamWork configuration file.
### synopsis
```
usage: Call with one of the following argument combinations:
--help Display this help message.
(no arguments) Parse stdin verbosely.
(files) Parse content of files verbosely.
-s (files) Silent mode. Parse content of files silently.
```
A StreamWork network definition describes a program as a list of data streams.
Each data stream consists of a source process, a director, and a sink process.
The network definition is a compact text-based representation of a data flow diagram.
StreamWork is concerned primarily with Go language FBP programs,
however, the network definition described here is language agnostic. It is quite possible
to generate corresponding code in other languages from a StreamWork ND.
The script, **_swgo_**, reads and executes a network definition employing **_sw_** to properly launch and connect the processes. By default, **_sw_** generates
and builds a single main Go program which:
* imports sw compatible component package(s) as specified in the network definition,
* launches a goroutine for each process,
* connects these processes via Golang interface channels, and
* waits for all processes to finish.
* Once the processes launched, StreamWork has no subsequent
interaction with them.
On option, **_sw_** will read, parse and interpret a network definition file then create either a:
* a StreamWork Go main program,
* a gofbp Go main program (see github.com/jpaulm/gofbp),
* a GraphViz .dot file(script,( **_swgraph_** will create and display a graph(s) directly)
* a bash script, (**_swproject_** generates a complete GO language project tree and
tests it)
* an abstract syntax tree (a reverse polish notation rendition of the essential language components),
OR
* a linearized tree (a network definition reconstruction).
A network definition file (in the SWL language[see below]) consists of a list of
streams(aka dataflows); and may also contain subnet definitions,
INCLUDE and PREFIX statements, and comments.
INCLUDEd files are also network definitions which may in turn contain
INCLUDE statements. Exceeding 100 levels of includes will
cause program termination.
A stream definition looks something like:
```
(a C "ARG..." ) -ipType> (b D);
or [reversing direction and using default component names]
(E)IN <- OUT.1(F)0 <- (G);
```
and consists of a list of streams. Each stream consists of
a source process, a stream director, a sink process and a terminating semicolon(;). Each process consists of
* a processname
* optional component identifier
* optional string arguments, and
* optional process attributes [surrounded by brackets]. Attributes
consist of KEY=STRING|NUMBER statements, used mainly to enhance
graphic images.
all surrounded by parens, and followed and/or preceded by optional
port identifiers consisting of a port number preceded by an optional port name and period(.). Omitted port identifiers default to zero(0).
The stream director points from the source process to the sink process. A director may
include a type identifier and/or a stream buffer size: e.g. "". If not included, buffer size defaults to zero.
Example with anonymous process names and default components:
```
(_) -int 3> (_); #[integer stream with buffersize of three];
```
In this example, the default components [as specified in a config file, send their process names to the channel or to stdout:
```
(Hello) <- (World);
```
produces:
```
Hello World
```
A process is defined by its process name, its component identifier, component arguments,
and process attributes; all surrounded by parens.
The component identifier, arguments, and attributes will default if omitted.
Process names are not apparent to Go and so need not follow Go naming
rules, just the StreamWork conventions.
A component identifier consists of its import module/package identifier,
a slash, '/', and the component name. Go component names need to be capitalized.
If the path is omitted, a configuration default path is assumed.
If the component name is also omitted, configuration default component names
are assumed for sink, source, and filter(has both inputs and outputs) processes.
Subnet components are identified by ```'``` prefixing its identifier, as in
(A ^SubnetName) ...
Stream directors consist of ```<```, an optional, comma-separated list of type identifiers,
an optional buffersize integer, and ```-```. Example: ```<100-```
The reverse (Ex: `-myDataType,yourType 100>`) is also valid.
Information packets(IPs) are designed as nil(empty) interfaces. The
data type is determined by the sending component. It is possible for
properly coded sink components to handle multiple data types.
A type mis-match will be reported by incompatible receiving
components. Components can be coded to handle any
type, including user-defined types and structures.
Some components can process strings and integers;
other components just a single type; on each receiving port.
**_sw_** versions beginning with v1.0.0 are backward compatible within the same major
version(currently v0). There is no such guarantee with v0.N.N versions, but we will
increase the minor version in such cases. The master github branch may have many commits without creating a new version.
**_sw_** builds a network model in memory, then optionally generates
either
* an abstract syntax tree,
* a linearized tree(a network definition reconstruction),
* a GraphViz .dot file,
* input to **_swproject_** to build a complete working project tree,
* or Go source code from the network model.
* or Go main source code for JFM's gofbp.
C and other languages could also be generated, but this has not been implemented.
JavaFBP is possible, for instance.
Comments and critiques are welcome. Contributors are encouraged.
Please do not submit code before contacting the project. Contact by
e-mailing streamwork@twyoung.com or Discord/gofbp message is preferred to posting a request on Github.
## QuickStart on Linux
### Requirements
* Gcc, etc.
* Graphviz should be installed, but installation may proceed
without it.
* The ...github/tyoung3/StreamWork backend is no longer required.
* **_sw_** is written in C.
### Installation
* cd to any convenient workspace (/usr/local/src for example).
* Download the latest sw-0.28.0.tar.gz file
from https://github.com/tyoung3/sw/build
* Run 'tar -xzf .../sw-0.28.0.tar.gz' to extract source files
* cd sw-0.28.0
* Run ./configure && make check
* Run sudo make install.
**_sw_** and associated scripts will be installed in /usr/local/bin
### Post Install
#### Foo Bar test
mkdir /tmp/x/
* **_echo_** "(Foo) <- (Bar);" | **_sw_** > /tmp/x/fb.go
**_cd_** /tmp/x/
**_go_** mod tidy
* Run: **_go_** run /tmp/fb.go
This is what the **_swgo_** script does:
try **_swgo_** .../nds/fb.sw
```
OUTPUT:
Foo Bar-1
Foo Bar-2
Foo Bar-3
Foo Bar-4
Foo Bar-5
Foo Bar-6
Foo Bar-7
```
* Install autotools, ctags, libyaml-dev, bnfc, bison, and flex
* cd to your project workspace (like /usr/local/src )
* git clone https://github.com/tyoung3/sw.git
* bin/sw.sh build; # Runs autotools including .../sw/bin/configure.
* bin/sw.sh c; # Runs make check in .../sw/src
* bin/sw.sh bw; # Builds Windows/sw.exe. Requires mingw. Run make distclean first.
## StreamWork Language(SWL) Notes
SWL is an unambiguous, context free grammar, making it directly interpretable, without preprocessing.
See the bnfc StreamWork language description file, SWL.pdf.
SWL has few reserved words, making it somewhat natural language
agnostic.
Every effort is made to ensure that all currently valid SWL
statements will remain valid. Should
this prove impractible, the minor version will be incremented.
Once v1.0.0 is reached, the major version will be incremented instead.
Backward conpatibility is enforced for network definition files,
**_sw_**, and published Go modules only.
Linux scripts and Go modules, are separately versioned.
#### Stream types and component arguments
Stream channels can be any Go language type: int, string, struct,
or interface. '_' is shorthand for an interface channel type.
Components with channels of varying types, should group
types to simplify their entry points.
Autojoin and autosplit work only with interface types.
While StreamWork packages a number of components, none of them
are required. It is quite possible to define a network with no
dependencies (other than Go packages) at all.
The only requirement for a component is
an entry point with two or more parameters: wait group, argument string
slice, and a channel for each port.
#### Variables
SWL accepts string($NAME), environment string($_NAME), symbol(&NAME), and numeric(%NAME) variables. See SWL.pdf for acceptable usage.
StreamWork uses the following environment variables:
* $BROWSER -- WWW browser
* $GOPATH -- Go language path
* $DEBUG -- 'Y' will cause some messages to print.
#### Semi-colons
Statements in SWL, are terminated with a semi-colon. Semi-colons in code are like
periods at the end of English statements --
they tell the reader (and the SWL interpreter) when
you have reached the end of a statement. This makes reading the statements easier.
Imagine trying to read a book without any periods (or initial capitals).
Additionally, without semi-colons line breaks become part of the language
definition, leading to awkward, confusing syntax rules; difficult for the
reader to interpret.
#### Left Arrow comments
The, ```<-```, token is employed in order to be consistent with its
usage in the Go language. The ```->``` token is also available:
"(A) -> (B)1 <- (C);" is valid SWL.
Run the following (in Linux):
```
mkdir /tmp/collate
sw < sw/nds/collate.sw > /tmp/collate.go
go run /tmp/collate.go |grep Match0
to produce:
Match0 Int: 5
Match0 Int: 7
Match0 Int: 9
Match0 Int: 11
Match0 Int: 17
Match0 Int: 23
```
```
echo "(HELLO)->(WORLD);" |sw > /tmp/hello.go
go run /tmp/hello.go
Produces:
WORLD HELLO-1
WORLD HELLO-2
WORLD HELLO-3
WORLD HELLO-4
WORLD HELLO-5
WORLD HELLO-6
WORLD HELLO-7
```
```
sw -m 3 < sw/nds/collate.sw | dot -Tsvg > /tmp/collate.svg
```
produces
an image of the collate
network.
## Bash Scripts
### **_swgo_** script
The bash script, **_swgo_**, will generate and run a GO program from a network definition:
```
.../nds/hw.sw
(Hello) Hello.0
wg.Add(1)
var cp1 []chan interface{}
cp1=append(cp1,cs0)
go sw.Gens(&wg,[]string{"World"},cp1) // World
var cp2 []chan interface{}
cp2=append(cp2,cs0)
go sw.Print(&wg,[]string{"Hello"},cp2) // Hello
wg.Wait()
}
Which in turn produces:
Hello World-1
Hello World-2
Hello World-3
Hello World-4
Hello World-5
Hello World-6
```
Another bash script, **_swgraph_**, will display a graphic image of a network definition:
```
.../nds/mvc.sw
(m Model)1 -change> (v View);
(c)2 -update> 1(v);
(v)2 -event> (c Control);
(c)1 -request> (m);
```
**_swgraph_** .../nds/mvc
produces:
for instance.
Processes are colored such that all components in the same package have the same color.
Not shown here, unfortunately, are the tooltips and html references. Arrows are colored according to stream buffersize: 0 - black; 1-green; 2 or more - orange. Coloring rules
are not guaranteed to remain unchanged in future sw versions.
**WARNING:** buffersize is related to deadlock potential in cyclic networks.
The resulting image is an annotated Data Flow Diagram.
### **_gofbp_ script**
**_gofbp_** produces and runs a gofbp program
(see [github.com/jpaulm/gofbp](https://github.com/jpaulm/gofbp) )
from a StreamWork net definition file.
#### gofbp Example (from sw/src/model/tests/gofbp/loadbal.sw)
```
$core = "github.com/jpaulm/gofbp/core";
$trtn = "github.com/jpaulm/gofbp/components/testrtn";
(Sender $trtn/Sender "10 COUNT=10")OUT -load> IN.0(LoadBalance $trtn/LoadBalance );
(LoadBalance)OUT.1 -load> IN(Receiver0 $trtn/DelayedReceiver);
(LoadBalance)OUT.2 -load> IN(Receiver1 $trtn/Receiver);
(LoadBalance)OUT.3 -load> IN(Receiver2 $trtn/Receiver);
```
`gofbp -b loadbal`
produces
and (on console)
```
Input to Receiver: Receiver1 > IP - # 1
Input to Receiver: Receiver1 > IP - # 4
Input to Receiver: Receiver1 > IP - # 6
Input to Receiver: Receiver1 > IP - # 8
Input to Receiver: Receiver2 > IP - # 0
Input to Receiver: Receiver2 > IP - # 3
Input to Receiver: Receiver2 > IP - # 7
Input to DelayedReceiver: Receiver0 > IP - # 2
Input to DelayedReceiver: Receiver0 > IP - # 5
Input to DelayedReceiver: Receiver0 > IP - # 9
```
### Generate a project
**_swproject_** -yaml YAML -cfg ./src/nds/postage.cfg ./src/model/tests/postage.sw
produces the following source tree:
```
├── cmd
│ └── postage.go
├── ComputePostage.go
├── ComputePostage_test.go
├── config.go
├── GetLoc.go
├── GetLoc_test.go
├── GetWeight.go
├── GetWeight_test.go
├── go.mod
├── go.sum
├── internal
│ ├── gen.sh
│ ├── postage.sw
│ └── sw.cfg
├── ipT.go
├── PostageT.go
├── postage.yaml
├── PrintLabel.go
├── PrintLabel_test.go
├── Rates.go
├── Rates_test.go
├── RateT.go
├── testdata
│ └── README
└── WeighT.go
3 directories, 23 files
```
then produces this graph
vets the code, and finally runs it.
The result is a running program ready for your enhancements.
```
sw.sh-1.0.1 USAGE:
b|build . Build project with Autotools.
c . Make check
cl . Show release check list.
d [OPTs] . Switch to docker container.
e . Exit SW shell.
d build [OPTs] . Build SWdemo docker container.
doc . Run and browse Doxygen
j . Graph collate.sw with swgraph
jl . Generate and view locusts.png
p [NAME..] . Generate project(s) named NAME...
poc . Build and run Proof of Concept
rm . View README in /opt/google/chrome/chrome
rc . Build and run Collate program
rl . Run locusts program
s . Enter SW shell. 'e' to exit the shell.
v . Display this script version
x . Edit this script
--help . Display this help
```
**_sw.Gen_** fails unless all three arguments are present.
Stream directors like => and <= are accepted, but treated the same
as -> and <-, except for appearing differently in graphs.
The meaning is reserved for future use, which could
break existing networks.
### Fixed
[Fixed with v0.28.7] Subnet process names must be unique,
but this is not guaranteed. Using the same process name in different
subnet definitions will no longer cause
unpredictable, negative consequences.
(https://sw.twyoung.com)
## Release Notes
### 0.18.0
* Reorganized repo tree structure
* Renamed package swbase to sw and moved code to repo root
* Changed package swutility to import package sw
### 0.18.4
* Fixes for restructure problems
* Fixes to bash scripts
* Fixes to githup builds
### 0.18.5
* More fixes
* Moved .../sw/c to .../sw/src
### 0.18.6
* Created gofbp script
* Added sw/testrtn package
### 0.18.7
* Moved build process to project tree root
### 0.18.8
* Fixed count in swbal.sw
### 0.18.9
* Fix testrtn/Sender
### 0.19.0
* Many fixes to **_swproject_** script
### 0.20.1
* Fix swgraph file search
### 0.20.6
* Incorporate component attributes in SWL for future graphing enhancements.
* Fix fan logic when connection is missing.
### 0.20.7
* Improve gofbp script
* Update README
### 0.20.8
* Created fileio package with GetFile and PutFile and FileT
### 0.20.10
* Provide process attributes.
### 0.21.0
* Can generate projects with or without YAML configurationswpro.
* Fixes to Wrap function.
### 0.21.1
* README updates
* Updated SWL.pdf
### 0.22.0
* Go code generation revamped
* Typed channels partly implemented, possibly breaking networks
with types specified in a director.
### 0.26.6
* Fixed double arrow graph error
### 0.26.7
* Introduced mkdocs
* Attempt to fix go.in yaml Dependabot alert
* Fix build error in swgraph.cswpro
### 0.26.8
* Removed 'default:' error from swgraph.c
## 0.26.9
* Fixed bug with subnet attributes
## 0.26.10
* Fixed maximum subnet level checking
* Introducted mkdocs web site docs
## 0.26.11
* Changed type mismatches from FAILing to WARNING.
## 0.27.0
* Changed package name from sw/linux to sw/os
* Added sw/os/Debug function
## 0.28.0
* Implemented typing of component channels.
* Defined underscore '_' as shorthand for 'interface{}' type.
* Changed component argument from interface slice to interface in SW components with a single port.
* For components with multiple ports; adjacent channels of the same
type are combined into a channel slice.
* Fixed numerous problems with swproject.
## 0.28.1
* Fixed IP type bugs
* Updated docs
## 0.28.2
* Fixed subnet expansion bugs
## 0.28.3
* Fix project generation i/o bug
## 0.28.4
* More project generation fixes
## 0.28.6
* Fix cfg initialization
* Check Wextra warnings.
* Remove some dead code
## 0.28.7
* Prefix subnet process names to make them unique.
## 0.28.8
* Fix go.mod yaml.v3 dependency failure -- 'require stretch..v1.1
* Point package names to github.com.../sw
* Use src/model/sw not bin/sw(removed) in sw.sh/run script
## Author
Tom Young, streamwork@twyoung.com
with thanks to J. Paul Morrison, Phillip W. Young, Sam Watkins,
the contributors to flow-based-programming@googlegroups.com,
and to the developers/maintainers of Go, gcc, Linux/Ubuntu,
BNFC, git, GraphViz, Github and others.