Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/msgbuf/msgbuf

Code generator for GWT-compatible Java data classes suitable for typed client-server messaging
https://github.com/msgbuf/msgbuf

dart data-class gwt java protocol-buffers

Last synced: about 2 months ago
JSON representation

Code generator for GWT-compatible Java data classes suitable for typed client-server messaging

Awesome Lists containing this project

README 

          
# msgbuf

Code generator for GWT-compatible Java data classes suitable for typed client-server messaging.



Inspired by Google's [protocol buffers](https://developers.google.com/protocol-buffers), `msgbuf` provides a code generator that produces data classes out of a concise protocol definition file. 



In contrast to `protobuf`, `msgbuf` supports:

 * Code generation compatible with the [GWT Java-to-Javascript compiler](http://www.gwtproject.org/).

 * Inheritance of data classes.

 * Abstract data classes defining the root of a hierarchy of exchangeable data fragments.

 * Polymorphic data compositions.

 * [Visitor pattern](https://en.wikipedia.org/wiki/Visitor_pattern) for processing polymorphic data structures.

 

`msgbuf` serializes messages in JSON, Binary, and XML formats. For GWT-compatibility it uses a modified `JsonReader`/`JsonWriter` from the [gson library](https://github.com/google/gson) that was abstracted from the unsupported `Reader`/`Writer` Java API.



## Setup with Maven



### Add the MsgBuf runtime library dependency to your project



```xml



    de.haumacher.msgbuf

    msgbuf-api

    1.2.1



```



### Add the MsgBuf generator to your pom.xml



To the `build/plugins` section add:



```xml



    de.haumacher.msgbuf

    msgbuf-generator-maven-plugin

    1.2.1

    

    

        

            generate-protocols

            

                generate

            

        

    



```



Now you are ready to create `*.proto` files in your source folder and build them with `mvn compile`.



### Where to place proto files



Proto files are placed in your Java source folder (`src/main/java/`) inside the directory matching their `package` declaration. The generated Java files are written next to the proto file. For example, a proto file with `package my.app.model;` should be placed at `src/main/java/my/app/model/shape.proto`, and the generated classes will appear in `src/main/java/my/app/model/`.



## Usage

 

The `msgbuf` definition language is an extension of the [proto format](https://developers.google.com/protocol-buffers/docs/proto3) from `protobuf`. A defined message can `extend` another message type, or it can be marked `abstract`. 



Assume, you want to describe shapes in a graphics application, you could define the data types as follows. You may start with an `abstract` shape class, defining coordinates of the origin of its coordinate system:



Add `src/main/java/my/app/model/shape.proto` with the following contents:

```protobuf

package my.app.model;



abstract message Shape {

  int32 xCoordinate;

  int32 yCoordinate;

}

```



Based on that, you create concrete classes for circles and rectangles:



```protobuf

message Circle extends Shape {

  int32 radius;

}



message Rectangle extends Shape {

  int32 width;

  int32 height;

}

```



Finally, you could create a group class that allows combining arbitrary shapes by placing them into a new coordinate system:



```protobuf

message Group extends Shape {

  repeated Shape shapes;

}

```



Passing these definitions to the `msgbuf` compiler gives you a class hierarchy with classes `Shape`, `Circle`, `Rectangle`, and `Group`. You can inspect the generation result in the test package [test.hierarchy](https://github.com/msgbuf/msgbuf/tree/main/de.haumacher.msgbuf.generator/src/test/java/test/hierarchy/data) of the compiler. The source of the example data class definitions can be seen in the [hierarchy.proto](https://github.com/msgbuf/msgbuf/tree/main/de.haumacher.msgbuf.generator/src/test/java/test/hierarchy/data/hierarchy.proto) file.



### Enum types



Enums define a fixed set of named constants. Each constant is terminated with a semicolon. Constants can optionally

have explicit numeric IDs assigned:



```protobuf

enum Corpus {

    UNIVERSAL = 0;

    WEB = 1;

    IMAGES = 2;

}

```



Enums can also be nested inside messages:



```protobuf

message SearchRequest {

    enum Corpus {

        UNIVERSAL = 0;

        WEB = 1;

    }

    Corpus corpus;

}

```



### Nested messages



Messages can be nested inside other messages:



```protobuf

message SearchResponse {

    message Result {

        string url;

        string title;

        repeated string snippets;

    }

    repeated Result results;

}

```



### Transient fields



Fields marked with `transient` are not serialized. They exist only in the in-memory representation:



```protobuf

message A {

    string name;

    transient string cachedValue;

}

```



### Backtick-quoted identifiers



Backticks can be used to escape identifiers that clash with msgbuf keywords:



```protobuf

message NullableValues {

    @Nullable

    int `int`;

    @Nullable

    boolean `boolean`;

}

```



### Primitive type aliases



In addition to the protobuf type names, `msgbuf` supports shorter aliases:

- `int` for `int32`

- `long` for `int64`

- `boolean` for `bool`



### Native JSON value type (`json`)



The `json` type represents an opaque JSON value — any valid JSON structure (object, array, string, number, boolean, or null). This avoids double-serialization when a message needs to carry dynamically-typed data:



```protobuf

message PatchEvent {

    string controlId;

    json patch;

}

```



In JSON format, the value is written natively without string-wrapping:



```json

{"controlId":"c1","patch":{"nodes":{"n1":{"x":100}}}}

```



The generated Java API uses `Object` as the field type. Supported value types are `Map`, `List`, `String`, `Long`, `Double`, `Boolean`, and `null`:



```java

Map patch = new LinkedHashMap<>();

patch.put("nodes", ...);

PatchEvent event = PatchEvent.create()

    .setControlId("c1")

    .setPatch(patch);



// Reading back:

Object value = event.getPatch(); // Map, List, String, Number, Boolean, or null

```



For binary serialization, json values are wrapped in a `JsonValue` message envelope (defined in `de.haumacher.msgbuf.json.value`). For XML, json values are encoded as JSON strings in element text.



## Global protocol options



### `option NoJson`

Disables generation of read and write methods for the JSON format.



### `option NoBinary`

Disables generation of read and write methods for binary format.



### `option NoXml`

Disables generation of read and write methods for XML format.



### `option NoXmlNames`

Disables generation of constants for the XML format.



### `option NoInterfaces`

Disables generation interfaces for data classes. Normally, data classes are represented by a Java interface. This

enables multiple inheritance for data classes. To reduce the amount of generated code, this can be disabled for simple

cases, where no multiple inheritance is required.



### `option NoListener`

Disables generation of listener interfaces and corresponding registration methods. Add this options, if observing 

data classes for changes is not required.



### `option NoReflection`

Disables generation of reflective access methods that allow access to properties through their property names.



### `option NoVisitor`

Disables generation of visitor interfaces and visit methods. 



### `option NoVisitorExceptions`

Produces visitor interfaces that cannot throw declared exceptions.



### `option NoTypeKind`

Suppresses the type kind enumeration for a data class hierarchy.



### `option SharedGraph`

Allows to handle multiple synchronized instances of a data class graph. Each graph can be observed for changes. Changes

generate synchronization messages to keep other instances of the same shared graph up to date. With this option,

a shared graph can be instantiated on a server, transferred to a client while keeping the state in sync when changes

occur on each side.



### `option UnorderedMaps`

Uses `HashMap` instead of `LinkedHashMap` for map-type properties. By default, map properties use `LinkedHashMap` to

preserve insertion order. This option switches to `HashMap` for better performance when insertion order is not important.



Example:

```protobuf

option UnorderedMaps;  // Use HashMap for better performance



message Config {

    map settings;  // Will use HashMap instead of LinkedHashMap

}

```



### `option OpenWorld`

Enables cross-file protocol extension for abstract type hierarchies. With this option, subtypes can be defined in

separate `.proto` files (and separate modules) using `import` and `extends`:



```protobuf

// base-module: events.proto

option OpenWorld;



abstract message Event {

    long timestamp;

}



message TextEvent extends Event {

    string text;

}

```



```protobuf

// extension-module: graph-events.proto

import "../base-module/events.proto";



message GraphPatchEvent extends base.pkg.Event {

    string controlId;

    string patch;

}

```



Extension types are discovered at runtime via `ServiceLoader`. The generator produces a registration class

(e.g. `GraphEventsTypes`) and a `META-INF/services` descriptor automatically when the `-resources` output

directory is configured. On platforms without `ServiceLoader` (e.g. GWT), call `GraphEventsTypes.init()` explicitly.



The base module's `.proto` files should be packaged as resources in its JAR (e.g. in `src/main/resources`).

The Maven plugin automatically resolves imports from compile-scope dependency JARs, so no file system paths

to the base module are needed.



Implications:

- Implies `option NoBinary` (only JSON and XML serialization are supported)

- Cannot be combined with `option NoInterfaces`

- The generated `Visitor` interface includes a `visitDefault()` fallback method for unknown extension types

- Extension types generate their own `Visitor` sub-interface with an `instanceof`-based dispatch



Maven plugin configuration (extension module):

```xml



    ${project.basedir}/src/main/resources



```



Imports are resolved automatically from the compile classpath. If the base module packages its `.proto` files

as resources, no additional configuration is needed beyond declaring the dependency.



CLI usage (standalone, without Maven):

```

java -jar msgbuf-generator.jar -out src/main/java -resources src/main/resources -cp base-module.jar extension.proto

```



The `-cp` option specifies JARs to search for imported `.proto` files. The `-I` option can still be used

for file system include paths.



## Message options



### Mix-in interfaces (`@Operations(...)`)



The data classes can extends mix-in interfaces with operations.



```protobuf

/** The data class */

@Operations("test.operations.DataOperations")

message Data {

  int x;

}

```

```java

/** The mix-in interface with operations on data. */

public interface DataOperations {

    /** Access to the data. */

    Data self();

    

    /** Operation added to data class. */

    default void inc() {

        self().setX(self().getX() + 1);

    }

}



/** Testing the mix-in operation. */

public void testOperations() {

    Data data = Data.create();

    data.inc();

    data.inc();

    assertEquals(2, data.getX());

}

```



## Property options



### `@Nullable`

A property of a primitive type that does not allow `null` values (e.g. `int` and `string`) can be explicitly marked to

allow `null` values.



### `@Singular("item")`

Specifies the singular form for a repeated field, used when generating `addXxx()` and `removeXxx()` methods. This annotation

takes precedence over the automatic pluralization heuristics. Useful for irregular plurals or when the heuristics produce

incorrect results.



Example:

```protobuf

message Container {

    @Singular("person")

    repeated string people;      // Generates addPerson() instead of addPeople()



    @Singular("child")

    repeated string children;    // Generates addChild() instead of addChildren()



    @Singular("datum")

    repeated string data;        // Generates addDatum() instead of addData()

}

```



### `@Name("myProp")`

Sets a custom property name. This name is used in JSON serialization.



### `@XmlName("myProp")`

Sets a custom tag name for XML serialization.



### `@Reverse("otherProp")`

Marks a reference to be the reverse end of the reference with the given name in the target type.



### `@Container`

Marks a reference point to the container of the current object.



### `@Ref`

Marks a reference as cross reference (non-composition). When setting values to fields marked as cross reference, container properties are not updated.



### `@type_id(4711)`

Sets a custom type discriminator ID for binary serialization of polymorphic hierarchies.



### Default values

Fields can specify default values inline using `= value` syntax:



```protobuf

message Config {

    string name = "default";

    int count = 42;

    double ratio = 3.14;

    bool enabled = true;

}

```



### `map` fields

Map-type fields are supported using the `map` syntax:



```protobuf

message Config {

    map settings;

    map counts;

}

```



### XML reference embedding (`@Embedded`)

When serializing data classes to XML, all data fields and references are normally represented by XML tags with the same 

name as the field or reference. By adding the `@Embedded` annotation to a reference, the tag for the reference can be 

omitted. The contents of the reference is placed directly within the tag for the containing element. Care must be taken 

that the tag names for referenced elements do not clash with tag names of other attributes and references of the 

container.



In the following example, a container with contents A and B can be written `` instead of 

wrapping the contents into an extra element as in ``. 

 

```protobuf

message Container {

    @Embedded

    repeated Base contents;

}



abstract message Base {}

message A extends Base {}

message B extends Base {}

```



However, even with the `@Embedded` annotation, the verbose serialization with the wrapping reference element is also 

understood.



## Plugin options



### `option DartLib = "path/to/output.dart";`

Generates a Dart library file containing Dart data classes with JSON serialization support for all messages and enums

defined in the proto file. The path is relative to the generator output directory.



```protobuf

option DartLib = "../lib/protocol.dart";



message MyMessage {

    string name;

    int count;

}

```



## Installation in Eclipse



There is an Eclipse plugin providing a project builder that automatically generates corresponding Java files whenever you create or modify a `*.proto` definition file. To install and enable the plugin with the following steps:



### Add update site



 * Open the dialog `Help > Install new Software`.

 * Enter `msgbuf - https://msgbuf.github.io/msgbuf/update-site/` in the `Work with` field, click the `Add...` button, and acknowledge the addition. 

 * Select the `MsgBuf Project Builder` checkbox and click `Finish`. 

 * Accept the license and the installation of unsigned content.



### Enable the MsgBuf Builder in your project



 * Select your project in the `Package Explorer`.

 * In the context menu, select `Configure > Enable MsgBuf Builder`.



### Test the installation



 * Create a `MyMessage.proto` file in one of your packages in the source folder.

 * Add the package definition and a message declaration.

 * Immediately, when you save your changes, a corresponding `MyMessage` class should appear that can be directly used in your code.



## Features



### Polymorphic JSON serialization



All of the generated data classes have get- and set-methods for their properties. Additionally, each class has methods for writing its contents to JSON format and reading it back:



```java

/** Reads a new instance from the given reader. */

public static Rectangle readRectangle(JsonReader in) throws IOException {

   ...

}



/** Writes this instance to the given output. */

public final void writeTo(JsonWriter out) throws IOException {

   ...

}

```



Note that `JsonReader` and `JsonWriter` use `de.haumacher.msgbuf.io.Reader` and `de.haumacher.msgbuf.io.Writer` instead of `java.io.Reader` and `java.io.Writer`. This is required for GWT compatibility, since `java.io` is not available in GWT. For in-memory string-based serialization, use `StringR` and `StringW`:



```java

// Writing to a JSON string:

StringW out = new StringW();

rectangle.writeTo(new JsonWriter(out));

String json = out.toString();



// Reading from a JSON string:

Rectangle copy = Rectangle.readRectangle(new JsonReader(new StringR(json)));

```



On the server side (where GWT compatibility is not needed), you can use `ReaderAdapter` and `WriterAdapter` from `de.haumacher.msgbuf.server.io` to wrap standard `java.io.Reader`/`java.io.Writer` instances:



```java

// Reading from a java.io.Reader:

Rectangle r = Rectangle.readRectangle(new JsonReader(new ReaderAdapter(javaIoReader)));



// Writing to a java.io.Writer:

rectangle.writeTo(new JsonWriter(new WriterAdapter(javaIoWriter)));

```



In polymorphic hierarchy of classes as defined above, it is not enough for a class to just write its own properties. Consider a `Group` instance from the example above. Its `shapes` list may contain multiple instances of either circles, rectangles, or even nested groups. Therefore, a class in a polymorphic hierarchy not only serializes its properties, but also its type. Reading back such polymorphic instance instantiates the correct class and fills it with its properties.



### Visitor pattern



In a client-server messaging scenario, the server (or client) receives polymorphic messages and must dispatch each one to the appropriate handler. Consider a protocol with different request types:



```protobuf

abstract message Request {

    string sessionId;

}



message LoginRequest extends Request {

    string username;

    string password;

}



message QueryRequest extends Request {

    string query;

    int32 limit;

}



message LogoutRequest extends Request {

}

```



You could use `instanceof` checks to handle each request type, but this is fragile — if a new request type is added to the protocol, the compiler won't warn you about the missing case, leading to silent failures at runtime.



The generated visitor pattern solves this. Each abstract message hierarchy generates a `Visitor` interface with a case for every concrete subtype. When a new message type is added, every visitor implementation fails to compile until the new case is handled, guaranteeing completeness at compile time.



```java

public class RequestHandler implements Request.Visitor {

    @Override

    public Response visit(LoginRequest self, Session session) {

        return authenticate(self.getUsername(), self.getPassword());

    }



    @Override

    public Response visit(QueryRequest self, Session session) {

        return executeQuery(self.getQuery(), self.getLimit());

    }



    @Override

    public Response visit(LogoutRequest self, Session session) {

        session.invalidate();

        return Response.ok();

    }

}



// Dispatching a received message:

Request request = Request.readRequest(new JsonReader(input));

Response response = request.visit(handler, session);

```



The visitor pattern also works for non-messaging use cases. For the shape hierarchy defined above, a renderer could be implemented as follows.



An `abstract` base class provides a `Visitor` interface and a `visit(...)` method accepting such a visitor:



```java

public abstract class Shape {



   /** Visitor interface for the {@link Shape} hierarchy.*/

   public interface Visitor {



      /** Visit case for {@link Circle}.*/

      R visit(Circle self, A arg);



     /** Visit case for {@link Rectangle}.*/

     R visit(Rectangle self, A arg);



     /** Visit case for {@link Group}.*/

     R visit(Group self, A arg);



   }

  

   ...



   /** Accepts the given visitor. */

   public abstract  R visit(Visitor v, A arg);

}

```



Each of the concrete sub-classes implement the `abstract` visit-method by delegating to the corresponding case-method from the `Visitor` interface:



```java

public class Rectangle extends Shape {



   ...



   @Override

   public  R visit(Shape.Visitor v, A arg) {

      return v.visit(this, arg);

   }

}

```



This allows creating e.g. a renderer implementation that handles all concrete types from the shape hierarchy:



```java

public class ShapeRenderer implements Shape.Visitor {

   @Override

   public Void visit(Rectangle self, Graphics2D g2d) {

      g2d.drawRect(self.getXCoordinate(), self.getYCoordinate(), self.getWidth(), self.getHeight());

      return null;

   }



   @Override

   public Void visit(Circle self, Graphics2D g2d) {

      ...

   }



   @Override

   public Void visit(Group self, Graphics2D g2d) {

      for (Shape shape : self.getShapes()) {

         shape.visit(this, g2d);

      }

      return null;

   }

}

```



Having an arbitrary `Shape` instance and a renderer from above, you can render the shape to a `Graphics2D` with the following code:



```java

Shape shape = ...;

ShapeRenderer renderer = ...;

Graphics2D g2d = ...;



shape.visit(renderer, g2d);

```