https://github.com/streamnative/lightproto
Protobuf compatible code generator
https://github.com/streamnative/lightproto
Last synced: 25 days ago
JSON representation
Protobuf compatible code generator
- Host: GitHub
- URL: https://github.com/streamnative/lightproto
- Owner: streamnative
- License: apache-2.0
- Created: 2024-02-16T04:23:56.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2026-05-07T19:44:08.000Z (2 months ago)
- Last Synced: 2026-05-07T21:34:29.908Z (2 months ago)
- Language: Java
- Size: 529 KB
- Stars: 4
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Notice: NOTICE
Awesome Lists containing this project
README
# LightProto
High-performance Protocol Buffers code generator for Java, optimized for serialization and deserialization speed.
## Features
- **Fastest Java Protobuf SerDe** — Unsafe-based serialization bypasses Netty ByteBuf boundary checks
- **100% wire-compatible** with proto2 and proto3 definitions
- **Zero-copy deserialization** using Netty `ByteBuf` (direct and heap memory)
- **Zero heap allocations** — reusable mutable objects, no Builder pattern overhead
- **Lazy string/bytes deserialization** — decoded only on access
- **Optimized string handling** — single-copy ASCII fast path via `sun.misc.Unsafe`
- **Protobuf-compatible JSON serialization and deserialization** — `toJson()` / `parseFromJson()` methods
- **Protobuf TextFormat (de)serialization** — opt-in `toTextFormat()` / `parseFromTextFormat()` for compatibility with `com.google.protobuf.TextFormat`
- **No runtime dependencies** — generated code is self-contained
- **Maven and Gradle plugins** for seamless build integration
## Usage
### Maven
Add the Maven plugin to your `pom.xml`:
```xml
io.streamnative.lightproto
lightproto-maven-plugin
0.7.3
generate
```
Place `.proto` files in `src/main/proto/` and LightProto will generate Java classes during the
`generate-sources` phase.
### Gradle
Add the plugin to your `build.gradle`:
```groovy
plugins {
id 'io.streamnative.lightproto' version '0.7.3'
}
```
Or using the `buildscript` block:
```groovy
buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath 'io.streamnative.lightproto:lightproto-gradle-plugin:0.7.3'
}
}
apply plugin: 'io.streamnative.lightproto'
```
Place `.proto` files in `src/main/proto/` and LightProto will generate Java classes automatically
before compilation. Optional configuration:
```groovy
lightproto {
classPrefix = '' // prefix for generated class names
singleOuterClass = false // wrap all messages in a single outer class
generateTextFormat = false // also generate protobuf TextFormat (de)serialization methods
protocVersion = '4.34.0' // protoc compiler version
// protocPath = '/usr/local/bin/protoc' // use a local protoc binary
}
```
For Maven, the same options are configured under `` on the plugin:
```xml
io.streamnative.lightproto
lightproto-maven-plugin
true
```
### API Example
LightProto generates mutable, reusable objects instead of the Builder pattern used by Google Protobuf:
```java
// Create and populate
MessageMetadata md = new MessageMetadata();
md.setProducerName("producer-1")
.setSequenceId(12345)
.setPublishTime(System.currentTimeMillis());
md.addProperty().setKey("key1").setValue("value1");
// Serialize to ByteBuf
ByteBuf buf = PooledByteBufAllocator.DEFAULT.buffer();
md.writeTo(buf);
// Deserialize from ByteBuf (zero-copy, lazy strings)
MessageMetadata parsed = new MessageMetadata();
parsed.parseFrom(buf, buf.readableBytes());
// Reuse the object for the next message
md.clear();
md.setProducerName("producer-2")...
```
### JSON Serialization and Deserialization
Every generated message has built-in JSON support compatible with
protobuf's [`JsonFormat`](https://protobuf.dev/programming-guides/json/):
```java
// Serialize to JSON string
String json = md.toJson();
// {"producerName":"producer-1","sequenceId":"12345","publishTime":"1711234567890",...}
// Or write directly to a ByteBuf for zero-copy networking
ByteBuf jsonBuf = PooledByteBufAllocator.DEFAULT.buffer();
md.writeJsonTo(jsonBuf);
// Deserialize from JSON string
MessageMetadata parsed = new MessageMetadata();
parsed.parseFromJson(json);
// Or from a ByteBuf (avoids String allocation)
parsed.parseFromJson(jsonBuf);
// Or from a byte array
parsed.parseFromJson(jsonBytes);
```
The JSON encoding follows protobuf conventions: lowerCamelCase field names, int64 values quoted
as strings, enum values as names, and bytes fields as base64. Unknown fields are silently ignored
during parsing, ensuring forward compatibility.
### Protobuf TextFormat (opt-in)
Set `generateTextFormat = true` (Gradle) or `true`
(Maven) to also emit TextFormat (de)serialization on every message. The output is compatible
with `com.google.protobuf.TextFormat.printer()` and `TextFormat.merge()` for backward
compatibility with existing TextFormat data:
```java
// Serialize to a multi-line, indented TextFormat string
String text = md.toTextFormat();
// producer_name: "producer-1"
// sequence_id: 12345
// publish_time: 1711234567890
// property {
// key: "key1"
// value: "value1"
// }
// Or write to a StringBuilder
StringBuilder sb = new StringBuilder();
md.writeTextFormatTo(sb);
// Deserialize from a String, byte[], or ByteBuf
MessageMetadata parsed = new MessageMetadata();
parsed.parseFromTextFormat(text);
```
Differences from JSON: field names are the original proto snake_case, `int64` values are
**not** quoted, enum values are emitted as bare identifiers, and bytes are written as
quoted strings with C-style escapes. The parser tolerates `# … ` comments, single- or
double-quoted strings, angle-bracket sub-messages (`field <…>`), `[v1, v2]` array syntax for
repeated fields, and ignores unknown fields.
### gRPC Integration
LightProto generates `*Grpc.java` service stubs directly from `service` definitions in `.proto`
files — no extra protoc plugin needed. The generated code follows the same structure as
`protoc-gen-grpc-java` but uses LightProto messages, so you get gRPC with zero-copy
serialization and no Google Protobuf runtime dependency.
Add the gRPC dependency to your project:
```xml
io.grpc
grpc-stub
1.68.0
```
Define a service in your `.proto` file:
```protobuf
syntax = "proto3";
package myapp;
message HelloRequest { string name = 1; }
message HelloResponse { string greeting = 1; }
service Greeter {
rpc SayHello (HelloRequest) returns (HelloResponse);
rpc SayHelloStream (HelloRequest) returns (stream HelloResponse);
}
```
Implement the service by extending the generated `ImplBase`:
```java
class GreeterImpl extends GreeterGrpc.GreeterImplBase {
@Override
public void sayHello(HelloRequest request, StreamObserver responseObserver) {
HelloResponse response = new HelloResponse();
response.setGreeting("Hello, " + request.getName() + "!");
responseObserver.onNext(response);
responseObserver.onCompleted();
}
}
```
Create a client using the generated stubs:
```java
// Blocking stub (unary and server-streaming)
GreeterGrpc.GreeterBlockingStub blocking = GreeterGrpc.newBlockingStub(channel);
HelloRequest request = new HelloRequest();
request.setName("World");
HelloResponse response = blocking.sayHello(request);
// Async stub (all method types including client/bidi streaming)
GreeterGrpc.GreeterStub async = GreeterGrpc.newStub(channel);
async.sayHelloStream(request, new StreamObserver() { ... });
```
The generated stubs support all four gRPC method types: unary, server streaming,
client streaming, and bidirectional streaming.
## Supported Features
| Feature | proto2 | proto3 |
|:---|:---:|:---:|
| Scalar fields (int32, int64, uint32, uint64, sint32, sint64, fixed32, fixed64, sfixed32, sfixed64, float, double, bool) | ✅ | ✅ |
| String fields | ✅ | ✅ |
| Bytes fields | ✅ | ✅ |
| Enum fields | ✅ | ✅ |
| Nested messages | ✅ | ✅ |
| `optional` fields (explicit presence) | ✅ | ✅ |
| `required` fields | ✅ | — |
| Implicit presence (no `has*()`, skip defaults) | — | ✅ |
| `oneof` | ✅ | ✅ |
| `repeated` fields | ✅ | ✅ |
| `repeated` packed encoding | ✅ | ✅ (default) |
| `map` fields | ✅ | ✅ |
| Nested enum / message definitions | ✅ | ✅ |
| Default values | ✅ | — |
| Multiple `.proto` files / `import` | ✅ | ✅ |
| `service` / RPC definitions (gRPC stubs) | ✅ | ✅ |
| JSON serialization and deserialization | ✅ | ✅ |
| TextFormat serialization and deserialization (opt-in) | ✅ | ✅ |
| Extensions | ❌ | — |
| `Any`, `Timestamp`, well-known types | ❌ | ❌ |
| `group` (deprecated) | ❌ | — |
## Performance
### Throughput Comparison (ops/μs, higher is better)
| Benchmark | Google Protobuf | LightProto | Speedup |
|:---|---:|---:|---:|
| **AddressBook** (nested messages, strings) | | | |
| Serialize | 6.4 | 22.6 | **3.5x** |
| Fill + Serialize | 2.2 | 12.8 | **5.9x** |
| Deserialize | 3.8 | 19.2 | **5.1x** |
| **Simple** (small numeric messages) | | | |
| Serialize | 30.8 | 245.6 | **8.0x** |
| Deserialize | 20.0 | 100.9 | **5.0x** |
| **Pulsar MessageMetadata** (strings, properties, batch fields) | | | |
| Serialize | 2.8 | 10.9 | **3.8x** |
| Deserialize | 4.2 | 14.1 | **3.4x** |
| **Pulsar BaseCommand+Send** (nested message, numerics) | | | |
| Serialize | 15.8 | 26.8 | **1.7x** |
| Deserialize | 12.8 | 40.3 | **3.2x** |
### Speedup Chart
```
Speedup over Google Protobuf (x times faster)
1x 2x 3x 4x 5x 6x 7x 8x
| | | | | | | |
AddressBook Ser |████████████████████ 3.5x
AddressBook Fill+Ser |██████████████████████████████████████████ 5.9x
AddressBook Deser |█████████████████████████████████ 5.1x
| | | | | | | |
Simple Ser |█████████████████████████████████████████████████ 8.0x
Simple Deser |█████████████████████████████████ 5.0x
| | | | | | | |
Pulsar MD Ser |██████████████████████ 3.8x
Pulsar MD Deser |███████████████████ 3.4x
| | | | | | | |
Pulsar Cmd Ser |████████ 1.7x
Pulsar Cmd Deser |█████████████████ 3.2x
| | | | | | | |
```
### Raw JMH Output
Click to expand full benchmark output
```
Benchmark Mode Cnt Score Error Units
-- AddressBook (nested messages with strings) --
ProtoBenchmark.protobufSerialize thrpt 3 6.422 ± 0.242 ops/us
ProtoBenchmark.protobufFillAndSerialize thrpt 3 2.190 ± 0.365 ops/us
ProtoBenchmark.protobufDeserialize thrpt 3 3.777 ± 0.253 ops/us
ProtoBenchmark.lightProtoSerialize thrpt 3 22.551 ± 0.355 ops/us
ProtoBenchmark.lightProtoFillAndSerialize thrpt 3 12.827 ± 2.816 ops/us
ProtoBenchmark.lightProtoDeserialize thrpt 3 19.166 ± 1.022 ops/us
-- Simple (small numeric messages) --
SimpleBenchmark.protobufSerialize thrpt 3 30.768 ± 14.107 ops/us
SimpleBenchmark.protobufDeserialize thrpt 3 19.988 ± 0.499 ops/us
SimpleBenchmark.lightProtoSerialize thrpt 3 245.575 ± 87.004 ops/us
SimpleBenchmark.lightProtoDeserialize thrpt 3 100.857 ± 2.221 ops/us
SimpleBenchmark.lightProtoDeserializeReadString thrpt 3 54.111 ± 14.056 ops/us
-- Pulsar MessageMetadata (strings, repeated properties, batch fields) --
PulsarApiBenchmark.protobufSerializeMessageMetadata thrpt 3 2.838 ± 0.580 ops/us
PulsarApiBenchmark.protobufDeserializeMessageMetadata thrpt 3 4.176 ± 0.241 ops/us
PulsarApiBenchmark.lightProtoSerializeMessageMetadata thrpt 3 10.852 ± 0.677 ops/us
PulsarApiBenchmark.lightProtoDeserializeMessageMetadata thrpt 3 14.141 ± 8.862 ops/us
-- Pulsar BaseCommand + CommandSend (nested message, mostly numerics) --
PulsarApiBenchmark.protobufSerializeBaseCommand thrpt 3 15.818 ± 4.591 ops/us
PulsarApiBenchmark.protobufDeserializeBaseCommand thrpt 3 12.771 ± 3.192 ops/us
PulsarApiBenchmark.lightProtoSerializeBaseCommand thrpt 3 26.814 ± 1.107 ops/us
PulsarApiBenchmark.lightProtoDeserializeBaseCommand thrpt 3 40.296 ± 2.816 ops/us
```
### Running Benchmarks
```bash
mvn -B install
java -jar benchmark/target/benchmarks.jar
```
## License
Licensed under the Apache License, Version 2.0.