Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/bluejoe2008/hippo-rpc
Hippo Transport Library enhances spark-commons with easy stream management & handling
https://github.com/bluejoe2008/hippo-rpc
kraps rpc spark stream
Last synced: about 2 months ago
JSON representation
Hippo Transport Library enhances spark-commons with easy stream management & handling
- Host: GitHub
- URL: https://github.com/bluejoe2008/hippo-rpc
- Owner: bluejoe2008
- License: bsd-2-clause
- Created: 2020-02-23T08:06:04.000Z (almost 5 years ago)
- Default Branch: hippo-rpc_2.12
- Last Pushed: 2021-04-26T19:59:09.000Z (almost 4 years ago)
- Last Synced: 2024-11-30T17:47:58.838Z (2 months ago)
- Topics: kraps, rpc, spark, stream
- Language: Scala
- Homepage:
- Size: 434 KB
- Stars: 2
- Watchers: 3
- Forks: 1
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
README
# hippo-rpc
Hippo Transport Library enhances spark-commons with easy stream management & handling
[](https://github.com/bluejoe2008/hippo-rpc/issues)
[](https://github.com/bluejoe2008/hippo-rpc/network)
[](https://github.com/bluejoe2008/hippo-rpc/stargazers)
[](https://github.com/bluejoe2008/hippo-rpc/blob/master/LICENSE)
```
,.I ....
... ZO.. .. M .
...=. .,,.
.,D ..?...O.
..=MD~,.. ., .O . O
.., +I. . .,N, ,$N,,...
O. .. .~.+. . N, .
7.,, . 8. .. ... ,O.
I.DMM,. .M .O ,D
...MZ . ~. .... ..N.. :
? .I. ,.. .. ,
+. ,MM= ..Z. .,. .MDMN~$
.I. .MMD ..M . .. =.. :. . ..
.,M .... .Z. . +=. . ..
~M~ ... 7D... .=~. . . .
..$Z... ...+MO.. .M .
.M. ,. .I .?. ..
.~ .. Z=I7.. .7. .ZM~+N.. ..
.O D . , .M ...M . . .: .
. NNN.I....O.... .. M:. .M,=8..
....,...,. .. ... ..
```
## build & install
```
mvn clean compile install
```
## using hippo-rpc
add repository in `pom.xml`:
```
org.grapheco
hippo-rpc
0.1.2
```
`HippoServer` enhances TransportServer with stream manager(open, streaming fetch, close)
```
val server = HippoServer.create("test", new HippoRpcHandler() {
...
}, 1224)
```
`HippoClient` enhances TransportClient with stream request and result boxing (as Stream[T])
```
val client = HippoClient.create("test", "localhost", 1224)
```
more examples, see
## methods of HippoClient
* `ask[T](message: Any, consumeResponse: (ByteBuffer) => T): Future[T]`
asks for an response, `message` as request, use `consumeResponse` to parse response message
* `askWithBuffer[T](message: Any, extra: ByteBuf*): Future[T]`
asks for an response, sends `message` and `extra` as request
* `getInputStream(request: Any, waitStreamTimeout: Duration): InputStream`
gets for an `InputStream`, e.g, stream of a remote file
* `getChunkedStream[T](request: Any, waitStreamTimeout: Duration): Stream[T]`
gets results as a `Stream`, e.g, results of a SQL execution
* `getChunkedInputStream(request: Any, waitStreamTimeout: Duration): InputStream`
gets for an `InputStream` chunk by chunk, e.g, stream of a remote file
## using hippo-rpc with spark-rpc
`spark-rpc`, or `kraps-rpc` uses a messaging mechanism to improve the performance of RPC calling.
`HippoRpcEnvFactory` is provided to enable RPC calls on `kraps-rpc`, and stream handling on `hippo-rpc`.
`HippoRpcEnv` enhances `NettyRpcEnv` with stream handling functions, besides RPC messaging
usage of `HippoRpcEnv` is like that of `NettyRpcEnv`:
```
rpcEnv = HippoRpcEnvFactory.create(config)
val endpoint: RpcEndpoint = new FileRpcEndpoint(rpcEnv)
rpcEnv.setupEndpoint("endpoint-name", endpoint)
rpcEnv.setRpcHandler(new MyRpcHandler())
...
...
val endPointRef = rpcEnv.setupEndpointRef(RpcAddress(...), "...");
```
to provide a customized `HippoRpcHandler`, a set of methods will be implemented:
* `openCompleteStream`: provides a stream for given request
* `openChunkedStream`: provides a chunkable stream for given request
* `receiveWithBuffer`: defines how to respond on received request buffer
more examples, see
## methods of HippoEndpointRef
* `ask[T](message: Any): Future[T]`
asks for an response, `message` as request
* `ask[T](message: Any, timeout: RpcTimeout): Future[T]`
asks for an response, with max timeout limit
* `ask[T](message: Any, timeout: Duration): Future[T]`
asks for an response, with max timeout limit
* `send(message: Any): Unit`
sends a message
* `ask[T](message: Any, consumeResponse: (ByteBuffer) => T): Future[T]`
asks for an response, `message` as request, use `consumeResponse` to parse response message
* `askWithBuffer[T](message: Any, extra: ByteBuf*): Future[T]`
asks for an response, sends `message` and `extra` as request
* `getInputStream(request: Any, waitStreamTimeout: Duration): InputStream`
gets for an `InputStream`, e.g, stream of a remote file
* `getChunkedStream[T](request: Any, waitStreamTimeout: Duration): Stream[T]`
gets results as a `Stream`, e.g, results of a SQL execution
* `getChunkedInputStream(request: Any, waitStreamTimeout: Duration): InputStream`
gets for an `InputStream` chunk by chunk, e.g, stream of a remote file
## dependencies
* `spark-commons`: spark common library, https://github.com/apache/spark
* `kraps-rpc`: A RPC framework leveraging Spark RPC module, https://github.com/neoremind/kraps-rpc
## licensing
`hippo-rpc` is licensed under the BSD 2-Clause "Simplified" License.