Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/ottogroup/flink-spector
Framework for Apache Flink unit tests
https://github.com/ottogroup/flink-spector
apache-flink flink streaming unit-testing
Last synced: about 2 months ago
JSON representation
Framework for Apache Flink unit tests
- Host: GitHub
- URL: https://github.com/ottogroup/flink-spector
- Owner: ottogroup
- License: apache-2.0
- Archived: true
- Created: 2015-12-01T13:11:11.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2019-04-28T19:55:55.000Z (about 5 years ago)
- Last Synced: 2024-04-24T11:41:49.224Z (2 months ago)
- Topics: apache-flink, flink, streaming, unit-testing
- Language: Java
- Homepage:
- Size: 749 KB
- Stars: 208
- Watchers: 18
- Forks: 59
- Open Issues: 14
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Lists
- awesome-flink - Flinnk spector - provides a framework to define unit tests for Apache Flink data flows. The framework executes data flows locally and verifies the output using predefined expectations. (Tests)
- awesome-flink-cn - 【flink-spector】（flink 单元测试框架） - forward.org/wp-content/uploads/2016/07/Alexander-Kolb-Flinkspector-%E2%80%93-Taming-the-squirrel.pdf) (相关优秀项目)
README
# Flinkspector
This project provides a framework to define unit tests for Apache Flink data flows.
The framework executes data flows locally and verifies the output using predefined expectations.
Features include:
- Concise DSL to define test scenarios.
- Powerful matchers to express expectations.
- Test base for JUnit.
- Test stream windowing with timestamped input.
Check out the [**wiki**](https://github.com/ottogroup/flink-spector/wiki) to learn how Flinkspector can assist you in developing Flink jobs.
## Examples
### Minimal:
```java
class Test extends DataSetTestBase {
@org.junit.Test
public myTest() {
DataSet dataSet = createTestDataSet(asList(1,2,3))
.map((MapFunction) (value) -> {return value + 1});
ExpectedRecords expected =
new ExpectedRecords().expectAll(asList(2,3,4))
assertDataSet(dataSet, expected);
}
}
```
### Streaming:
```java
@org.junit.Test
public void testWindowing() {
// Define the input DataStream:
DataStream> testStream =
createTimedTestStreamWith(Tuple2.of(1, "fritz"))
.emit(Tuple2.of(1, "hans"))
.emit(Tuple2.of(1, "heidi"), intoWindow(30, seconds)
.emit(Tuple2.of(3, "peter"), intoWindow(1, minutes)
.repeatAll(times(2))
.close();
// Lets you query the output tuples like a table:
OutputMatcher> matcher =
//define keys for the values in your tuple:
new MatchTuples>("value", "name")
.assertThat("value", is(3))
.assertThat("name", either(is("fritz")).or(is("peter")))
.onEachRecord();
assertStream(someWindowAggregation(testStream), matcher);
}
```
You can find more extensive examples here:
* [DataSet API test examples](flinkspector-dataset/src/test/java/io/flinkspector/dataset/examples).
* [DataStream API test examples](flinkspector-datastream/src/test/java/io/flinkspector/datastream/examples).
## Getting started
### Get the Latest Release:
> Note: The current build works with Flink version 1.8.0.
> There is no Flink 1.4.0 release for scala 2.10 so this has been dropped by flinkspector as well.
Include in your project's pom.xml:
```xml
io.flinkspector
flinkspector-dataset_2.11
0.9.4
```
or for the Flink DataStream API:
```xml
io.flinkspector
flinkspector-datastream_2.11
0.9.4
```
If you want to use assertions you should also include hamcrest:
```xml
org.hamcrest
hamcrest-all
1.3
jar
test
```
### Manual Build:
1. Clone this repo: `git clone https://github.com/ottogroup/flink-spector`.
2. Build with maven: `maven install`.
3. Include in your project's pom.xml:
```xml
io.flinkspector
flinkspector-dataset
0.9.5-SNAPSHOT
```
or for the Flink DataStream API:
```xml
io.flinkspector
flinkspector-datastream
0.9.5-SNAPSHOT
```
## Origins
The project was conceived at the Business Intelligence department of Otto Group.
## Build Status
[](https://travis-ci.org/ottogroup/flink-spector)
## License
Licensed under the [Apache License 2.0](https://github.com/ottogroup/schedoscope/blob/master/LICENSE)