{"id":19545448,"url":"https://github.com/forter/storm-data-contracts","last_synced_at":"2025-04-26T19:32:00.209Z","repository":{"id":19957226,"uuid":"23224094","full_name":"forter/storm-data-contracts","owner":"forter","description":"Storm bolts for validating input and output data","archived":false,"fork":false,"pushed_at":"2023-03-21T19:16:08.000Z","size":237,"stargazers_count":1,"open_issues_count":15,"forks_count":5,"subscribers_count":55,"default_branch":"master","last_synced_at":"2025-04-04T17:22:04.144Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/forter.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-08-22T12:09:59.000Z","updated_at":"2020-09-22T12:00:22.000Z","dependencies_parsed_at":"2023-01-11T20:40:02.920Z","dependency_job_id":null,"html_url":"https://github.com/forter/storm-data-contracts","commit_stats":null,"previous_names":[],"tags_count":46,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/forter%2Fstorm-data-contracts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/forter%2Fstorm-data-contracts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/forter%2Fstorm-data-contracts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/forter%2Fstorm-data-contracts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/forter","download_url":"https://codeload.github.com/forter/storm-data-contracts/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251041479,"owners_count":21527203,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T03:38:51.775Z","updated_at":"2025-04-26T19:31:59.784Z","avatar_url":"https://github.com/forter.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"storm-data-contracts \n====================\n\n[![Build Status](https://img.shields.io/travis/forter/storm-data-contracts.svg)](https://travis-ci.org/forter/storm-data-contracts/)\n\nThis project lets you write Storm Bolts in Java with strict data contracts:\n\nStrongly Typed\n--------------\nBolt input and output are POJOs\n```java\npublic class MyBolt implements IContractsBolt\u003cMyBoltInput,Collection\u003cMyBoltOutput\u003e\u003e {\n    @Override\n    public Collection\u003cMyBoltOutput\u003e execute(MyBoltInput input) {\n        MyBoltOutput output = new MyBoltOutput();\n        if (input.y.isPresent()) {\n            output.z = input.y.get() + input.x;\n        }\n        else {\n            output.z = \"default\" + input.x;\n        }\n        return Lists.newArrayList(output);\n    }\n\n    @Override\n    public Collection\u003cMyBoltOutput\u003e createDefaultOutput() {\n        return Lists.newArrayList();\n    }\n}\n```\n\nInput and Output Data Contracts\n-------------------------------\nSupport Guava Optional and Hibernate Validator for strict data contracts\n```java\npublic class MyBoltInput {\n    @NotNull\n    @Min(0)\n    public Integer x;\n\n    @UnwrapValidatedValue\n    @Pattern(regexp=\"\\\\p{L}*\")\n    public Optional\u003cString\u003e y;\n}\n\npublic class MyBoltOutput {\n    @NotNull\n    public String z;\n}\n```\n\n\nExceptions\n----------\n* All input contract violations are reported to storm.\n* All #execute() exceptions are reported to storm.\n* All output contract violations are reported to storm, and the default output is emitted instead.\n\n\nCaching\n-------\nBaseContractsBoltExecutor supports adding a caching mechanism via inheritance and overriding of\nBaseContractsBoltExecutor#createCacheDAO.\nCached input contracts should be annotated with @Cached annotation and fields which are used as cache keys should be\nannotated with @CacheKey\n```java\n@Cached\npublic class Input {\n\n    @Max(10)\n    @NotNull\n    @CacheKey\n    public Integer input1;\n\n    @Max(10)\n    @UnwrapValidatedValue\n    public Optional\u003cInteger\u003e optionalInput2;\n}\n\npublic class MyCacheDAO\u003cTOutput\u003e implements CacheDAO\u003cTOutput\u003e {\n\n    public Map\u003cMap\u003cString, Object\u003e, TOutput\u003e cache = new HashMap\u003c\u003e();\n\n    @Override\n    public Optional\u003cTOutput\u003e get(Map\u003cString, Object\u003e input) {\n        if (cache.containsKey(input)) {\n            return Optional.of(cache.get(input));\n        }\n        return Optional.absent();\n    }\n\n    @Override\n    public void save(TOutput output, Map\u003cString, Object\u003e inputKey, long startTimeMillis) {\n        cache.put(inputKey, output);\n    }\n}\n\npublic class MyCachedContractBoltExecutor\u003cTInput, TOutput, TContractsBolt extends IContractsBolt\u003cTInput, TOutput\u003e\u003e\n        extends BaseContractsBoltExecutor {\n\n    @Override\n    protected CacheDAO\u003cTInput, TOutput\u003e createCacheDAO(Map stormConf, TopologyContext context) {\n        return new MyCacheDAO\u003cTOutput\u003e();\n    }\n\n}\n```\n@CacheKey supports transformation of input for cache purposes (without changing the input the bolt receives in case of\ncache miss). For example:\n```java\n@Cached\npublic class Input {\n\n    @Max(10)\n    @NotNull\n    @CacheKey(transformers = {LowerCaseTransformer.class})\n    public String input1;\n}\n\npublic class LowerCaseTransformer implements CacheKeyTransformer {\n\n    public Object transform(Object key) {\n        return ((String) key).toLowerCase();\n    }\n}\n\n```\n\n\nCSV driven unit tests \n---------------------\nCSV file header is used to inject data into MyBoltInput and expected MyBoltOutput during unit tests\n\n*src/test/resources/MyTest.csv*\n\n```\ninput.x,input.y,output.z\n1,prefix,prefix1\n2,__NULL__,default2\n```\n\n*src/test/java/MyTest.java*\n\n```java\npublic class MyBoltTest {\n\n    private MyBolt bolt;\n\n    @BeforeClass\n    public void before() {\n        bolt = new MyBolt();\n        bolt.prepare(mock(Map.class),mock(TopologyContext.class));\n    }\n\n    @AfterClass\n    public void after() {\n        bolt.cleanup();\n    }\n\n    //reads from src/main/resources/MyBoltTest.csv\n    @Test(dataProviderClass=TestDataProvider.class, dataProvider=\"csv\")\n    public void testExecute(MyBoltInput input, MyBoltOutput expectedOutput) {\n        Collection\u003cMyBoltOutput\u003e outputs = bolt.execute(input);\n        MyBoltOutput output = Iterables.getOnlyElement(outputs);\n        assertReflectionEquals(expectedOutput, output);\n    }\n    \n    @Test\n    public void testDefaultOutput() {\n        assertTrue(ContractValidator.instance().validate(bolt.createDefaultOutput()).isValid());\n    }\n}\n```\n\nAdding Bolt into a Topology\n---------------------------\n```java\nTopologyBuilder builder = new TopologBuilder();\nbuilder.setBolt(\"myContractsBolt\",new BaseContractsBoltExecutor(new MyContractsBolt()))\n\n```\n\n**input**\n\nBolt expects a pair tuple (such as [id, data]). \nThe second item of the pair is expected to be one of the following:\n* `MyBoltInput` - the expected input type, will be validated by the bolt.\n* `ObjectNode` - a weakly typed object (Jackson parsed JSON object similar to Map). Converted to MyBoltInput and validated.\n* `Map` or `SomeOtherBoltInput` - converted into an `ObjectNode` and then converted into MyBoltInput and validated.\n\nThis behavior can be modified by overriding the BaseContractsBoltExecutor#transformInput() method.\n\n**output**\n\nThe bolt emits a pair tuple (such as [id, data]).\nThe second item of the pair is a MyBoltOutput`\n\nThis behavior can be modified by overriding the BaseContractsBoltExecutor#transformOutput() method:\n```java\npublic class ToMapContractsBoltExecutor\u003cTInput, TOutput, TContractsBolt extends IContractsBolt\u003cTInput, TOutput\u003e\u003e extends BaseContractsBoltExecutor\u003cTInput, TOutput, TContractsBolt\u003e {\n\n    public ToMapContractsBoltExecutor(TContractsBolt contractsBolt) {\n        super(contractsBolt);\n    }\n\n    @Override\n    protected Object transformOutput(Object output) {\n        return ContractConverter.instance().convertContractToMap(output);\n    }\n}\n```\n\nEnrichment Bolts\n-----\nNormally, contract bolts will \"absorb\" any attribute that passes by them. This means that the only attributes available to any bolt connected after a contract bolt will be the attributes specified in the output of that contract bolt.\nOne way around this is doing an old-fashioned join, but this because very hard to maintain if dealing with a large topology.\nA quick solution around this is the use of the `@EnrichmentBolt` annotation, which will indicate to the ContractBoltExecutor that this bolt is in \"upsert\" mode to the attributes map: it will only append (or update, if  already existent) to it and will let the other attributes bypass it for the next bolts to use.\n```java\n@EnrichmentBolt\npublic class MyEnrichmentBolt extends BaseContractBolt\u003cMyInput, MyOutput\u003e {\n    // This bolt will allow attributes not in its input/output pass right through it\n    ....\n}\n```\n\nMaven\n-----\n```\n    \u003cdependencies\u003e\n        \u003cdependency\u003e\n            \u003cgroupId\u003ecom.forter\u003c/groupId\u003e\n            \u003cartifactId\u003estorm-data-contracts\u003c/artifactId\u003e\n            \u003cversion\u003e0.2\u003c/version\u003e\n            \u003cscope\u003ecompile\u003c/scope\u003e\n        \u003c/dependency\u003e\n        \u003c!-- Annotation dependencies --\u003e\n        \u003cdependency\u003e\n            \u003cgroupId\u003ejavax.validation\u003c/groupId\u003e\n            \u003cartifactId\u003evalidation-api\u003c/artifactId\u003e\n            \u003cversion\u003e1.1.0.Final\u003c/version\u003e\n        \u003c/dependency\u003e\n        \u003cdependency\u003e\n            \u003cgroupId\u003eorg.hibernate\u003c/groupId\u003e\n            \u003cartifactId\u003ehibernate-validator\u003c/artifactId\u003e\n            \u003cversion\u003e5.1.2.Final\u003c/version\u003e\n        \u003c/dependency\u003e\n        \u003c!-- testing dependencies --\u003e\n        \u003cdependency\u003e\n            \u003cgroupId\u003ecom.forter\u003c/groupId\u003e\n            \u003cartifactId\u003estorm-data-contracts-testng\u003c/artifactId\u003e\n            \u003cversion\u003e0.2\u003c/version\u003e\n            \u003cscope\u003etest\u003c/scope\u003e\n        \u003c/dependency\u003e\n        \u003cdependency\u003e\n            \u003cgroupId\u003eorg.unitils\u003c/groupId\u003e\n            \u003cartifactId\u003eunitils-core\u003c/artifactId\u003e\n            \u003cscope\u003etest\u003c/scope\u003e\n        \u003c/dependency\u003e\n    \u003c/dependencies\u003e\n    \u003crepositories\u003e\n        \u003crepository\u003e\n            \u003cid\u003eforter-public\u003c/id\u003e\n            \u003cname\u003eforter public\u003c/name\u003e\n            \u003curl\u003ehttp://oss.forter.com/repository\u003c/url\u003e\n            \u003creleases\u003e\n                \u003cchecksumPolicy\u003efail\u003c/checksumPolicy\u003e\n            \u003c/releases\u003e\n            \u003csnapshots\u003e\n                \u003cchecksumPolicy\u003efail\u003c/checksumPolicy\u003e\n                \u003cenabled\u003etrue\u003c/enabled\u003e\n            \u003c/snapshots\u003e\n        \u003c/repository\u003e\n    \u003c/repositories\u003e\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fforter%2Fstorm-data-contracts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fforter%2Fstorm-data-contracts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fforter%2Fstorm-data-contracts/lists"}