{"id":13739903,"url":"https://github.com/olacabs/fabric","last_synced_at":"2026-02-05T23:03:13.078Z","repository":{"id":12488259,"uuid":"71972621","full_name":"olacabs/fabric","owner":"olacabs","description":"Realtime stream processing framework for apache mesos / marathon","archived":false,"fork":false,"pushed_at":"2023-04-13T17:20:38.000Z","size":271,"stargazers_count":55,"open_issues_count":7,"forks_count":37,"subscribers_count":17,"default_branch":"develop","last_synced_at":"2025-05-08T19:46:05.853Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/olacabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-10-26T06:21:42.000Z","updated_at":"2025-04-03T16:43:59.000Z","dependencies_parsed_at":"2024-11-19T02:02:23.064Z","dependency_job_id":null,"html_url":"https://github.com/olacabs/fabric","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/olacabs/fabric","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olacabs%2Ffabric","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olacabs%2Ffabric/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olacabs%2Ffabric/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olacabs%2Ffabric/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/olacabs","download_url":"https://codeload.github.com/olacabs/fabric/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/olacabs%2Ffabric/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29137754,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-05T23:02:30.544Z","status":"ssl_error","status_checked_at":"2026-02-05T23:02:24.945Z","response_time":65,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T04:00:39.196Z","updated_at":"2026-02-05T23:03:13.054Z","avatar_url":"https://github.com/olacabs.png","language":"Java","funding_links":[],"categories":["大数据","Frameworks"],"sub_categories":["微服务框架","Data Processing"],"readme":"## Fabric - A real-time stream processing framework \u003cbr\u003e\n\n### What?\nA scalable, practical and safe real-time computation framework designed for easy operability and extension.\u003cbr\u003e\n\nFabric is proven to work very well for:\u003cbr\u003e\n- High velocity multi-destination event ingestion with guaranteed persistence.\u003cbr\u003e\n- Rules/Filter based real-time triggers for advertising/broadcast\u003cbr\u003e\n- Online Fraud detection\u003cbr\u003e\n- Real-time pattern matching\u003cbr\u003e\n- Basic Streaming analytics\u003cbr\u003e\n\n### Why?\n* Highly scalable and guaranteed availability using battle-tested clustering capabilities provided by Apache Mesos and Marathon\n* Framework level guarantees against message loss, support for replay, multiple sources and complex tuple trees.\n* Event batching is supported at the core level.\n* Source level event partitioning used as unit for scalability.\n* Uses capabilities provided by docker to ensure strong application and partition level isolation guarantees\n* Supports multiple topology types\n\t* Handcrafted topologies (ala Apache Storm) packaged as docker containers\n\t* Configuration driven, dynamically generated topologies that are created on the fly\n* Complete separation of specification from the code, so that minimal code changes are required to update running topologies and deploy new ones.\n* On the fly topology creation and deployment by dynamically assembling topologies using components directly from artifactory.\n* Inbuilt support for custom metrics and custom code level healthchecks to catch application failures right when they happen.\n* Framework level time based aggregation support with completely independent processor level clock pulse generators.\n* Single JVM processing per partition effectively eliminating message transfers over the wire, thereby providing very high throughputs upto 150k events per second per partition.\n* Easy to write components(processors and sources) without getting bogged down by the complexities of stream processing. The compute framework abstracts out the threading model and provides a simplified single-threaded code model (ala NodeJS), thereby eliminating the need to implement complex threading models.\n* Provides capabilities to create processors that can be easily configured from outside.\n* Inherent support for Jackson based json serialization and deserialization thus allowing events to have nested structure and be processed easily using common libraries like json-path.\n\n## Getting Started\n### Glossary\n**Computation**: A computation is a directed acyclic graph which describes the flow of data and the computations done on them. In this graph, the nodes are the components written by users. Nodes can be of two types, *SOURCE* and *PROCESSOR*. A single instance of a topology runs in a single JVM. A computation consists of sources and processors written by users linked together according to a json specification. All sources written by users must extend *PipelineSource*. All processors written by users must extend *StreamingProcessor* or *ScheduledProcessor*.\u003cbr\u003e\n**EventSet**: An event set is a collection of events. An event set is the basic transmission unit within the computation.\u003cbr\u003e\n**Source**: A source is a component that ingests event sets into the computation. A source is responsible for managing the state of the events ingested into the computation. Events can be in acknowledged or unacknowledged state.\u003cbr\u003e\n**Processor**: A processor is a component that performs some computation on an incoming event set and emits an outgoing event set. A processor can be of two types, *Streaming Processor* and *Scheduled Processor*.\u003cbr\u003e\n**Streaming Processor**: A Streaming Processor is a processor that is triggered whenever and event set is sent to the processor.\u003cbr\u003e\n**Scheduled Processor**: A Scheduled Processor is a processor which is triggered whenever a fixed period of time elapses in a periodic fashion.\u003cbr\u003e\n\n### Walkthrough \u003cbr\u003e\nLet’s write a word count computation that processes a list of sentences and outputs words frequency counts.\u003cbr\u003e\nWe need three components for this computation:\u003cbr\u003e\n1. A source that generates random sentences - RandomSentenceSource\u003cbr\u003e\n2. A processor that splits the sentences by space into its constituent words - SplitterProcessor\u003cbr\u003e\n3. A processor that outputs the word frequency counts at regular intervals - WordCountProcessor\u003cbr\u003e\n\n#### RandomSentenceSource.java\n```\n// Add this annotation for registering the source with the metadata server\n@Source(\n        namespace = \"global\",\n        name = \"random-sentence-source\",\n        version = \"0.1\",\n        description = \"A source that generates random sentences from a pool of sentences\",\n        cpu = 0.1,\n        memory = 64,\n        requiredProperties = {},\n        optionalProperties = {\"randomGeneratorSeed\"}\n)\npublic class RandomSentenceSource implements PipelineSource {\n\n    Random random;\n    String[] sentences = {\n            \"A quick brown fox jumped over the lazy dog\",\n            \"Life is what happens to you when you are busy making other plans\",\n            \"Mama always said that life is a box of chocolates\",\n            \"I am going to make you an offer you cannot refuse\",\n            \"I am speaking to a dead man on the other side of the phone\",\n            \"The path of the righteous man is beset on all sides by the inequities of the selfish and the tyranny of evil men\"\n    };\n\n    @Override\n    public void initialize(final String instanceName,\n            final Properties global,\n            final Properties local,\n            final ProcessingContext processingContext,\n            final ComponentMetadata componentMetadata) throws Exception {\n        // this method is called to initialize the source\n        // use this utility method to read properties passed\n        int seed = ComponentPropertyReader\n                .readInteger(local, global, \"randomGeneratorSeed\", instanceName, componentMetadata, 42);\n        random = new Random(seed);\n    }\n\n    @Override\n    public RawEventBundle getNewEvents() {\n        // this method is called to get new events\n        return RawEventBundle.builder()\n                .events(getSentences(5).stream()\n                        .map(sentence -\u003e Event.builder().id(random.nextInt()).data(sentence.toLowerCase()).build())\n                        .collect(Collectors.toCollection(ArrayList::new)))\n                .meta(Collections.emptyMap())\n                .partitionId(Integer.MAX_VALUE)\n                .transactionId(Integer.MAX_VALUE)\n                .build();\n    }\n\n    private List\u003cString\u003e getSentences(int n) {\n        List\u003cString\u003e listOfSentences = new ArrayList\u003c\u003e();\n        for (int i = 0; i \u003c n; i++) {\n            listOfSentences.add(sentences[random.nextInt(sentences.length)]);\n        }\n        return listOfSentences;\n    }\n}\n```\n\n#### SplitterProcessor.java\n```\n@Processor(\n        namespace = \"global\",\n        name = \"splitter-processor\",\n        version = \"0.1\",\n        cpu = 0.1,\n        memory = 32,\n        description = \"A processor that splits sentences by a given delimiter\",\n        processorType = ProcessorType.EVENT_DRIVEN,\n        requiredProperties = {},\n        optionalProperties = {\"delimiter\"}\n)\npublic class SplitterProcessor extends StreamingProcessor {\n    String delimiter;\n\n    @Override\n    protected EventSet consume(final ProcessingContext processingContext, final EventSet eventSet) throws\n            ProcessingException {\n        List\u003cEvent\u003e events = new ArrayList\u003c\u003e();\n        eventSet.getEvents().stream()\n                .forEach(event -\u003e {\n                    String sentence = (String) event.getData();\n                    String[] words = sentence.split(delimiter);\n                    events.add(Event.builder()\n                            .data(words)\n                            .id(Integer.MAX_VALUE)\n                            .properties(Collections.emptyMap())\n                            .build());\n                });\n        return EventSet.eventFromEventBuilder()\n                .isAggregate(false)\n                .partitionId(eventSet.getPartitionId())\n                .events(events)\n                .build();\n    }\n\n    @Override\n    public void initialize(final String instanceName,\n            final Properties global,\n            final Properties local,\n            final ComponentMetadata componentMetadata) throws InitializationException {\n        delimiter = ComponentPropertyReader.readString(local, global, \"delimiter\", instanceName, componentMetadata, \",\");\n\n    }\n\n    @Override\n    public void destroy() {\n        // do some cleanup if necessary\n    }\n}\n```\n\n#### WordCountProcessor.java\n```\n@Processor(\n        namespace = \"global\",\n        name = \"word-count-processor\",\n        version = \"0.2\",\n        description = \"A processor that prints word frequency counts within a tumbling window\",\n        cpu = 0.1,\n        memory = 128,\n        processorType = ProcessorType.TIMER_DRIVEN,\n        requiredProperties = {\"triggering_frequency\"},\n        optionalProperties = {}\n)\npublic class WordCountProcessor extends ScheduledProcessor {\n    Map\u003cString, Integer\u003e wordCounts = new HashMap\u003c\u003e();\n\n    @Override\n    protected void consume(final ProcessingContext processingContext, final EventSet eventSet) throws\n            ProcessingException {\n        eventSet.getEvents().stream()\n                .forEach(event -\u003e {\n                    String[] words = (String[]) event.getData();\n                    for (String word: words) {\n                        if (wordCounts.containsKey(word)) {\n                            wordCounts.put(word, wordCounts.get(word) + 1);\n                        } else {\n                            wordCounts.put(word, 1);\n                        }\n                    }\n                });\n    }\n\n    @Override\n    public void initialize(final String instanceName,\n            final Properties global,\n            final Properties local,\n            final ComponentMetadata componentMetadata) throws InitializationException {\n        // nothing to initialize here\n    }\n\n    @Override\n    public List\u003cEvent\u003e timeTriggerHandler(ProcessingContext processingContext) throws ProcessingException {\n        // this method will be called after a fixed interval of time, say 5 seconds\n        System.out.println(Joiner.on(\",\").withKeyValueSeparator(\"=\").join(wordCounts));\n        wordCounts.clear();\n        // nothing to send to downstream processors\n        return Collections.emptyList();\n\n    }\n\n    @Override\n    public void destroy() {\n        wordCounts.clear();\n    }\n}\n```\n\n`RandomSentenceSource -\u003e SplitterProcessor -\u003e WordCountProcessor`\n\nLet’s suppose all of these classes reside in a single maven project sample-topology\u003cbr\u003e\nSuppose the jar sample-topology.jar is hosted at http://localhost:8080\nTo register these sources and processors, build and release this project into artifactory and then use the following fabric-server API\u003cbr\u003e\n\nThe json specification for this computation will look like this\u003cbr\u003e\n\n```\n{\n    \"name\": \"word-count-topology\",\n    \"sources\": [{\n        \"id\": \"random-sentence-source\",\n        \"meta\": {\n            \"id\": \"1cfda5cb-99d4-34e3-83f0-5b1364a92cce\",\n            \"type\": \"SOURCE\",\n            \"namespace\": \"global\",\n            \"name\": \"random-sentence-source\",\n            \"version\": \"0.1\",\n            \"description\": \"A source that generates random sentences from a pool of sentences\",\n            \"processorType\": null,\n            \"requiredProperties\": [],\n            \"optionalProperties\": [\n                \"randomGeneratorSeed\"\n            ],\n            \"cpu\": 0.1,\n            \"memory\": 64,\n            \"source\": {\n                \"type\": \"jar\",\n                \"url\": \"file:///fabric-sample-processors/target/fabric-sample-processors-1.0-SNAPSHOT.jar\"\n            }\n        },\n        \"properties\": {}\n    }],\n    \"processors\": [{\n        \"id\": \"splitter-processor\",\n        \"meta\": {\n            \"id\": \"0b749006-d2dd-3684-a521-f76d6ab0dec8\",\n            \"type\": \"PROCESSOR\",\n            \"namespace\": \"global\",\n            \"name\": \"splitter-processor\",\n            \"version\": \"0.1\",\n            \"description\": \"A processor that splits sentences by a given delimiter\",\n            \"processorType\": \"EVENT_DRIVEN\",\n            \"requiredProperties\": [],\n            \"optionalProperties\": [\n                \"delimiter\"\n            ],\n            \"cpu\": 0.1,\n            \"memory\": 32,\n            \"source\": {\n                \"type\": \"jar\",\n                \"url\": \"file:///fabric-sample-processors/target/fabric-sample-processors-1.0-SNAPSHOT.jar\"\n            }\n        },\n        \"properties\": {\n            \"processor.splitter-processor.delimiter\": \" \"\n        }\n    }, {\n        \"id\": \"word-count-processor\",\n        \"meta\": {\n            \"id\": \"59f4fe28-b09b-3447-8bb2-26d3c23dd885\",\n            \"type\": \"PROCESSOR\",\n            \"namespace\": \"global\",\n            \"name\": \"word-count-processor\",\n            \"version\": \"0.2\",\n            \"description\": \"A processor that prints word frequency counts within a tumbling window\",\n            \"processorType\": \"TIMER_DRIVEN\",\n            \"requiredProperties\": [\n                \"triggering_frequency\"\n            ],\n            \"optionalProperties\": [],\n            \"cpu\": 0.1,\n            \"memory\": 128,\n            \"source\": {\n                \"type\": \"jar\",\n                \"url\": \"file:///fabric-sample-processors/target/fabric-sample-processors-1.0-SNAPSHOT.jar\"\n            }\n        },\n        \"properties\": {\n            \"processor.word-count-processor.triggering_frequency\": \"5000\"\n        }\n    }],\n    \"connections\": [{\n        \"fromType\": \"SOURCE\",\n        \"from\": \"random-sentence-source\",\n        \"to\": \"splitter-processor\"\n    }, {\n        \"fromType\": \"PROCESSOR\",\n        \"from\": \"splitter-processor\",\n        \"to\": \"word-count-processor\"\n    }],\n    \"properties\": {\n        \"computation.name\": \"word-count-topology\",\n        \"computation.eventset.is_serialized\": \"false\"\n    }\n}\n```\n\n### Benchmarks\n#### Performance Test Configuration\nNo of messages: 1 million\u003cbr\u003e\nPayload size: 258 bytes\u003cbr\u003e\n#### Topology\nKafka Source -\u003e Event Counter (Prints number of total events consumed every one second to the console)\u003cbr\u003e\nNo of partitions: 1\u003cbr\u003e\nTopic Name: end-to-end-latency-perf\u003cbr\u003e\nNo of instances of topology: 1\u003cbr\u003e\nKafka source buffer size: 3 MB\u003cbr\u003e\nDocker CPU (number of cpu shares): 1.0\u003cbr\u003e\nDocker Memory: 2 GB\u003cbr\u003e\nJVM Heap Size: 2 GB\u003cbr\u003e\n#### Kafka Broker configuration\n2 cores Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz 8 GB RAM\u003cbr\u003e\n#### Mesos Host configuration\n8 cores Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz 32 GB RAM\u003cbr\u003e\n\nEnd to end latency to process all messages in seconds (ceiling) averaged over multiple runs is presented below\u003cbr\u003e\u003cbr\u003e\n\n\n| Configuration           | Without gzip | With gzip |\n| ----------------------- | ------------ | --------- |\n| Local Zk, Local Kafka   |     8        |    5      |\n| Remote Zk, Remote Kafka |     36       |    6      |\n\n\nThroughput with best configuration T ~ 166666 events / second\u003cbr\u003e\n\nNOTE: Using Disruptor with YieldWaitingStrategy instead of LBQ for channel communication actually reduced the throughput\u003cbr\u003e\n\n### Using Processor Maven Archetype\n\nTo create a processor project, please run following command - \n\n```\nmvn archetype:generate -DarchetypeGroupId=com.olacabs.fabric -DarchetypeArtifactId=fabric-processor-archetype -DarchetypeVersion=0.0.1-SNAPSHOT -DartifactId=\u003cartifact_id_of_your_project\u003e -DgroupId=\u003cgroup_id_of_your_project\u003e -DinteractiveMode=ture\n```\nExample -\n\n```\nmvn archetype:generate -DarchetypeGroupId=com.olacabs.fabric -DarchetypeArtifactId=fabric-processor-archetype -DarchetypeVersion=0.0.1-SNAPSHOT -DartifactId=fabric-my-processor -DgroupId=com.olacabs.fabric -DinteractiveMode=ture\n```\nWhat it does -\n  - Creates the pom project for the processor with all the updated version of compute and other related jars.\n  - Creates boiler plate code, with example, for scheduled and stream processor. You can modify the example java file as per your need.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folacabs%2Ffabric","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Folacabs%2Ffabric","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folacabs%2Ffabric/lists"}