{"id":14965902,"url":"https://github.com/agile-lab-dev/wasp","last_synced_at":"2025-04-09T12:06:10.187Z","repository":{"id":50173013,"uuid":"253750082","full_name":"agile-lab-dev/wasp","owner":"agile-lab-dev","description":"WASP is a framework to build complex real time big data applications. It relies on a kind of Kappa/Lambda architecture mainly leveraging Kafka and Spark.  If you need to ingest huge amount of heterogeneous data and analyze them through complex pipelines, this is the framework for you.","archived":false,"fork":false,"pushed_at":"2025-03-22T13:44:52.000Z","size":7946,"stargazers_count":30,"open_issues_count":5,"forks_count":11,"subscribers_count":8,"default_branch":"develop","last_synced_at":"2025-04-02T10:12:33.060Z","etag":null,"topics":["akka","elasticsearch","hadoop","hbase","hdfs","jdbc","kafka","parquet","scala","solr","spark","spark-streaming","yarn"],"latest_commit_sha":null,"homepage":"https://www.agilelab.it/wasp-platform/","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/agile-lab-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-07T09:40:38.000Z","updated_at":"2025-03-22T13:44:59.000Z","dependencies_parsed_at":"2023-12-15T06:27:30.267Z","dependency_job_id":"6eed0469-8fd3-4838-a06f-c281b7c8925a","html_url":"https://github.com/agile-lab-dev/wasp","commit_stats":{"total_commits":537,"total_committers":39,"mean_commits":13.76923076923077,"dds":0.770949720670391,"last_synced_commit":"d2b2aaf1b377d1dd2d80945f78eb70f411d8dae2"},"previous_names":[],"tags_count":51,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agile-lab-dev%2Fwasp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agile-lab-dev%2Fwasp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agile-lab-dev%2Fwasp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agile-lab-dev%2Fwasp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/agile-lab-dev","download_url":"https://codeload.github.com/agile-lab-dev/wasp/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248036063,"owners_count":21037092,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["akka","elasticsearch","hadoop","hbase","hdfs","jdbc","kafka","parquet","scala","solr","spark","spark-streaming","yarn"],"created_at":"2024-09-24T13:35:32.657Z","updated_at":"2025-04-09T12:06:10.166Z","avatar_url":"https://github.com/agile-lab-dev.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"![WASP_logo](documentation/icons/WASP_logo.jpg)\n\n# \u003ca href=\"https://agilefactory.gitlab.io/Agile.Wasp2/\"\u003eWASP\u003c/a\u003e - Wide Analytics Streaming Platform\n\n**Official documentation website** : [Wasp documentation](https://agilefactory.gitlab.io/Agile.Wasp2/)\n\n\n[![Join the chat at https://gitter.im/agile-lab-dev/wasp](https://badges.gitter.im/agile-lab-dev/wasp.svg)](https://gitter.im/agile-lab-dev/wasp?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\n## Table of contents\n\n- [\u003ca href=\"https://agilefactory.gitlab.io/Agile.Wasp2/\"\u003eWASP\u003c/a\u003e - Wide Analytics Streaming Platform](#wasp---wide-analytics-streaming-platform)\n  - [Table of contents](#table-of-contents)\n    - [General](#general)\n      - [Overview](#overview)\n      - [WASP in the wild](#wasp-in-the-wild)\n      - [Background](#background)\n      - [Architecture](#architecture)\n      - [Glossary](#glossary)\n      - [Services](#services)\n        - [Kafka](#kafka)\n        - [Spark](#spark)\n        - [Akka](#akka)\n        - [MongoDB](#mongodb)\n        - [Pluggable Datastore](#pluggable-datastore)\n      - [Using WASP](#using-wasp)\n        - [Setting up the development environment](#setting-up-the-development-environment)\n        - [Just try wasp without dev dependencies](#just-try-wasp-without-dev-dependencies)\n        - [Extension points](#extension-points)\n      - [Have a look at what's going on](#have-a-look-at-whats-going-on)\n\n\n### General\n\n#### Overview\nWASP is a framework to build complex real time big data applications.\nIt relies on a kind of Kappa/Lambda architecture mainly leveraging Kafka and Spark.\n\nIf you need to ingest huge amount of heterogeneous data and analyze them through complex pipelines, this is the framework for you.\nIf you need a point and click product, this is not the tool for you.\n\nWASP is a big data framework that allows you to not waste time with devops architectures and integrating different components. WASP lets you focus on your data, business logic and algorithms, without worrying about tipical big data problems like:\n\n- at least once or exactly once delivery\n- periodically training a machine learning model\n- publishing your results in real time, to be reactive\n- applying schemas to unstructured data\n- feeding different datastores from the same data flow in a safe way\n- etc.\n\nFor more technical documentation, head to the [documentation](documentation/) folder.\n\n#### WASP in the wild\nWASP has been added to Cloudera Solution Gallery as an Open Source tool to simplify streaming workflows.\n\n**You can see it [here](https://www.cloudera.com/solutions/gallery/agilelab-wide-streaming-analytics-platform.html)!**\n\n\n#### Background\nHandling huge streams of data in near real time is a hard task. So we want to build a reference architecture to speed up fast data application development and to avoid common mistakes about fault tolerance and reliability.\nKafka is the central pillar of the architecture and helps to handle streams in the correct way. We have been inspired by the Kappa architecture definition.\n\n#### Architecture\nYou can refer to the diagrams ([Wasp1](documentation/diagrams/Wasp1.png) and [Wasp2](documentation/diagrams/Wasp2.png)) to gain a general overview of the architecture.\nThe project is divided into sub modules:\n\n- **wasp-core**: provides all basic functionalities, pojo and utilities\n- **wasp-master**: it provides the main entry point to control your application, exposing the WASP REST API. In the future, this will also provide a complete web application for monitoring and configuration.\n- **wasp-producers**: a thin layer to easily expose endpoints for ingestion purposes. Leveraging Akka-Camel we can provide Http, Tcp, ActiveMQ, JMS, File and many other connectors. This ingestion layer pushes data into Kafka.\n- **wasp-consumers-rt**: ...\n- **wasp-consumers-spark**: the consumer layer incapsulates Spark Streaming to dequeue data from Kafka, apply business logic to it and then push the output on a target system.\n\nAll the components are coordinated, monitored and owned by an Akka Cluster layer, that provides scalability and fault tolerance for each component. For example you can spawn multiple identical producers to balance the load on your http endpoint, and then fairly distribute the data on Kafka.\n\n#### Glossary\n- **Pipegraph**: a directed acyclic graph of data transformations. Each step is lazy and loosely coupled from previous and the next one. It is basically an ordered list of ETL blocks, with Inputs and Outputs.\n- **ETL**: represents a Spark Streaming job. It can consume data from one or more Inputs, elaborate the incoming data and push it to an Output. You can't have more than an Output for an ETL block, in order to avoid misalignment between outputs. If you want to write the same data on different datastores, you must consume the topic data with two different ETL blocks. Both Streaming and Batch ETLs are supported.\n- **Input**: a source of data for an ETL block.\n- **Output**: a destination for data produced by an ETL block. Can be any of various datastores or messaging systems.\n- **Topic**: the representation of a Kafka topic with an associated Avro schema. Can be either an Input or an Output.\n- **Index**: the representation of an index in an indexed datastore (either ElasticSearch or Solr) adn its associated schema. Can be either an Input or an Output.\n- **KVStore**: an abstraction for a Key-Value store, like Cassandra and HBase, for when you need high performance access by key. Can only be used as an Output. This is not implemented yet.\n- **OLAP**: an abstraction for an Online Analytical Processing system. It will help to provide OLAP capabilities to the application. Druid and Kylin will be the available options. This is not implemented yet.\n- **Raw**: any of a number of datastores based on files; for example, HDFS or S3. Can be either an Input or an Output.\n- **Producer**: Producers are independent from pipegraphs. They ingest data from different sources and write data to a Kafka topic, after formatting it according to a the schema.\n\n#### Services\n![components](documentation/diagrams/components.png)\n\n##### Kafka\nKafka is the central element of this architecture blue print.\nEach topic must have an associated Avro schema. This enforces type consistency and is the first step towards reliable real time data quality, something we will work on in the next future.\nAvro has been chosen because more typed and descriptive than JSON and because of its compatibility with Spark and the Hadoop world in general.\nKafka decouples the ingestion layer from the analysis one. This allows updating algorithms and models without impacting the ingestion layer, and vice versa.\n\n##### Spark\nSpark is the data engine powering WASP, and is used in two components: Streaming ETL and Batch ETL. It can also be used to provide a JDBC interface using Thrift server.\nWASP supports running Spark in three different ways:\n- embedded, using Spark's local mode, which is recommended for development only\n- on YARN, used when running with an existing Hadoop cluster\n- with Spark's standalone clustering (master + workers)\n\n##### Akka\nAkka is our middleware: each component of WASP is an actor and relies on a clustered Actor System. In this way each component can be a separate process, and even run on different machines, and we can handle fault tolerance in a trasparent way to the whole application.\nThis is a general overview of the [ActorSystem](documentation/diagrams/actor_system.png)\n\n##### MongoDB\nMongoDB is the central repository for all configurations, ML models, and entities. It is fault tolerant and it simplifies the deployment in a distributed environment because each node just needs the MongoDB address to be ready to go.\n\n##### Pluggable Datastore\n\nWASP system is integrated with Elasticsearch, Solr, Kafka, HBase, Mongo, Jdbc Datasources and HDFS. All data stored inside the datastore is indexed and searchable via the specific query language of the datastore.\n\n#### Using WASP\n\n##### Setting up the development environment\nWASP is written in Scala, and the build is managed with SBT.\n\nThe recommended development environment is Linux; developing on Windows or MacOS is certainly possible, but is not supported, sorry about that!\n\nBefore starting:\n- Install JDK 8\n- Install SBT\n- Install Git\n\n\nThe steps to getting WASP up and running for development are pretty simple:\n\n*Github mirror*\n\n- Clone this repository:\n\n    `git clone https://github.com/agile-lab-dev/wasp.git`\n\n- Build the image locally (it is a cloudera based deployment so we cannot distribute it)\n\n```sh\ncd whitelabel/docker/cdh6/cdh-docker\ndocker build . -t  registry.gitlab.com/agilefactory/agile.wasp2/cdh-docker:6.3.2\n```\n\n - Use the scripts to start WASP:\n\n    `whitelabel/docker/cdh6/start-wasp.sh`\n\n*Gitlab* \n\n- Clone this repository:\n\n    `git clone https://gitlab.com/AgileFactory/Agile.Wasp2/` \n\n- Perform a docker login on gitlab registry:\n  \n    `docker login registry.gitlab.com`\n\n- Use the scripts to start WASP:\n    \n    `whitelabel/docker/cdh6/start-wasp.sh`\n    \n##### Just try wasp without dev dependencies\n\n*Github mirror*\n\nSorry at the moment no public accessible image is available due to distribution concerns, you can build it yourself\nbut you still need a development environment set up\n\n*Gitlab mirror*\n\n- Login to the registry:\n\n    `docker login registry.gitlab.com`\n\n- Run wasp chosing a version (replace `$TAG`):\n\n    `docker run -it --rm -p 8088:8088 -p 2891:2891 -p 8042:8042 -p 5005:5005 -p 5006:5006 -p 5007:5007 -p 5008:5008 -p8983:8983 -p4040:4040 -p4041:4041 registry.gitlab.com/agilefactory/agile.wasp2/try-wasp:$TAG`\n\n##### Extension points\nIn order to build you application there are several extension points where to place your algorithms.\n\n- **Producer**: If your data can flow directly into Kafka, for example if you already have a Flume ingestion layer, you don't need to have producers. Otherwise you should extend WaspProducerActor, maybe leveraging Camel, to expose some kind of endpoint (tcp, http, jms, etc).  \nAt this point you only need to worry about formatting your incoming data according to an Avro schema, as the base class will handle the Kafka connectivity for you.\nIn order to have each producer acting independently, you should also overwrite WaspProducerNodeGuardian.\n\nAn example of an extended WaspProducerNodeGuardian:\n\n```scala\nfinal class YahooFinanceProducer(env: { val producerBL: ProducerBL; val topicBL: TopicBL })\n    extends WaspProducerNodeGuardian(env) {\n\n  val name = YahooFinanceProducer.name\n\n  def startChildActors() = {\n    logger.info(s\"Starting child actors on ${cluster.selfAddress}\")\n\n    val stocks = producer.configuration\n      .flatMap(bson =\u003e bson.getAs[BSONArray](\"stocks\").map(array =\u003e array.values.map(s =\u003e s.seeAsOpt[String].get)))\n      .getOrElse(YahooFinanceProducer.stocks)\n\n    stocks foreach { s =\u003e\n      println(\"stock: \" + s)\n      val aRef = context.actorOf(Props(new StockActor(s, kafka_router, associatedTopic)))\n      aRef ! StartMainTask\n    }\n  }\n}\n```\n\nAn example of an extended WaspProducerActor:\n\n```scala\ncase class Envelope(value: JsValue)\ncase object Tick\n\nprivate[wasp] class StockActor(stock: String, kafka_router: ActorRef, topic: Option[TopicModel])\n    extends ProducerActor[JsValue](kafka_router, topic) {\n\n  import system.dispatcher\n  val client = new HttpClient()\n  val url =\n    s\"\"\"https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.quotes%20where%20symbol%20in%20(%22$stock%22)\u0026format=json\u0026env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys\u0026callback=\"\"\"\n\n  override def preStart(): Unit = logger.info(s\"Starting $stock\")\n\n  def mainTask(): Unit = {\n    logger.info(s\"Starting main task for actor: ${this.getClass.getName}\")\n    self ! Tick\n  }\n\n  override def receive: Receive = super.receive.orElse {\n    case toSend: Envelope =\u003e\n      // actually send the message to Kafka\n      sendMessage(toSend.value)\n      // schedule the stock fetch in 10 seconds\n      task = Some(context.system.scheduler.scheduleOnce(10.seconds, self, Tick))\n    case Tick =\u003e\n      val jsonToSend = for {\n        res        \u003c- client.url(url).getStream()\n        body       = res._2\n        jsonString \u003c- body |\u003e\u003e\u003e Iteratee.fold(\"\")((json, bytes) =\u003e json + new String(bytes, \"UTF-8\"))\n        jsonValue  = JsonParser(jsonString)\n        _          = logger.debug(\"Forwarding producer message to Kafka: \" + jsonString)\n      } yield Envelope(jsonValue)\n      akka.pattern.pipe(jsonToSend).pipeTo(self)\n    case Status.Failure(f) =\u003e\n      // if the http request failed, retry immediately\n      logger.error(f.getMessage, f)\n      self ! Tick\n  }\n\n  def generateOutputJsonMessage(inputJson: JsValue): String = {\n\n    /* The following mappings are just an example made to show the producer in action */\n    val id_event    = (inputJson \\\\ \"count\").map(_.as[Double]).headOption\n    val source_name = (inputJson \\\\ \"StockExchange\").map(t =\u003e t.asOpt[String].getOrElse(\"\")).headOption.getOrElse(\"\")\n    val topic_name  = topic.map(_.name).getOrElse(YahooTopicModel.yahooTopic.name)\n    val metric_name = stock\n\n    val ts = (inputJson \\\\ \"created\")\n      .map(time =\u003e\n        time.asOpt[String] match {\n          case None    =\u003e TimeFormatter.format(new Date())\n          case Some(t) =\u003e TimeFormatter.format(t, YahooFinanceProducer.timeFormat)\n        }\n      )\n      .head\n    val latitude   = 0\n    val longitude  = 0\n    val value      = (inputJson \\\\ \"Bid\").map(t =\u003e t.asOpt[String].map(_.toDouble).getOrElse(0.0)).headOption.getOrElse(0.0)\n    val bid        = value\n    val ask        = (inputJson \\\\ \"Ask\").map(t =\u003e t.asOpt[String].map(_.toDouble).getOrElse(0.0)).headOption.getOrElse(0.0)\n    val stock_name = (inputJson \\\\ \"Name\").map(t =\u003e t.asOpt[String].getOrElse(stock)).headOption.getOrElse(stock)\n    val percent_change = (inputJson \\\\ \"PercentChange\")\n      .map(t =\u003e t.asOpt[String].map(s =\u003e s.replace(\"%\", \"\").toDouble).getOrElse(0.0))\n      .headOption\n      .getOrElse(0.0)\n    val volume =\n      (inputJson \\\\ \"Volume\").map(t =\u003e t.asOpt[String].map(_.toDouble).getOrElse(0.0)).headOption.getOrElse(0.0)\n    val currency = (inputJson \\\\ \"Currency\").map(t =\u003e t.asOpt[String].getOrElse(\"\")).headOption.getOrElse(\"\")\n    val payload  = inputJson.toString().replaceAll(\"\\\"\", \"\") // String bonification\n\n    val myJson = s\"\"\"{\n\t\t     \"id_event\":${id_event.getOrElse(\"0\")},\n\t\t     \"source_name\":\"$source_name\",\n\t\t     \"topic_name\":\"$topic_name\",\n\t\t     \"metric_name\":\"$metric_name\",\n\t\t     \"timestamp\":\"$ts\",\n\t\t     \"latitude\":$latitude,\n\t\t     \"longitude\":$longitude,\n\t\t     \"value\":$value,\n\t\t     \"payload\":\"$payload\",\n\t\t     \"stock_name\":\"$stock_name\",\n\t\t     \"bid\":$bid,\n\t\t     \"ask\":$ask,\n\t\t     \"percent_change\":$percent_change,\n\t\t     \"volume\":$volume,\n\t\t     \"currency\":\"$currency\"\n\t\t     }\"\"\"\n\n    myJson\n  }\n}\n```\n\n- **Consumer**: to create a consumer you only need to implement the Strategy trait with a concrete class. The full qualifier of the class will then be used in the ETL block inside the Pipegraph definition or in the Batch definition.\n\n- **Models**: these define all the details of your Pipegraph, such as the Inputs, ETL blocks and Outputs, along with some metadata. Topics, Index, Raw etc. are first defined separately using the corresponding model classes, and then are used inside the Pipegraph definition. This has to be added to MongoDB; a convenient place to do this in is the prepareWorkloads() method in the FrameworkLauncher trait. Simply extend the trait in a class, implement the method, then use your Launcher as the main class for your application.\n\n- **Pipegraph**:\n\n    The following diagrams represent a pipegraph overview diagram:\n\n    ![pipegraph](documentation/diagrams/pipegraph.png)\n\n    while this is  a more specific model representation of it:\n\n    ![pipegraph_model](documentation/diagrams/pipegraph_model.png)\n\n    The pipegraph is the core of WASP, because it allows to abstract a pipeline with no coupling between components. It's really easy to change a pipegraph in order to add a datastore or more transformation steps.\n    The structure of a Pipegraph forces you to implement in the right direction to avoid architectural mistakes. It forces you to have just one single output for each stream, so if you need to write your data into two datastore you are obliged to redirect the stream to Kafka topic and to consume it with two indipendent consumers.\n\n    An example of a Pipegraph definition:\n\n```scala\nobject MetroPipegraphModel {\n\n  lazy val metroPipegraphName      = \"MetroPipegraph6\"\n  lazy val metroPipegraph          = MetroPipegraph()\n  lazy val conf: Config            = ConfigFactory.load\n  lazy val defaultDataStoreIndexed = conf.getString(\"default.datastore.indexed\")\n\n  private[wasp] object MetroPipegraph {\n\n    def apply() =\n      PipegraphModel(\n        name = MetroPipegraphModel.metroPipegraphName,\n        description = \"Los Angeles Metro Pipegraph\",\n        owner = \"user\",\n        system = false,\n        creationTime = WaspSystem.now,\n        etl = List(\n          ETLModel(\n            \"write on index\",\n            List(\n              ReaderModel(MetroTopicModel.metroTopic._id.get, MetroTopicModel.metroTopic.name, TopicModel.readerType)\n            ),\n            WriterModel.IndexWriter(\n              MetroIndexModel.metroIndex._id.get,\n              MetroIndexModel.metroIndex.name,\n              defaultDataStoreIndexed\n            ),\n            List(),\n            Some(StrategyModel(\"it.agilelab.bigdata.wasp.pipegraph.metro.strategies.MetroStrategy\", None))\n          )\n        ),\n        rt = List(),\n        dashboard = None,\n        isActive = false,\n        _id = Some(BSONObjectID.generate)\n      )\n  }\n}\n```\n\nAn other important part of the pipegraph is the strategy. Using strategy, you can apply custom transformation directly to the dataframe, when the DStream is processed with Spark.\n\nAn example of a Pipegraph strategy definition:\n\n```scala\ncase class MetroStrategy() extends Strategy {\n\n  def transform(dataFrames: Map[ReaderKey, DataFrame]) = {\n\n    val input = dataFrames.get(ReaderKey(TopicModel.readerType, \"metro.topic\")).get\n\n    /** Put your transformation here. */\n    input.filter(input(\"longitude\") \u003c -118.451683d)\n  }\n}\n```\n\nIn this example the DataFrame is filtered at runtime with a \"longitude\" condition (i.e. \u003c -118.451683D). Is possible apply more complicated trasformations using all the Spark DataFrame APIs like select, filter, groupBy and count [Spark DataFrame APIs](https://spark.apache.org/docs/1.6.2/api/scala/index.html#org.apache.spark.sql.DataFrame).\n\n#### Have a look at what's going on\n- \u003chttp://localhost:2891/pipegraphs\u003e, \u003chttp://localhost:2891/producers\u003e, \u003chttp://localhost:2891/batchjobs\u003e for the current state of your Pipegraphs / Producers / BatchJobs\n- \u003chttp://localhost:8088\u003e YARN Web UI\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagile-lab-dev%2Fwasp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fagile-lab-dev%2Fwasp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagile-lab-dev%2Fwasp/lists"}