{"id":26267979,"url":"https://github.com/oslabs-beta/evastudio","last_synced_at":"2025-10-09T01:13:52.603Z","repository":{"id":40750039,"uuid":"507156842","full_name":"oslabs-beta/evaStudio","owner":"oslabs-beta","description":"An Apache Kafka monitoring tool to prototype and scale real-time streaming data pipelines, and test parallelization of multi-stage ML models before production, with metrics for potential automation, in preconfigured Docker containers.","archived":false,"fork":false,"pushed_at":"2022-11-07T21:23:46.000Z","size":256374,"stargazers_count":59,"open_issues_count":0,"forks_count":4,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-08T10:53:15.851Z","etag":null,"topics":["apache-kafka","java","react-redux","typescript"],"latest_commit_sha":null,"homepage":"https://www.evastudio.dev/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oslabs-beta.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-24T22:15:33.000Z","updated_at":"2024-10-25T13:02:21.000Z","dependencies_parsed_at":"2022-07-11T14:18:41.822Z","dependency_job_id":null,"html_url":"https://github.com/oslabs-beta/evaStudio","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oslabs-beta%2FevaStudio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oslabs-beta%2FevaStudio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oslabs-beta%2FevaStudio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oslabs-beta%2FevaStudio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oslabs-beta","download_url":"https://codeload.github.com/oslabs-beta/evaStudio/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251767513,"owners_count":21640543,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","java","react-redux","typescript"],"created_at":"2025-03-14T04:18:59.324Z","updated_at":"2025-10-09T01:13:47.558Z","avatar_url":"https://github.com/oslabs-beta.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n   \u003cimg src=\"https://github.com/oslabs-beta/evaStudio/blob/readme/src/assets/evaStudio_small.png\" height=300/\u003e\n\u003cdiv\u003e\n\n\u003cbr/\u003e\n\u003cbr/\u003e\n\n[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/oslabs-beta/evastudio)\n![Release: 0.0.1](https://img.shields.io/badge/Release-0.0.1-red)\n![License: MIT](https://img.shields.io/badge/License-MIT-orange.svg)\n![Contributions Welcome](https://img.shields.io/badge/Contributions-welcome-blue.svg)\n[![Github stars](https://img.shields.io/github/stars/oslabs-beta/evastudio?style=social)](https://github.com/oslabs-beta/evaStudio)\n\n\u003ch2\u003eDesign and scale a real-time streaming data pipeline at the click of a button.\u003c/h2\u003e\n\n\u003cbr/\u003e\n\n\u003cp align=\"left\"\u003e \u003cb\u003e evaStudio (beta) \u003c/b\u003e is a web GUI and testing playground for quickly designing and scaling your event-driven data architecture with Kafka brokers and Zookeeper. Designed with data scientists in mind, EvaStudio makes it easier to develop, orchestrate, experiment, and monitor machine learning workflows at scale, so you can focus on analytical challenges instead of infrastructure.\u003c/p\u003e\n\n\u003cbr/\u003e\n\n\u003cdiv align=\"left\"\u003e\n\n\n\n\n# Getting Started\nTo install our web application locally, you will need Docker, Node Package Manager, Java JDK 11 and Maven. Clone this repository, and run `npm install` at the root of the directory. Then run `mvn spring-boot:run` to start the \"java_backend\" microservice. \n\n## Features\n* Design your pipeline using a drag-and-drop tool for extracting, transforming, and loading data in and out of Kafka messaging brokers.\n* Source streaming data from a CSV, SQL database, or HTTP/API, to Apache Spark or Jupyter Notebook for analysis.\n* Manage topics and messages directly inside our web GUI. \n* \u003cb\u003e Monitor cluster health, and check performance metrics for load, latency, throughput, disk usage, and messages and bytes in and out per second, in order to test and scale your data architecture before loading into production. \u003c/b\u003e\n\nYou can also connect your existing Kafka clusters on AWS, provided that you follow our setup instructions in our documentation for Prometheus, JMX, and Grafana installation.\n\n# Interface\n### 1. Drag-and-drop data pipeline designer\n\n   \u003cimg src=\"https://github.com/oslabs-beta/evaStudio/blob/demo/src/assets/CreateCluster_demo.gif\" style=\"height: 35rem;\" /\u003e\n\n### 2. Stream data via HTTP / API or by uploading a CSV to run through Kafka clusters\n\n   \u003cimg src=\"https://github.com/oslabs-beta/evaStudio/blob/demo/src/assets/StreamingCSV_demo.gif\" style=\"height: 35rem;\" /\u003e\n\n### 3. Monitor cluster health. Manage brokers, topics, partitions, and messages.\n      \n   \u003cimg src=\"https://github.com/oslabs-beta/evaStudio/blob/demo/src/assets/ClusterHealth.png\" style=\"width: 45rem;\" /\u003e\n\n### 4. Monitor system metrics: fetch response latency, throughput (messages \u0026 bytes per second), CPU and JVM usage, Zookeeper failures.\n\n  \u003cimg src=\"https://github.com/oslabs-beta/evaStudio/blob/demo/src/assets/SystemLoad.png\" style=\"height: 35rem;\" /\u003e\n\n\n# Application Architecture and Logic\n\nevaStudio is a browser-based application built using ReactJs and Node/Express, with a Java Spring Boot microservice to interface with Kafka servers and Zookeeper. In our beta version, we are exposing data endpoints on Kafka clusters using Prometheus and JMX, and displaying analytics with Grafana. In iteration, we plan to make loading and real-time updating of these panels faster using D3. We use React Flow for a drag-and-drop GUI for sketching out a data pipeline. We connect data sources and sinks to a user-selected number of Kafka servers locally, to be spun up with Docker.\n\nCurrently, we allow for data to be imported as a CSV and \"mock-streamed\" for real-time analytics. We also connect to the PostgreSQL database, and allow for data to be streamed from an HTTP or API endpoint. We provide the option of Jupyter Hub or Spark for analytics. Future iterations will allow for more data transformations, mapping/reducing/filtering, and connections to more microservices through the GUI tool that enable easier deployment of machine learning models and applications. We will also be making our application available on DockerHub.\n\n\n## Built With\n* Apache Kafka v3.2.0\n* Spring for Apache Kafka v2.8.9\n* Java Spring Boot v2.6.9 with Spring Web Services v3.1.1\n* Apache Maven v3.8.6 requiring Java SE Development Kit v11.0.16 \n* ReactJS v17\n* Redux 4.1\n* React Router 6.3\n* React Flow 10.3.8\n* NodeJS with Express v4.16 \n* Typescript 4.7\n* Tailwind.css v3.1\n* PostgreSQL v14\n* Docker 20.10\n* Prometheus \u0026 JMX Exporter v0.17.0\n* Grafana v9.0\n* Material UI v5.9 \u0026 v5.8\n* JUnit, Embedded Kafka \u0026 Mockito for unit \u0026 integration testing\n* Amazon Web Services \u0026 Github Actions for CI/CD\n   \n### DOCUMENTATION:\n* Next.js with Vercel\n   \n   \n# Core Team\n\n\u003ctable\u003e\u003ctbody\u003e\u003ctr\u003e\n  \u003ctd align=\"center\" width=\"180\"\u003e\n    \u003cimg src=\"https://github.com/oslabs-beta/evaStudio/blob/readme/src/assets/andres.jpg\" style=\"height: 8rem; width: 8rem;\" /\u003e\n    \u003cbr/\u003e\n    \u003cstrong\u003eAndres Chaparro\u003c/strong\u003e\n    \u003cbr/\u003e\n    \u003ca href=\"https://github.com/anch09\"\u003eGitHub\u003c/a\u003e\n    \u003cbr/\u003e\n    \u003ca href=\"https://www.linkedin.com/in/andreschc/\"\u003eLinkedIn\u003c/a\u003e\n  \u003c/td\u003e\n   \n  \u003ctd align=\"center\" width=\"180\"\u003e\n    \u003cimg src=\"https://github.com/oslabs-beta/evaStudio/blob/readme/src/assets/kaitlin.png\" style=\"height: 8rem; width: 8rem;\" /\u003e\n    \u003cbr/\u003e\n    \u003cstrong\u003eKaitlin Zhang\u003c/strong\u003e\n    \u003cbr/\u003e\n    \u003ca href=\"https://github.com/kaizengrowth\"\u003eGitHub\u003c/a\u003e\n    \u003cbr/\u003e\n    \u003ca href=\"https://www.linkedin.com/in/kaizengrowth/\"\u003eLinkedIn\u003c/a\u003e\n  \u003c/td\u003e\n   \n   \n  \u003ctd align=\"center\" width=\"180\"\u003e \n    \u003cimg src=\"https://github.com/oslabs-beta/evaStudio/blob/readme/src/assets/max.jpeg\" style=\"height: 8rem; width: 8rem;\" /\u003e\n    \u003cbr/\u003e\n    \u003cstrong\u003eMaxwell Shick\u003c/strong\u003e\n    \u003cbr/\u003e\n    \u003ca href=\"https://github.com/mshick97/\"\u003eGitHub\u003c/a\u003e\n    \u003cbr/\u003e\n    \u003ca href=\"https://www.linkedin.com/in/maxwell-shick/\"\u003eLinkedIn\u003c/a\u003e\n   \u003c/td\u003e\n   \n  \u003ctd align=\"center\" width=\"180\"\u003e\n    \u003cimg src=\"https://github.com/oslabs-beta/evaStudio/blob/readme/src/assets/perla.jpeg\" style=\"height: 8rem; width: 8rem;\" /\u003e\n    \u003cbr/\u003e\n    \u003cstrong\u003ePerla Royer\u003c/strong\u003e\n    \u003cbr/\u003e\n    \u003ca href=\"https://github.com/perlaroyerc\"\u003eGitHub\u003c/a\u003e\n    \u003cbr/\u003e\n    \u003ca href=\"https://www.linkedin.com/in/perlaroyerc/\"\u003eLinkedIn\u003c/a\u003e\n  \u003c/td\u003e\n   \n  \u003ctd align=\"center\" width=\"180\"\u003e\n    \u003cimg src=\"https://github.com/oslabs-beta/evaStudio/blob/readme/src/assets/sattwik.jpeg\" style=\"height: 8rem; width: 8rem;\" /\u003e\n    \u003cbr/\u003e\n    \u003cstrong\u003eSattwik Biswal \u003c/strong\u003e\n    \u003cbr/\u003e\n    \u003ca href=\"https://github.com/sattwyk\"\u003eGitHub\u003c/a\u003e\n    \u003cbr/\u003e\n    \u003ca href=\"https://www.linkedin.com/in/sattwyk/\"\u003eLinkedIn\u003c/a\u003e\n  \u003c/td\u003e\n\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\n\n## With Special Thanks To:\nOur advisors [Mia Zeng](https://www.linkedin.com/in/miaowen-zeng/) and [Brandon McKallagat](https://www.linkedin.com/in/braden-mckallagat-1a8b77b3/)\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foslabs-beta%2Fevastudio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foslabs-beta%2Fevastudio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foslabs-beta%2Fevastudio/lists"}