{"id":19686111,"url":"https://github.com/vhive-serverless/vswarm","last_synced_at":"2025-12-14T20:35:47.698Z","repository":{"id":37797409,"uuid":"383709892","full_name":"vhive-serverless/vSwarm","owner":"vhive-serverless","description":"A suite of representative serverless cloud-agnostic (i.e., dockerized) benchmarks","archived":false,"fork":false,"pushed_at":"2025-03-13T12:42:15.000Z","size":473921,"stargazers_count":53,"open_issues_count":77,"forks_count":25,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-29T13:06:25.200Z","etag":null,"topics":["benchmarking","faas","function-as-a-service","knative","knative-eventing","knative-serving","serverless","vhive","vswarm"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vhive-serverless.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/contributing_to_vhive.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-07-07T07:21:24.000Z","updated_at":"2025-03-05T19:59:29.000Z","dependencies_parsed_at":"2023-10-03T15:11:08.629Z","dependency_job_id":"74593ec6-3ccb-4d36-9969-83b643176634","html_url":"https://github.com/vhive-serverless/vSwarm","commit_stats":{"total_commits":467,"total_committers":23,"mean_commits":"20.304347826086957","dds":0.6124197002141327,"last_synced_commit":"3c9c95a303380598cfe6909898c2edf968ad2093"},"previous_names":["ease-lab/vswarm","ease-lab/vhive-benchmarking"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vhive-serverless%2FvSwarm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vhive-serverless%2FvSwarm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vhive-serverless%2FvSwarm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vhive-serverless%2FvSwarm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vhive-serverless","download_url":"https://codeload.github.com/vhive-serverless/vSwarm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247345850,"owners_count":20924102,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking","faas","function-as-a-service","knative","knative-eventing","knative-serving","serverless","vhive","vswarm"],"created_at":"2024-11-11T18:26:08.902Z","updated_at":"2025-12-14T20:35:47.627Z","avatar_url":"https://github.com/vhive-serverless.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n\n# vSwarm - Serverless Benchmarking Suite\n\nWelcome! vSwarm is a collection of ready-to-run serverless benchmarks, each\ntypically consisting of a number of interconnected serverless functions, and with a general\nfocus on realistic data-intensive workloads.\n\nThis suite is part of the [vHive Ecosystem](https://vhive-serverless.github.io/). Its a turnkey and fully\ntested solution meant to used in conjunction with\n[vHive](https://github.com/ease-lab/vhive), and is compatible with all technologies that it supports,\nnamely, containers, Firecracker and gVisor microVMs. The majority of benchmarks support distributed\ntracing with [Zipkin](https://zipkin.io/) which traces both the infra components and the user\nfunctions.\n\nIn addition to the multi-function benchmarks, the vSwarm suite contains a set of [standalone functions](./benchmarks/README.md#standalone-functions-benchmark-summary), which support both x86 and arm64 architectures. Most of the standalone functions are compatible with [vSwarm-u](https://github.com/vhive-serverless/vSwarm-u), which allows to run them in the [gem5](https://www.gem5.org/) cycle-accurate full-system CPU simulator and study microarchitectural implications of serverless computing.\nthe state-of-the-art research platform for system-and microarchitecture.\nThe standalone functions can therefore be used as microbenchmarks to first pin-point microarchitectural bottlenecks in execution of serverless workloads using [Top-Down](https://www.intel.com/content/www/us/en/develop/documentation/vtune-cookbook/top/methodologies/top-down-microarchitecture-analysis-method.html) analysis ([tool](https://github.com/andikleen/pmu-tools/wiki/toplev-manual)) on real hardware and then further explore and optimize these bottlenecks using the [gem5](https://www.gem5.org/) cycle-accurate simulator.\n\n\n## Directory Structure\n\n- `benchmarks` contains all of the available benchmark source code and manifests.\n- `utils` contains utilities for use within serverless functions, e.g. the tracing module.\n- `tools` is for command-line tools and services useful outside of serverless functions, such as\n   deployment or invocation.\n- `runner` is for setting up self-hosted GitHub Actions runners.\n- `docs` contains additional documentation on a number of relevant topics.\n\n\n## Summary of Benchmarks\n- 2 microbenchmarks for benchmarking chained functions performance, data transfer performance in\nvarious patterns (pipeline, scatter, gather), and using different communication medium (AWS S3\nand inline transfers)\n- 2 microbenchmarks for simulating I/O bound tasks and CPU bounded tasks (Sleeping and Spinning benchmarks)\n- 8 real-world benchmarks\n   - MapReduce: [Corral](/benchmarks/corral) (golang), and an [aws-reference](/benchmarks/map-reduce)\n    python implementation of Aggregation Query from the representative\n    [AMPLab Big Data Benchmark](https://www.cs.cmu.edu/~pavlo/papers/benchmarks-sigmod09.pdf)\n   1node dataset.\n   - Real-time [video analytics](/benchmarks/video-analytics) (Python and Golang): recognizes objects in a video fragment\n   - ML models training: [stacking ensemble training](/benchmarks/stacking-training) and\n   [iterative hyperparameter tuning](/benchmarks/tuning-halving)\n   - ExCamera video decoding (gg): decoding of a video in parallel\n   - distributed compilation (gg): compiles LLVM in parallel\n   - fibonacci (gg): classic recursive implementation to find `n`th number in the sequence by calculating `n-1` and `n-2` in parallel\n - 33 standalone functions\n   - [AES](https://github.com/vhive-serverless/vSwarm/tree/main/benchmarks/aes), [Auth](https://github.com/vhive-serverless/vSwarm/tree/main/benchmarks/auth), [Fibonacci](https://github.com/vhive-serverless/vSwarm/tree/main/benchmarks/fibonacci): Same functionality implemented in the three different runtimes: Python, NodeJS, Golang.\n   - [Online shop](https://github.com/vhive-serverless/vSwarm/tree/main/benchmarks/online-shop): 9 functions implemented in various runtimes, ported from Googles [Online Boutique](https://github.com/GoogleCloudPlatform/microservices-demo)\n   - [Hotel reservation](https://github.com/vhive-serverless/vSwarm/tree/main/benchmarks/hotel-app): 7 microservices from DeathStarBenchs [Hotel Reservation Application](https://github.com/delimitrou/DeathStarBench/tree/master/hotelReservation) ported as standalone serverless microbenchmarks.\n   - [Video Processing](https://github.com/vhive-serverless/vSwarm/tree/main/benchmarks/video-processing): Converts an input video into grayscale. Utilises and depends on a database to store videos.\n   - [RNN Serving](https://github.com/vhive-serverless/vSwarm/tree/main/benchmarks/rnn-serving): Generates a string using an RNN model given a specific language.\n   - [Image Rotate](https://github.com/vhive-serverless/vSwarm/tree/main/benchmarks/image-rotate): Rotates input image by 90 degrees. Implemented in Python, Golang. Stores images in a database.\n   - [BERT](https://github.com/vhive-serverless/vSwarm/tree/main/benchmarks/bert), [GPTJ](https://github.com/vhive-serverless/vSwarm/tree/main/benchmarks/gptj): Large language models for inference tasks.\n   - [Video Analytics Standalone](https://github.com/vhive-serverless/vSwarm/tree/main/benchmarks/video-analytics-standalone): Preprocesses a video and runs an object detection model on it.\n   - [Compression](https://github.com/vhive-serverless/vSwarm/tree/main/benchmarks/compression): File compression using zlib.\n\n\nRefer to [this document](/benchmarks/README.md) for more detail on the differences and supported features of each benchmark.\n\n\n## Running Benchmarks\n\nDetails on each specific benchmark can be found in their relative subfolders. Every benchmark can\nbe run on a knative cluster, and most can also be run locally with `docker-compose`. Please see the\n[running benchmarks document](/docs/running_benchmarks.md) for detailed instructions on how to\nrun a benchmark locally or on a cluster.\n\nWe have a detailed outline on the benchmarking methodology used, which you can find [here](/docs/methodology.md).\n\n\n## Contributing a Benchmark\n\nWe openly welcome any contributions, so please get in touch if you're interested!\n\nBringing up a benchmark typically consists of dockerizing the benchmark functions to deploy and\ntest them with docker-compose, then integrating the functions with knative, and including the\nbenchmark in the CI/CD pipeline. Please refer to our documentation on\n[bringing up new benchmarks](/docs/adding_benchmarks.md)\nfor more guidance.\n\nWe also have some basic requirements for contributions the the repository, which are described\nin detail in our\n[Contributing to vHive](/docs/contributing_to_vhive.md)\ndocument.\n\n\n## License and copyright\n\nvSwarm is free. We publish the code under the terms of the MIT License that allows distribution, modification, and commercial use.\nThis software, however, comes without any warranty or liability.\n\nThe software is maintained by the [vHive Ecosystem](https://vhive-serverless.github.io/), [EASE lab](https://easelab.inf.ed.ac.uk/) the University of Edinburgh,\n[Stanford Systems and Networking Research](https://github.com/StanfordSNR).\n\n\n### Maintainers\n\n- Invoker, timeseriesdb, runners - Dmitrii Ustiugov: [GitHub](https://github.com/ustiugov),\n[twitter](https://twitter.com/DmitriiUstiugov/), [web page](https://ustiugov.github.io)\n- Corral Benchmark - Dmitrii Ustiugov: [GitHub](https://github.com/ustiugov),\n[twitter](https://twitter.com/DmitriiUstiugov/), [web page](https://ustiugov.github.io)\n- Map-Reduce, Stacking-training and Tuning-halving - Alan Nair [GitHub](https://github.com/alannair)\n- Chained Functions and Video analytics - Shyam Jesalpura [GitHub](https://github.com/shyamjesal)\n- GG benchmarks - Shyam Jesalpura [GitHub](https://github.com/shyamjesal)\n- Standalone functions - David Schall [GitHub](https://github.com/dhschall),[web page](https://dhschall.github.io/)\n- Multi cloud containers (lambda) - Alan Nair [GitHub](https://github.com/alannair)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvhive-serverless%2Fvswarm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvhive-serverless%2Fvswarm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvhive-serverless%2Fvswarm/lists"}