{"id":13787847,"url":"https://github.com/spcl/serverless-benchmarks","last_synced_at":"2025-04-12T18:49:47.360Z","repository":{"id":39635855,"uuid":"227820874","full_name":"spcl/serverless-benchmarks","owner":"spcl","description":"SeBS: serverless benchmarking suite for automatic performance analysis of FaaS platforms.","archived":false,"fork":false,"pushed_at":"2025-03-26T02:40:31.000Z","size":2982,"stargazers_count":162,"open_issues_count":56,"forks_count":78,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-12T17:28:51.444Z","etag":null,"topics":["benchmark-framework","benchmarking","faas","performance-analysis","serverless"],"latest_commit_sha":null,"homepage":"https://mcopik.github.io/projects/sebs/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spcl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-12-13T11:04:54.000Z","updated_at":"2025-04-11T13:54:14.000Z","dependencies_parsed_at":"2023-02-09T15:16:30.636Z","dependency_job_id":"74bf100f-12a4-4c1d-ab5b-37cb78a16546","html_url":"https://github.com/spcl/serverless-benchmarks","commit_stats":{"total_commits":1156,"total_committers":24,"mean_commits":"48.166666666666664","dds":0.09083044982698962,"last_synced_commit":"bf5bc3562769cd875d1e54a3047f00fbe1fc7b97"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spcl%2Fserverless-benchmarks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spcl%2Fserverless-benchmarks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spcl%2Fserverless-benchmarks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spcl%2Fserverless-benchmarks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spcl","download_url":"https://codeload.github.com/spcl/serverless-benchmarks/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248618218,"owners_count":21134199,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark-framework","benchmarking","faas","performance-analysis","serverless"],"created_at":"2024-08-03T21:00:32.305Z","updated_at":"2025-04-12T18:49:47.339Z","avatar_url":"https://github.com/spcl.png","language":"Python","funding_links":[],"categories":["Conference"],"sub_categories":["Middleware"],"readme":"\n[![CircleCI](https://circleci.com/gh/spcl/serverless-benchmarks.svg?style=shield)](https://circleci.com/gh/spcl/serverless-benchmarks)\n![Release](https://img.shields.io/github/v/release/spcl/serverless-benchmarks)\n![License](https://img.shields.io/github/license/spcl/serverless-benchmarks)\n![GitHub issues](https://img.shields.io/github/issues/spcl/serverless-benchmarks)\n![GitHub pull requests](https://img.shields.io/github/issues-pr/spcl/serverless-benchmarks)\n[![Slack](https://img.shields.io/badge/Slack-Join%20%23serverless--benchmark-purple?logo=Slack)](https://join.slack.com/t/serverlessbenchmark/shared_invite/zt-30622ov74-_S9QeDjAJLZSe9bJC8tStw)\n\n# SeBS: Serverless Benchmark Suite\n\n**FaaS benchmarking suite for serverless functions with automatic build, deployment, and measurements.**\n\n![Overview of SeBS features and components.](docs/overview.png)\n\nSeBS is a diverse suite of FaaS benchmarks that allows automatic performance analysis of\ncommercial and open-source serverless platforms. We provide a suite of\n[benchmark applications](docs/benchmarks.md) and [experiments](docs/experiments.md)\nand use them to test and evaluate different components of FaaS systems.\nSee the [installation instructions](#installation) to learn how to configure SeBS to use selected\ncommercial and open-source serverless systems.\nThen, take a look at [usage instructions](docs/usage.md) to see how\nSeBS can automatically launch serverless functions and entire experiments in the cloud!\n\n\nSeBS provides support for **automatic deployment** and invocation of benchmarks on\ncommercial and black-box platforms\n[AWS Lambda](https://aws.amazon.com/lambda/),\n[Azure Functions](https://azure.microsoft.com/en-us/services/functions/),\nand [Google Cloud Functions](https://cloud.google.com/functions).\nFurthermore, we support the open-source platform [OpenWhisk](https://openwhisk.apache.org/)\nand offer a custom, Docker-based local evaluation platform.\nSee the [documentation on cloud providers](docs/platforms.md)\nfor details on configuring each platform in SeBS.\nThe documentation describes in detail [the design and implementation of our\ntool](docs/design.md), and see the [modularity](docs/modularity.md)\nsection to learn how SeBS can be extended with new platforms, benchmarks, and experiments.\nFind out more about our project in [a paper summary](https://mcopik.github.io/projects/sebs/).\n\nDo you have further questions that were not answered by our documentation?\nDid you encounter trouble installing and using SeBS?\nOr do you want to use SeBS in your work and you need new features?\n[Join our community on Slack](https://join.slack.com/t/serverlessbenchmark/shared_invite/zt-30622ov74-_S9QeDjAJLZSe9bJC8tStw) or open a GitHub issue.\n\nFor more information on how to configure, use, and extend SeBS, see our\ndocumentation:\n\n* [How to use SeBS?](docs/usage.md)\n* [Which benchmark applications are offered?](docs/benchmarks.md)\n* [Which experiments can be launched to evaluate FaaS platforms?](docs/experiments.md)\n* [How to configure serverless platforms?](docs/platforms.md)\n* [How SeBS builds and deploys functions?](docs/build.md)\n* [How SeBS package is designed?](docs/design.md)\n* [How to extend SeBS with new benchmarks, experiments, and platforms?](docs/modularity.md)\n\n### Publication\n\nWhen using SeBS, please cite our [Middleware '21 paper](https://dl.acm.org/doi/abs/10.1145/3464298.3476133).\nAn extended version of our paper is [available on arXiv](https://arxiv.org/abs/2012.14132), and you can\nfind more details about research work [in this paper summary](https://mcopik.github.io/projects/sebs/).\nYou can cite our software repository as well, using the citation button on the right.\n\n```\n@inproceedings{copik2021sebs,\n  author = {Copik, Marcin and Kwasniewski, Grzegorz and Besta, Maciej and Podstawski, Michal and Hoefler, Torsten},\n  title = {SeBS: A Serverless Benchmark Suite for Function-as-a-Service Computing},\n  year = {2021},\n  isbn = {9781450385343},\n  publisher = {Association for Computing Machinery},\n  address = {New York, NY, USA},\n  url = {https://doi.org/10.1145/3464298.3476133},\n  doi = {10.1145/3464298.3476133},\n  booktitle = {Proceedings of the 22nd International Middleware Conference},\n  pages = {64–78},\n  numpages = {15},\n  keywords = {benchmark, serverless, FaaS, function-as-a-service},\n  location = {Qu\\'{e}bec city, Canada},\n  series = {Middleware '21}\n}\n```\n\n## Installation\n\nRequirements:\n- Docker (at least 19)\n- Python 3.7+ with:\n    - pip\n    - venv\n- `libcurl` and its headers must be available on your system to install `pycurl`\n- Standard Linux tools and `zip` installed\n\n... and that should be all. We currently support Linux and other POSIX systems with Bash available.\nOn Windows, we recommend using WSL.\n\nTo install the benchmarks with a support for all platforms, use:\n\n```\n./install.py --aws --azure --gcp --openwhisk --local\n```\n\nIt will create a virtual environment in `python-venv`, and install necessary Python\ndependencies and third-party dependencies. To use SeBS, you must first active the new Python\nvirtual environment:\n\n```\n. python-venv/bin/activate\n```\n\nNow you can deploy serverless experiments :-)\n\nThe installation of additional platforms is controlled with the `--{platform}` and `--no-{platform}`\nswitches. Currently, the default behavior for `install.py` is to install only the\nlocal environment.\n\nTo verify the correctness of installation, you can use [our regression testing](docs/usage.md#regression).\n\n\u003e [!WARNING]\n\u003e Please do not use SeBS with `sudo`. There is no requirement to use any superuser permissions. **Make sure** that your Docker daemon is running and your user has sufficient permissions to use it (see [Docker documentation](https://docs.docker.com/engine/install/linux-postinstall/) on configuring your user to have non-sudo access to containers). Otherwise, you might see many \"Connection refused\" and \"Permission denied\" errors when using SeBS.\n\n\u003e [!WARNING]\n\u003e We use libcurl to make HTTP requests. `pycurl` will attempt to build its bindings and needs headers for that - make sure you have all development packages installed. If you see an error like this one: `src/pycurl.h:206:13: fatal error: gnutls/gnutls.h: No such file or directory`, it means that you are missing some of the dependencies.\n\n## Authors\n\n* [Marcin Copik (ETH Zurich)](https://github.com/mcopik/) - main author.\n* [Michał Podstawski (Future Processing SA)](https://github.com/micpod/) - contributed graph and DNA benchmarks, and worked on Google Cloud support.\n* [Laurin Brandner (ETH Zurich)](https://github.com/lbrndnr) - contributed serverless workflows.\n* [Nico Graf (ETH Zurich)](https://github.com/ncograf/) - contributed to the implementation of regression tests and bugfixes and helped with testing and documentation.\n* [Kacper Janda](https://github.com/Kacpro), [Mateusz Knapik](https://github.com/maknapik), [JmmCz](https://github.com/JmmCz), AGH University of Science and Technology - contributed together Google Cloud support.\n* [Grzegorz Kwaśniewski (ETH Zurich)](https://github.com/gkwasniewski) - worked on the modeling experiments.\n* [Paweł Żuk (University of Warsaw)](https://github.com/pmzuk) - contributed OpenWhisk support.\n* [Sascha Kehrli (ETH Zurich)](https://github.com/skehrli) - contributed local measurement of Docker containers.\n* [Kaleab](https://github.com/Kaleab-git) - contributed to SeBS local backend to make it portable between platforms and more robust on non-Linux systems.\n* [lawrence910426](https://github.com/lawrence910426) - contributed color-coded output to SeBS CLI.\n* [Abhishek Kumar](https://github.com/octonawish-akcodes) - contributed new language versions and Knative support.\n* [Prajin Khadka](https://github.com/prajinkhadka) - contributed new language versions, container support, and ARM builds.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspcl%2Fserverless-benchmarks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspcl%2Fserverless-benchmarks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspcl%2Fserverless-benchmarks/lists"}