{"id":19448385,"url":"https://github.com/carv-ics-forth/frisbee","last_synced_at":"2025-04-25T02:31:03.872Z","repository":{"id":38019107,"uuid":"389942374","full_name":"CARV-ICS-FORTH/frisbee","owner":"CARV-ICS-FORTH","description":"Kubernetes-native framework for declarative testing of distributed systems. ","archived":false,"fork":false,"pushed_at":"2025-04-18T23:19:28.000Z","size":111599,"stargazers_count":51,"open_issues_count":26,"forks_count":5,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-19T09:41:52.642Z","etag":null,"topics":["benchmarking-suite","chaos-engineering","e2e-testing","fault-detection-and-diagnosis","kubernetes","load-testing","scalability-analysis","testing"],"latest_commit_sha":null,"homepage":"https://frisbee.dev","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CARV-ICS-FORTH.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-07-27T10:34:09.000Z","updated_at":"2025-03-07T13:19:29.000Z","dependencies_parsed_at":"2023-02-19T13:01:26.281Z","dependency_job_id":"277e7abc-1cb1-4407-91f5-dd260544d7f1","html_url":"https://github.com/CARV-ICS-FORTH/frisbee","commit_stats":null,"previous_names":[],"tags_count":41,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CARV-ICS-FORTH%2Ffrisbee","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CARV-ICS-FORTH%2Ffrisbee/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CARV-ICS-FORTH%2Ffrisbee/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CARV-ICS-FORTH%2Ffrisbee/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CARV-ICS-FORTH","download_url":"https://codeload.github.com/CARV-ICS-FORTH/frisbee/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250741898,"owners_count":21479693,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking-suite","chaos-engineering","e2e-testing","fault-detection-and-diagnosis","kubernetes","load-testing","scalability-analysis","testing"],"created_at":"2024-11-10T16:26:29.684Z","updated_at":"2025-04-25T02:31:03.361Z","avatar_url":"https://github.com/CARV-ICS-FORTH.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Frisbee - A Test Automation Framework For Kubernetes\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://www.vectorstock.com/royalty-free-vector/disc-golf-frisbee-eps-vector-25179185\"\u003e\n        \u003cimg src=\"docs/logo.jpg\" width=\"400\"\u003e\n    \u003c/a\u003e\n\u003c/p\u003e\n\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://frisbee.dev/\"\u003eWebsite\u003c/a\u003e |\n    \u003ca href=\"https://frisbee.dev/blog\"\u003eBlog\u003c/a\u003e |\n    \u003ca href=\"https://frisbee.dev/docs/\"\u003eDocs\u003c/a\u003e |\n    \u003ca href=\"mailto: fnikol@ics.forth.gr\"\u003eContact\u003c/a\u003e\n    \u003cbr /\u003e\u003cbr /\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca Code build and checks\u003e\n        \u003cimg src=\"https://github.com/CARV-ICS-FORTH/frisbee/actions/workflows/test-unit.yml/badge.svg\"\u003e\n    \u003c/a\u003e        \n    \u003ca href=\"https://github.com/carv-ics-forth/frisbee/blob/main/CONTRIBUTING.md\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/PRs-welcome-brightgreen.svg\"\u003e\n    \u003c/a\u003e    \n    \u003ca Releases\u003e\n        \u003cimg src=\"https://img.shields.io/github/v/release/carv-ics-forth/frisbee.svg?include_prereleases\"\u003e\n    \u003c/a\u003e    \n    \u003ca Downloads\u003e\n        \u003cimg src=\"https://img.shields.io/github/downloads/carv-ics-forth/frisbee/total.svg\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/carv-ics-forth/frisbee/blob/main/LICENSE\"\u003e\n        \u003cimg src=\"https://img.shields.io/github/license/carv-ics-forth/frisbee\"\u003e\n    \u003c/a\u003e    \n\u003c/p\u003e    \n\n## What is Frisbee ?\n\n**Frisbee** is the first Kubernetes framework designed to support **declarative end-to-end system testing** of containerized applications.\n\n:heavy_check_mark: Design end-to-end testing scenarios in an intuitive language!\n\n:heavy_check_mark: Run the actual production software - no mocks!\n\n:heavy_check_mark: Replay complex workloads through automated workflows!\n\n:heavy_check_mark: Combine Chaos Engineering with large-scale performance testing!\n\n:heavy_check_mark: Verify program behavior through programmable assertions!\n\n\nTo learn more about Frisbee, check the **[Walkthrough](https://frisbee.dev/docs/walkthrough)** tutorial or visit our **[Website](https://frisbee.dev)**.\n\n\n\n## Testing Patterns\n\nIn declarative testing, a test scenario focuses on what to accomplish rather than on the imperative details of how to manipulate the state of an application under test and verify the final application state against an expected state.\n\nThis approach not make tests more *readable, maintainable, and reproducible*, but it also help devops in identifying testing patterns.\n\nHere, you can see some testing patterns we have identified across different application domains.\n\n👉 [Databases](./examples/patterns/databases)\n\n👉 [Federated Learning](./examples/patterns/federated-learning)\n\n👉 [HPC](./examples/patterns/hpc)\n\n👉 [CI](./examples/patterns/ci)\n\n\n\n\n## Getting Started\n\n\n### Step 1 – Prerequsities:\n\nTo run Frisbee you must have access to a running Kubernetes cluster and ensure that [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)\nand  [Helm](https://helm.sh/docs/intro/install/) are installed on your system. \n\n\nFor quick testing, you can use [microk8s](https://microk8s.io/docs/getting-started) to deploy a local Kubernetes cluster on your machine. \n\n\n```shell\nsudo microk8s config \u003e ~/.kube/config\nsudo microk8s enable dns\nsudo microk8s enable ingress\nsudo microk8s enable helm3\nsudo snap alias microk8s.kubectl kubectl\nsudo snap alias microk8s.helm3 helm\n```\n\n\n\n### Step 2 – Install Frisbee on K8s cluster:\n\nFirstly, we need to download the Frisbee CLI:\n\n```shell\ncurl -sSLf https://frisbee.dev/install.sh | sudo bash\nexport PATH=$PATH:/usr/local/bin\n```\n\n\n\nSince the majority of Frisbee operations happens through this  CLI, it is suggested to enable the autocompletion.\n\n\n```shell\nsource \u003c(kubectl-frisbee completion bash)\n```\n\n\u003e As usual, use `tab twice` to get the CLI fields auto-completed.\n\n\n\nThe next step is to install Frisbee in your Kubernetes cluster. \n\n```shell\nkubectl-frisbee install production\nkubectl logs -l control-plane=frisbee-operator -n frisbee --follow\n```\n\n\n\nFinally, you can download the Frisbee project to get access to the ready-to-use examples.\n\n```shell\ngit clone git@github.com:CARV-ICS-FORTH/frisbee.git\ncd frisbee\n```\n\n\n\n### Step 3 – Submit a testing job:\n\nTo submit a testing job, the general syntax is:\n\n```shell\nkubectl-frisbee submit test \u003ctestName\u003e \u003cscenario.yaml\u003e \u003cpath to dependent charts\u003e\n```\n\nConventionally, we separate the test-cases from the templates of the system under evaluation. \n\n* [examples](examples): contains a list of test-cases.\n* [charts](charts): contains Helm charts that provide templates used in the test-cases.\n\n\nLet's try to run a scenario from the tutorial. \n\n```shell\nkubectl-frisbee submit test demo- ./examples/tutorial/15.performance-monitoring.yml ./examples/apps/iperf2/ ./charts/system/ \n```\n* **demo-**: demo is the naming prefix, and the `-` indicate an autogenerated postfix (e.g, demo-326)\n* [./examples/tutorial/15.performance-monitoring.yml](./examples/tutorial/15.performance-monitoring.yml): the scenario executes the iperf benchmark and the monitoring stack for observing its execution.\n* [./examples/apps/iperf2](./examples/apps/iperf2): provides the application templates used within the scenario.\n* [./charts/system](./charts/system): provides system-wide templates for the telemetry stack, chaos injection, etc.\n\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"docs/readme.assets/cli-submit.png\" width=\"400\"\u003e\n\u003c/p\u003e\n\n\n\n#### Inspect Submitted Jobs:\n\nTo get a list of submitted tests, use:\n\n```shell\nkubectl frisbee get tests\n```\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"docs/readme.assets/cli-get.png\" width=\"400\"\u003e\n\u003c/p\u003e\n\n\nNote that every test-case runs on a dedicated namespace (named after the test). To further dive into execution details\nuse:\n\n```shell\nkubectl frisbee inspect tests demo-326\n```\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"docs/readme.assets/cli-inspect.png\" width=\"900\"\u003e\n\u003c/p\u003e\n\n### Step 4 – Live Progress Monitoring:\n\nThe last section of `inspect` provides the URLs for accessing Prometheus/Grafana. \n\nNote that every scenario has its own monitoring stack in order to avoid interfering metrics.\n\n```shell\nfirefox $(kubectl frisbee inspect tests demo-326 | grep grafana- | awk '{print $3'})\n```\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"docs/readme.assets/grafana.png\" width=\"900\"\u003e\n    \u003cimg src=\"docs/readme.assets/grafana2.png\" width=\"900\"\u003e\n\u003c/p\u003e\n\n\nIn contrast to the vanilla Grafana which plots only the performance metrics, Frisbee\nprovides `Contextualized Visualizations` that contain information for:\n\n* Joining nodes (blue vertical lines)\n* Exiting nodes (orange vertical lines)\n* Fault-Injection (red ranges)\n\nInformation like that helps in `root-cause analysis`, as it makes it easy to correlate\nan `observed behavior back to a testing event`.\n\nFor example, in the next figure, it fairly easy to understand that `INSERT_ERROR` messages (`yellow line`) are triggered\nby a `fault-injection event`.\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"docs/readme.assets/contextualized-visualization.png\" width=\"900\"\u003e\n\u003c/p\u003e\n\n### Step 5 – Auto-generate test reports:\n\nFinally, Frisbee provides an advanced functionality for auto-generating reports for the tests.\n\n```shell\nkubectl-frisbee report test demo-326 ~/frisbee-reports --pdf  --force\n```\n\nThis will create report on `~/frisbee-reports` directory including the pdf from Grafana.\n\n\n\n## Features\n\n👉 Workflow templating to store commonly used workflows in the cluster.\n\n👉 DAG based declaration of testing workflows.\n\n👉 Step level input \u0026 outputs (template parameterization).\n\n👉 Conditional Execution (Time-Driven, Status-Driven, Performance-Driven).\n\n👉 Live Progress monitoring via Prometheus/Grafana.\n\n👉 Assertions and alerting of SLA violations.\n\n👉 Placement Policies (affinity/tolerations/node selectors).\n\n👉 Archiving Test results after executing for later access.\n\n👉 Garbage collection of completed resources.\n\n👉 Chaos-Engineering and Fault-Injection via Chaos-Mesh.\n\n👉 On-Demand reliable container attached storage.\n\n👉 CLI applications to test management and test inspection.\n\n\n\n## Citation\n\nIf you publish work that uses Frisbee, please cite Frisbee as follows:\n\n```bibtex\n@article{nikolaidis2021frisbee,\ntitle={Frisbee: automated testing of Cloud-native applications in Kubernetes},\nauthor={Nikolaidis, Fotis and Chazapis, Antony and Marazakis, Manolis and Bilas, Angelos},\njournal={arXiv preprint arXiv:2109.10727},\nyear={2021}\n}\n```\n\n\n\n## Contributing to Frisbee\n\nWe welcome contributions. Please see [CONTRIBUTING.md](CONTRIBUTING.md) to get started!\n\n\n\n## Acknowledgements\n\nThis project has received funding from the European Union's Horizon 2020 research and innovation programme under grant\nagreement No. 894204 (Ether, H2020-MSCA-IF-2019).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcarv-ics-forth%2Ffrisbee","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcarv-ics-forth%2Ffrisbee","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcarv-ics-forth%2Ffrisbee/lists"}