{"id":41183880,"url":"https://github.com/semagrow/kobe","last_synced_at":"2026-01-22T20:13:47.354Z","repository":{"id":42652979,"uuid":"47556626","full_name":"semagrow/kobe","owner":"semagrow","description":"Framework for benchmarking SPARQL query federators","archived":false,"fork":false,"pushed_at":"2024-06-26T14:17:51.000Z","size":59971,"stargazers_count":10,"open_issues_count":16,"forks_count":1,"subscribers_count":6,"default_branch":"devel","last_synced_at":"2024-06-26T17:29:47.711Z","etag":null,"topics":["benchmarking","big-data","database-benchmarking","distributed","federated","semagrow","sparql"],"latest_commit_sha":null,"homepage":"https://semagrow.github.io/kobe/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/semagrow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-12-07T14:21:39.000Z","updated_at":"2024-06-26T14:17:51.000Z","dependencies_parsed_at":"2024-06-26T17:23:27.000Z","dependency_job_id":"63a66988-b532-4b2d-8e10-3fd7e7330d50","html_url":"https://github.com/semagrow/kobe","commit_stats":{"total_commits":222,"total_committers":9,"mean_commits":"24.666666666666668","dds":"0.49549549549549554","last_synced_commit":"847bf4a93ccec7d44a5f2946ac75b1b9c3af43c0"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/semagrow/kobe","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/semagrow%2Fkobe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/semagrow%2Fkobe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/semagrow%2Fkobe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/semagrow%2Fkobe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/semagrow","download_url":"https://codeload.github.com/semagrow/kobe/tar.gz/refs/heads/devel","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/semagrow%2Fkobe/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28670366,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-22T19:36:09.361Z","status":"ssl_error","status_checked_at":"2026-01-22T19:36:05.567Z","response_time":144,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking","big-data","database-benchmarking","distributed","federated","semagrow","sparql"],"created_at":"2026-01-22T20:13:47.263Z","updated_at":"2026-01-22T20:13:47.326Z","avatar_url":"https://github.com/semagrow.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# KOBE: Cloud-Native Open Benchmark Engine for SPARQL Query Processors\n\nKOBE is a benchmarking system that leverages\n[Docker](https://docker.io) and [Kubernetes](https://kubernetes.io) in\norder to reproduce experiments of federated query processing in a\ncollections of data sources.\n\n## Overview\n\nIn the SPARQL query processing community, as well as in the wider\ndatabases community, benchmark reproducibility is based on releasing\ndatasets and query workloads. However, this paradigm breaks down for\nfederated query processors, as these systems do not manage the data\nthey serve to their clients but provide a data-integration abstraction\nover the actual query processors that are in direct contact with the\ndata.\n\nThe KOBE benchmarking engine is a system that aims to provide a\ngeneric platform to perform benchmarking and experimentation that can\nbe reproducible in different environments. It was designed with the\nfollowing objectives in mind:\n\n1. to allow for benchmark and experiment specifications to be\n   reproduced in different environments and be able to produce\n   comparable and reliable results;\n2. to ease the deployment of complex benchmarking experiments by\n   automating the tedious tasks of initialization and execution.\n\n## Installation\n\n### Prerequisites\n\n- `Kubernetes` \u003e= 1.8.0\n- `kubectl` configured for the Kubernetes cluster\n- `Helm` version 3 (for the Evaluation Metrics Extraction subsystem)\n- `nfs-commons` installed in the nodes of the cluster\n\n\u003csub\u003e**Note:** The following instructions were tested on Debian 12. Minor adjustments may be necessary for installation on other Linux distributions or operating systems.\u003c/sub\u003e\n\n### Get Kubernetes\n```sh\ncurl -LO \"https://dl.k8s.io/release/v1.20.7/bin/linux/amd64/kubectl\"\ncurl -LO \"https://dl.k8s.io/release/v1.20.7/bin/linux/amd64/kubectl.sha256\"\nsudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl\n```\n\n### Use Minikube [Optional]\n\nIf you are not using an existing Kubernetes cluster, you can quickly set up a local environment for testing and development using Minikube:\n\nDownload and install Minikube on your system:\n```sh\ncurl -LO https://storage.googleapis.com/minikube/releases/v1.21.0/minikube-linux-amd64\nsudo install minikube-linux-amd64 /usr/local/bin/minikube\n```\nPrepare Docker installation and setup:\n```sh\nsudo apt-get update\nsudo apt-get install ca-certificates curl\nsudo install -m 0755 -d /etc/apt/keyrings\nsudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc\nsudo chmod a+r /etc/apt/keyrings/docker.asc\necho \"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian $(. /etc/os-release \u0026\u0026 echo \"$VERSION_CODENAME\") stable\" | sudo tee /etc/apt/sources.list.d/docker.list \u003e /dev/null\nsudo apt-get update\nsudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin\nsudo usermod -aG docker $USER\nnewgrp docker\n```     \n\nStart Minikube with Docker driver:\n```sh\nminikube start --driver=docker\nkubectl cluster-info\n```\n\n### Enable NFS Support\n```sh\nsudo apt-get install nfs-common\n```\n\n### Install the Kubernetes operator\n\nKOBE needs the Kubernetes operator that needs to be installed in the\nKubernetes cluster. To quickly install the KOBE operator in a\nKubernetes cluster, you can use the `kobectl` script found in the\n[bin](bin/) directory:\n\n```\nexport PATH=`pwd`/bin:$PATH\nkobectl install operator .\n```\nIf you are using kubernetes version 1.15 and below you should instead use \n```\nkobectl install operator-v1beta1 \n```\n\nAlternatively, you could run the following commands:\n\n```\nkubectl apply -f operator/deploy/crds\nkubectl apply -f operator/deploy/service_account.yaml\nkubectl apply -f operator/deploy/clusterrole.yaml\nkubectl apply -f operator/deploy/clusterrole_binding.yaml\nkubectl apply -f operator/deploy/operator.yaml\n```\nFor Kubernetes version 1.15 and below  swap\n\n```\nkubectl apply -f operator/deploy/crds\n```\nwith\n```\nkubectl apply -f operator/deploy/crds-v1beta1\n```\n\nYou will get a confirmation message that each resource has\nsuccessfully been created.\nThis will set the operator running in your Kubernetes cluster and\nneeds to be done only once.\n\n### Install the Networking subsystem\n\nKOBE uses [Istio](https://istio.io/) to support network delays between the different \ndeployments. To install Istio first define the version:\n\n```\nexport ISTIO_VERSION=1.11.3\n```\n\nthen deploy Istio:\n```\nkobectl install istio .\n```\nAlternatively, you can consult the official \n[installation guide](https://istio.io/docs/setup/getting-started/) \nor you can type the following commands.\n\n```\ncurl -L https://istio.io/downloadIstio | sh -\n./istio-*/bin/istioctl manifest apply --set profile=default\n```\n\n### Install Helm\n\nKOBE uses Helm to simplify the management of dependencies within Kubernetes environments. To install Helm on your system, run:\n\n```\ncurl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3.\nchmod 700 get_helm.sh.\n./get_helm.sh.\n```\n\n### Install the Evaluation Metrics Extraction subsystem\n\nTo enable the evaluation metrics extraction subsystem, run\n```\nkobectl install efk .\n```\nor alternatively the following\n```\nhelm repo add elastic https://helm.elastic.co\nhelm repo add kiwigrid https://kiwigrid.github.io\nhelm install elasticsearch elastic/elasticsearch --set persistence.enabled=false --set replicas=1 --version 7.6.2\nhelm install kibana elastic/kibana --set service.type=NodePort --version 7.6.2\nhelm install fluentd kiwigrid/fluentd-elasticsearch -f operator/deploy/efk-config/fluentd-values.yaml --version 8.0.1\nkubectl apply -f operator/deploy/efk-config/kobe-kibana-configuration.yaml\n```\n\nThese result in the simplest setup of an single-node\n[Elasticsearch](https://github.com/elastic/helm-charts/blob/master/elasticsearch)\nthat does not persist data across pod recreation, a\n[Fluentd](https://github.com/kiwigrid/helm-charts/tree/master/charts/fluentd-elasticsearch)\n`DaemonSet` and a\n[Kibana](https://github.com/elastic/helm-charts/tree/master/kibana)\nnode that exposes a `NodePort`. \n\nAfter all pods are in `Running` state Kibana dashboards can be accessed\nat \n```\nhttp://\u003cNODE-IP\u003e:\u003cNODEPORT\u003e/app/kibana#/dashboard/\n``` \nwhere `\u003cNODE-IP\u003e` the IP of any of the Kubernetes worker nodes and\n`\u003cNODEPORT\u003e` the result of `kubectl get -o\njsonpath=\"{.spec.ports[0].nodePort}\" services kibana-kibana`.\n\nThe setup can be customized by changing the configuration parameters\nof each helm chart. Please check the corresponding documentation of\neach chart for more info.\n\n### Recommended Versions\n\nTo ensure compatibility, we recommend using the following versions of the dependencies:\n- `Kubernetes` = v1.20.7\n- `Minikube` = v1.21.0 (if used)\n- `Istio` = v1.11.3\n- `Helm` = v3\n- `Elasticsearch` = 7.6.2\n- `Kibana` = 7.6.2\n- `Fluentd` = 8.0.1\n\nThese versions have been tested and verified to work together.\n\n## Example\n\nThe typical workflow of defining a KOBE experiment is the following.\n1. Create one [DatasetTemplate](operator/docs/api.md#datasettemplate)\n   for each dataset server you want to use in your benchmark.\n2. Define your [Benchmark](operator/docs/api.md#benchmark),\n   which should contain a list of datasets and a list of queries.\n2. Create one [FederatorTemplate](operator/docs/api.md#federatortemplate)\n   for the federator engine you want to use in your experiment. \n3. Define an [Experiment](operator/docs/api.md#experiment) over your previously defined benchmark.\n\nSeveral examples of the above specifications can be found in the [examples](examples/) directory.\n\nIn the following, we show the steps for deploying an experiment on a simple benchmark that comprises\nthree queries over a Semagrow federation of two Virtuoso endpoints.\n\nYou can use the `kobectl` script found in the [bin](bin/) directory for controlling your experiments:\n\n```\nexport PATH=`pwd`/bin:$PATH\nkobectl help\n```\n\nFirst, apply the templates for Virtuoso and Semagrow:\n\n```\nkobectl apply examples/dataset-virtuoso/virtuosotemplate.yaml\nkobectl apply examples/federator-semagrow/semagrowtemplate.yaml\n```\nThen, apply the benchmark.\n\n```\nkobectl apply examples/benchmark-toybench/toybench-simple.yaml\n```\nBefore running the experiment, you should verify that the datasets are loaded.\nUse the following command:\n```\nkobectl show benchmark toybench-simple\n```\nWhen the datasets are loaded, you should get the following output:\n```\nNAME  STATUS\ntoy1  Running\ntoy2  Running\n```\nProceed now with the execution of the experiment:\n```\nkobectl apply examples/experiment-toybench/toyexp-simple.yaml\n```\nAs perviously, you can review the state of the experiment with the following command:\n```\nkobectl show experiment toyexp-simple\n```\nYou can now view the evaluation metrics in the Kibana dashboards.\n\nFor removing all of the above, issue the following commands:\n```\nkobectl delete experiment toyexp-simple\nkobectl delete benchmark toybench-simple\nkobectl delete federatortemplate semagrowtemplate\nkobectl delete datasettemplate virtuosotemplate\n```\nFor more advanced control options for KOBE, use [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/).\n\n## Removal\n\nTo remove KOBE from your cluster, run the following command:\n```\nkobectl purge .\n```\nTo remove KOBE operator manually, run\n```\nkubectl delete -f operator/deploy/operator.yaml\nkubectl delete -f operator/deploy/role.yaml\nkubectl delete -f operator/deploy/clusterrole_binding.yaml\nkubectl delete -f operator/deploy/clusterrole.yaml\nkubectl delete -f operator/deploy/service_account.yaml\nkubectl delete -f operator/deploy/crds\n```\nTo remove Istio manually, run\n```\n./istio-*/bin/istioctl manifest generate --set profile=default | kubectl delete -f -\nkubectl delete namespace istio-system\n```\nTo remove the evaluation metrics extraction subsystem manually, run\n```\nhelm uninstall elasticsearch\nhelm uninstall kibana\nhelm uninstall fluentd\nhelm repo remove elastic\nhelm repo remove kiwigrid\nkubectl delete jobs.batch kobe-kibana-configuration\nkubectl delete configmaps kobe-kibana-config\n```\nand then in each Kubernetes node\n```\nrm -rf /var/log/fluentd-buffers/\nrm /var/log/containers.log.pos\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsemagrow%2Fkobe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsemagrow%2Fkobe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsemagrow%2Fkobe/lists"}