{"id":34730191,"url":"https://github.com/lucadibello/gramine-java-benchmark","last_synced_at":"2026-04-27T02:31:44.261Z","repository":{"id":319523943,"uuid":"1078519023","full_name":"lucadibello/gramine-java-benchmark","owner":"lucadibello","description":"Evaluating Java runtime performance and overhead under Gramine (Intel SGX)","archived":false,"fork":false,"pushed_at":"2025-10-27T18:22:34.000Z","size":2661,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-27T19:29:44.882Z","etag":null,"topics":["benchmark","confidential-computing","graalvm","graalvm-native-image","gramine","java","multithreading","sgx","tls"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucadibello.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-17T21:40:44.000Z","updated_at":"2025-10-22T23:42:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"6fa0f774-0158-4f43-aa39-579a3a557a6e","html_url":"https://github.com/lucadibello/gramine-java-benchmark","commit_stats":null,"previous_names":["lucadibello/gramine-java-benchmark"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lucadibello/gramine-java-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucadibello%2Fgramine-java-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucadibello%2Fgramine-java-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucadibello%2Fgramine-java-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucadibello%2Fgramine-java-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucadibello","download_url":"https://codeload.github.com/lucadibello/gramine-java-benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucadibello%2Fgramine-java-benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32320238,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-26T23:26:28.701Z","status":"online","status_checked_at":"2026-04-27T02:00:06.769Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","confidential-computing","graalvm","graalvm-native-image","gramine","java","multithreading","sgx","tls"],"created_at":"2025-12-25T02:56:27.945Z","updated_at":"2026-04-27T02:31:44.250Z","avatar_url":"https://github.com/lucadibello.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Gramine Java Benchmark\n\nBenchmark suite that evaluates a differentially private binary aggregation tree inside a Gramine-protected JVM. A TLS client streams floating-point values to the enclave; the server keeps the noisy private sum and exposes the familiar `initBinaryAggregation`, `addToBinaryAggregation`, and `getBinaryAggregationSum` calls. The layout mirrors the companion Java aggregation benchmarks so that results stay comparable while swapping Teaclave SGX for Gramine.\n\n## Project Layout\n\n- `src/server/` – TLS enclave server (`com.benchmark.gramine.enclave.BenchServer`) executed inside Gramine. Implements the command protocol (`INIT`, `ADD`, `GET`) and logs each ingestion for traceability.\n- `src/server/com/.../dp/BinaryAggregationTree.java` – Gaussian-noise aggregation tree shared with the other benchmark suites to preserve behaviour.\n- `src/client/` – Host harness (`com.benchmark.gramine.host.BenchClient`) that prepares workloads, drives weak/strong scaling runs, and prints JSON metrics.\n- `tools/run-benchmarks.py` – Automation wrapper that builds each server variant, runs the client once, and writes combined CSV/JSON artifacts under `scaling-results/\u003ctimestamp\u003e/`.\n- `tools/generate_plots.py` – Turns the CSV/JSON outputs into PNG plots (per-variant throughput, speedup/efficiency, and startup-time comparisons).\n\n## Benchmark Approach\n\nEach benchmark execution follows three phases that align with the companion Teaclave suite:\n\n1. **Baseline workload** – The client uses `GRAMINE_BENCH_DATA_SIZE` with the smallest requested thread count to derive the per-thread workload. A short warmup runs before measurements.\n2. **Weak scaling** – The number of worker threads increases while the per-thread workload remains fixed.\n3. **Strong scaling** – The total workload stays fixed while the number of worker threads increases.\n\nThe client prints JSON summaries for all phases so downstream plotting scripts can consume the output directly.\n\n## Results\n\nExample dataset: `scaling-results/20251030_004543/` (data size = 1024, sigma = 0.5, warmup = 3, measure = 5). The run compares three server variants executed through Gramine: `jvm-gramine`, `native-dynamic`, and `native-static`. Artifacts land in `scaling-results/\u003ctimestamp\u003e/` and plots in `plots/`.\n\n### Strong Scaling\n\nThroughput scales from a single client up to 16 clients before Gramine scheduling overheads flatten the gains. Speedup and efficiency are highest for the native variants (`native-dynamic`, `native-static`), reaching ~5× speedup at eight clients. Refer to `plots/\u003cvariant\u003e_strong_throughput.png` and `plots/\u003cvariant\u003e_strong_speedup_efficiency.png` for the detailed traces.\n\n#### JVM (Gramine)\n\n![JVM Gramine strong throughput](plots/jvm-gramine_strong_throughput.png)\n\n![JVM Gramine strong speedup \u0026 efficiency](plots/jvm-gramine_strong_speedup_efficiency.png)\n\n#### Native (Dynamic)\n\n![Native dynamic strong throughput](plots/native-dynamic_strong_throughput.png)\n\n![Native dynamic strong speedup \u0026 efficiency](plots/native-dynamic_strong_speedup_efficiency.png)\n\n#### Native (Static)\n\n![Native static strong throughput](plots/native-static_strong_throughput.png)\n\n![Native static strong speedup \u0026 efficiency](plots/native-static_strong_speedup_efficiency.png)\n\n### Weak Scaling\n\nWeak scaling stays close to linear: aggregate throughput increases steadily as threads are added, and the native variants show the largest improvements. See `plots/\u003cvariant\u003e_weak_throughput.png` and `plots/\u003cvariant\u003e_weak_speedup_efficiency.png`.\n\n#### JVM (Gramine)\n\n![JVM Gramine weak throughput](plots/jvm-gramine_weak_throughput.png)\n\n![JVM Gramine weak speedup \u0026 efficiency](plots/jvm-gramine_weak_speedup_efficiency.png)\n\n#### Native (Dynamic)\n\n![Native dynamic weak throughput](plots/native-dynamic_weak_throughput.png)\n\n![Native dynamic weak speedup \u0026 efficiency](plots/native-dynamic_weak_speedup_efficiency.png)\n\n#### Native (Static)\n\n![Native static weak throughput](plots/native-static_weak_throughput.png)\n\n![Native static weak speedup \u0026 efficiency](plots/native-static_weak_speedup_efficiency.png)\n\n### Startup Time\n\n`plots/startup_times.png` summarises the bootstrap time for each server variant when launched through Gramine. Native images start faster than the JVM-backed variant once the enclave is provisioned.\n\n![Startup times](plots/startup_times.png)\n\n### Reproducing the Figures\n\n```bash\npython -m venv .venv\nsource .venv/bin/activate\npip install --upgrade pip matplotlib numpy\npython tools/generate_plots.py \\\n  --results scaling-results/20251030_004543/scaling_results.csv \\\n  --startup scaling-results/20251030_004543/benchmark_results.json \\\n  --output plots\n```\n\n## Getting Started\n\nYou can work from the host OS or inside the preconfigured devcontainer.\n\n### Option 1: Devcontainer (recommended)\n\n1. Install Docker and either the VS Code Dev Containers extension or the `devcontainer` CLI.\n2. From the repository root run `task devcontainer` **or** open the folder in VS Code and choose “Reopen in Container”.\n3. The container provisions GraalVM, Gramine, and the helper scripts automatically.\n\nConvenient `go-task` entries:\n\n| Task | Description |\n|------|-------------|\n| `task devcontainer` | Build, start, and attach to the devcontainer (wrapper around the tasks below). |\n| `task devcontainer-up` | Start or reuse the devcontainer without attaching. |\n| `task devcontainer-attach` | Exec into the running devcontainer shell. |\n| `task devcontainer-down` | Stop and remove the container and volumes. |\n| `task devcontainer-recreate` | Rebuild the container from scratch for a clean environment. |\n\n### Option 2: Local Toolchain\n\nInstall the dependencies locally:\n\n- GraalVM (JVM plus `native-image`)\n- Gramine (`gramine`, `gramine-sgx`, `gramine-manifest`)\n- GNU Make, Python 3.9+, and OpenJDK tooling\n\nClone the repository and continue with the build instructions below.\n\n## Building\n\nFrom the project root:\n\n```bash\n# Compile client/server classes\nmake server client\n\n# Generate TLS certificates (if not already produced)\n./tools/generate-certs.sh\n\n# Optional: build GraalVM native images for the TLS server/client set\nmake APP_NAME=native-bench-dynamic STATIC_NATIVE=0 SGX=1 all\nmake APP_NAME=native-bench-static STATIC_NATIVE=1 SGX=1 all\n```\n\n`tools/run-benchmarks.py` runs `make clean` before each variant build so the benchmarks always start from a known baseline.\n\n## Running Benchmarks\n\nThe automation wrapper builds the selected variants, launches the Gramine server, and drives one client run per variant:\n\n```bash\npython tools/run-benchmarks.py --variants jvm-gramine native-dynamic native-static\n```\n\nKey options:\n\n| Option | Description |\n|--------|-------------|\n| `--variants \u003clist\u003e` | Limit the run to the specified variants. |\n| `--all` | Execute all variants (default when no subset is provided). |\n| `--output \u003cdir\u003e` | Override the target directory under `scaling-results/`. |\n\nEach run produces:\n\n- `scaling-results/\u003ctimestamp\u003e/benchmark_results.json` – Startup metrics and per-variant summaries.\n- `scaling-results/\u003ctimestamp\u003e/scaling_results.csv` – Flattened metrics per variant and scaling mode.\n- `scaling-results/\u003ctimestamp\u003e/logs/*.out` / `*.err` – Raw client output (and stderr when present).\n\n## Configuration\n\nDefaults come from environment variables or an `.env` file at the repository root:\n\n```\nGRAMINE_BENCH_SIGMA=0.5\nGRAMINE_BENCH_WEAK_SCALES=1,2,4,8,16,32\nGRAMINE_BENCH_STRONG_SCALES=1,2,4,8,16,32\nGRAMINE_BENCH_DATA_SIZE=1024\nGRAMINE_BENCH_WARMUP=3\nGRAMINE_BENCH_MEASURE=5\nGRAMINE_BENCH_NATIVE_PARALLELISM=32\n```\n\nApply them in one shot:\n\n```bash\nset -a\nsource .env\nset +a\n```\n\nCLI flags override any setting (`java com.benchmark.gramine.host.BenchClient --help` lists all options).\n\n## Collecting Metrics\n\nRedirect the client output to capture the workload, weak scaling, and strong scaling summaries for custom post-processing:\n\n```bash\npython tools/run-benchmarks.py --variants native-dynamic \u003e scaling-results/latest-run.json\n```\n\nThe JSON contains per-pass throughput/latency measurements; the CSV is convenient for plotting tools that prefer tabular data.\n\n## Plotting\n\nAfter collecting metrics, generate the PNG figures:\n\n```bash\npython tools/generate_plots.py \\\n  --results scaling-results/\u003ctimestamp\u003e/scaling_results.csv \\\n  --startup scaling-results/\u003ctimestamp\u003e/benchmark_results.json \\\n  --output plots\n```\n\nThe script writes:\n\n- `plots/startup_times.png`\n- `plots/\u003cvariant\u003e_strong_throughput.png`\n- `plots/\u003cvariant\u003e_strong_speedup_efficiency.png`\n- `plots/\u003cvariant\u003e_weak_throughput.png`\n- `plots/\u003cvariant\u003e_weak_speedup_efficiency.png`\n\n## References\n\n- [Gramine Documentation](https://gramine.readthedocs.io/)\n- [GraalVM Native Image](https://www.graalvm.org/native-image/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucadibello%2Fgramine-java-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucadibello%2Fgramine-java-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucadibello%2Fgramine-java-benchmark/lists"}