{"id":20433885,"url":"https://github.com/flashbots/geth-sgx-gramine","last_synced_at":"2025-04-12T21:08:50.974Z","repository":{"id":71774072,"uuid":"578280770","full_name":"flashbots/geth-sgx-gramine","owner":"flashbots","description":"Geth-in-SGX provides an example of running go-ethereum in SGX","archived":false,"fork":false,"pushed_at":"2024-03-19T16:36:02.000Z","size":53,"stargazers_count":75,"open_issues_count":3,"forks_count":8,"subscribers_count":20,"default_branch":"main","last_synced_at":"2025-04-12T21:08:31.477Z","etag":null,"topics":["go-ethereum","gramine","sgx"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flashbots.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.addendum.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-14T17:21:47.000Z","updated_at":"2025-01-14T18:47:33.000Z","dependencies_parsed_at":"2024-01-11T12:44:49.663Z","dependency_job_id":"0a0f742d-845c-4681-8d07-91dfd071b9a6","html_url":"https://github.com/flashbots/geth-sgx-gramine","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flashbots%2Fgeth-sgx-gramine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flashbots%2Fgeth-sgx-gramine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flashbots%2Fgeth-sgx-gramine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flashbots%2Fgeth-sgx-gramine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flashbots","download_url":"https://codeload.github.com/flashbots/geth-sgx-gramine/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248631676,"owners_count":21136562,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go-ethereum","gramine","sgx"],"created_at":"2024-11-15T08:22:10.082Z","updated_at":"2025-04-12T21:08:50.948Z","avatar_url":"https://github.com/flashbots.png","language":"C","funding_links":[],"categories":["Codes \u0026 Repos"],"sub_categories":[],"readme":"# Geth-in-SGX\n\nThis repository contains an example for running Geth in [Gramine](https://gramine.readthedocs.io/en/stable/), a libOS for SGX enclaves. It includes the Makefile, a template for generating the manifest and a small patch-set to make go-ethereum run in Gramine. A design decision was taken to store the complete geth database in memory via `tmpfs`. This decision was made for performance reasons but also to obfuscate IO access. Therefor the memory requirements of Geth-in-SGX are significant. It also means the database state does not persist after the application exits.\n\nFor more background and details, see also:\n\n* https://writings.flashbots.net/block-building-inside-sgx ([discussion](https://collective.flashbots.net/t/block-building-inside-sgx/1373))\n* https://writings.flashbots.net/geth-inside-sgx ([discussion](https://collective.flashbots.net/t/running-geth-within-sgx-our-experience-learnings-and-code-flashbots/938))\n\n# Prerequisites\n\n**OS:** Ubuntu 20.04, Linux Kernel \u003e= 5.11\n\n**go-ethereum:** v1.10.22 and up supported\n\n**Hardware:** CPU supporting SGX2 (Intel Skylake and newer), +64GB EPC Enclave Memory (for mainnet), +1TB Swap (for mainnet)\n\nFollow the [Gramine Quickstart](https://gramine.readthedocs.io/en/stable/quickstart.html) in particular the sections [Install Gramine](https://gramine.readthedocs.io/en/stable/quickstart.html#install-gramine) and [Prepare a signing key](https://gramine.readthedocs.io/en/stable/quickstart.html#prepare-a-signing-key).\n\n## Build dependencies\n```\nsudo snap install --classic ego-dev\nsudo apt-get install -y libssl-dev gnupg software-properties-common build-essential ca-certificates git\n```\n\n## Attestation dependencies\n\nAttestation requires additional attestation infrastructure to be set up and configured. Enclaves running on Azure can make use of Azure's attestation infrastructure `a)`, whereas elsewhere, `b)` the attestation infrastructure including the provisioning certificate caching server (PCCS) needs to be set up and configured properly.\n\n```\nsudo apt-key adv --fetch-keys 'https://download.01.org/intel-sgx/sgx_repo/ubuntu/intel-sgx-deb.key'\nsudo add-apt-repository \"deb [arch=amd64] https://download.01.org/intel-sgx/sgx_repo/ubuntu `lsb_release -cs` main\"\nsudo apt-get update \u0026\u0026 sudo apt-get install -y libsgx-dcap-ql\n```\n### a) Azure cloud attestation\n```\nsudo apt-key adv --fetch-keys 'https://packages.microsoft.com/keys/microsoft.asc'\nsudo apt-add-repository 'https://packages.microsoft.com/ubuntu/20.04/prod main'\nsudo apt-get update \u0026\u0026 sudo apt-get install -y az-dcap-client\n```\n### b) Attestation via Intel - PCCS\n```\nsudo apt-get install -y libsgx-dcap-default-qpl sgx-dcap-pccs\n```\n\n# Builder Quickstart\n\n\u003e **IMPORTANT REQUIREMENTS:** Be sure to have a beacon-chain running, and a jwt secret configured at `/etc/jwt.hex`.\n\nThe instructions are tailored to Sepolia because of the minimal hardware requirements. The same instructions also work for a mainnet builder setup, although it's hardware requirements are much higher.\n\nGenerate the SGX signing key\n```\ngramine-sgx-gen-private-key\n```\n\n## Build Geth Enclanve\n```\n# Sepolia\nmake SGX=1 TLS=1 ENCLAVE_SIZE=64G SEPOLIA=1\n\n# Mainnet\nmake SGX=1 TLS=1 ENCLAVE_SIZE=2048G MAINNET=1\n```\n\n## Start enclave\n```\nBUILDER_SECRET_KEY=\u003cbuilder bls key\u003e BUILDER_TX_SIGNING_KEY=\u003cbuilder tx key\u003e gramine-sgx ./geth\n```\n\n## Attest enclave\n\nExtract measurements from Gramine signature\n```\ngramine-sgx-sigstruct-view geth.sig\n```\n\nBuild the `attest` tool\n```\nmake attest\n```\n\nAttest the enclave via the https endpoint\n```\nAPPLICATION_HOST=\u003cenclave host\u003e APPLICATION_PORT=8545 ./attest dcap \\\n        \u003cexpected mrenclave\u003e \u003cexpected mrsigner\u003e \u003cexpected isv_prod_id\u003e \u003cexpected isv_svn\u003e\n```\n\n```\n./attest dcap ce63bf3f3c29cb5b2c4f8ace497a602b4d3778d051922ba493dc08ebd0649ef3 39a3807530c976387e90a3134ea8bec28bcb4857e79db3ab5eb0e7df6996608e 0 0\n[ using our own SGX-measurement verification callback (via command line options) ]\n  - ignoring ISV_PROD_ID\n  - ignoring ISV_SVN\n\n  . Seeding the random number generator... ok\n  . Connecting to tcp/localhost/8545... ok\n  . Setting up the SSL/TLS structure... ok\n ok\n  . Installing RA-TLS callback ... ok\n  . Performing the SSL/TLS handshake... ok\n  . Verifying peer X.509 certificate... ok\n  \u003e Write to server: 18 bytes written\n\nGET / HTTP/1.0\n\n  \u003c Read from server: 89 bytes read\n\nHTTP/1.0 200 OK\nVary: Origin\nDate: Thu, 23 Feb 2023 15:18:57 GMT\nContent-Length: 0\n```\n\n# Setting Geth Arguments\n\nTo be able to trust an instance of Geth running in SGX, Geth's arguments need to be attestable. To achieve this, the arguments are defined at build time and saved to a file, which is then added to the list of trusted files and thus part of the TCB.\n\n```\ngramine-argv-serializer \\\n        ./geth_init \\\n                --http \\\n                --ws \\\n                --authrpc.jwtsecret=/etc/jwt.hex \\\n        \u003e geth.args\n```\n\n# Generating the manifest\n\n\n## Important Makefile Environment Variables\n\n| Variable       | Default                                | Description                                                                                                            |\n|----------------|----------------------------------------|------------------------------------------------------------------------------------------------------------------------|\n| `SGX`          | _empty_                                | Creates files for `gramine-sgx` execution, if `SGX!=1`, files for `gramine-direct` execution are created               |\n| `DEBUG`        | _empty_                                | Controls Gramine's `DEBUG_LOG` verbosity, `1=='debug'`, else `'error'`                                                 |\n| `ENCLAVE_SIZE` | `1024G`                                | Amount of memory allocated to the enclave, `1024G` is the minimum amount required to fit the Ethereum mainnet DB       |\n| `GETH_REPO`    | `https://github.com/flashbots/builder` | Location of the `go-ethereum` source code                                                                              |\n| `GETH_BRANCH`  | `main`                                 | `go-ethereum` branch                                                                                                   |\n| `MAINNET`      | _empty_                                | Set to `1` to create a `geth.args` file for the default mainnet builder configuration                                  |\n| `SEPOLIA`      | _empty_                                | Set to `1` to create a `geth.args` file for the default sepolia builder configuration                                  |\n| `RA_TYPE`      | `dcap`                                 | Controls attestation functionality, set to `none` to disable                                                           |\n| `TLS`          | _empty_                                | Set to `1` to apply `go-ethereum` `RA-TLS` patch                                                                       |\n| `PROTECT`      | _empty_                                | Set to `1` to apply patch that limits `go-ethereum` unauthorized http and ws endpoints to the `eth_sendBundle` method. |\n| `ISVPRODID`    | `0`                                    | Product ID of the Enclave                                                                                              |\n| `ISVSVN`       | `0`                                    | Security Version Number of the Enclave                                                                                 |\n\n## Building for Linux\n\nRun `make` (non-debug) or `make DEBUG=1` (debug) in the directory.\n\n## Building for SGX\n\nRun `make SGX=1` (non-debug) or `make SGX=1 DEBUG=1` (debug) in the directory.\n\n## Building with different Geth source repository\n\n`go-ethereum` version `v1.10.22` and newer should compile without modification. Older versions need adaption of Makefile and go-ethereum patch-set. The patch-set contains minor changes to go-ethereum and the goleveldb dependencies which are required to run Geth in Gramine.\n\n# Run Geth-in-SGX\n\nRun Geth under Gramine by executing this command:\n```\ngramine-sgx ./geth\n```\n\nTo run Gramine in non-SGX (direct) mode, replace `gramine-sgx` with\n`gramine-direct` and remove `SGX=1` in the commands above.\n\n## Runtime environment variables\n\n| Variable                 \t| Default            \t| Description                                                                                                                                                 \t|\n|--------------------------\t|--------------------\t|-------------------------------------------------------------------------------------------------------------------------------------------------------------\t|\n| `COPY_DATABASE`          \t| _empty_            \t| Set to `true`. Copies an existing geth database into the enclave. The existing database **must** be located in the hard coded path `data/synced_state/` \t|\n| `FAKE_PROPOSER`          \t| _empty_            \t| Sets a fake validator that will automatically be configured to always propose the next block. Required for the `Boost Relay communication PoC`              \t|\n| `BUILDER_SECRET_KEY`     \t| _empty_            \t| Builder key used for signing blocks                                                                                                                         \t|\n| `BUILDER_TX_SIGNING_KEY` \t| _empty_            \t| private key of the builder used to sign payment transaction, must be the same as the coinbase address                                                       \t|\n| `RATLS_CRT_PATH`         \t| `/tmp/tlscert.der` \t| Location of the RA-TLS certificate. **Must** be set in the manifest at build time to be part of TCB                                                         \t|\n| `RATLS_KEY_PATH`         \t| `/tmp/tlskey.der`  \t| Location of the RA-TLS key. **Must** be set in the manifest at build time to be part of TCB                                                                 \t|\n\n## Provide jwt secret file\n\nThe jwt secret file for authrpc communication is expected in the hard coded path `/etc/jwt.hex`. The path of the file still needs to be passed via argument input: `--authrpc.jwtsecret /etc/jwt.hex`\n\n## Running Enclave with memory size that does not fit in RAM\n\nIt is possible (and in case of running geth in mainnet - necessary) to run an SGX enclave where the enclave size is bigger than the systems RAM. In such situations, memory pages will get swapped out to a swap file. Make sure you have a big enough swap file mounted. To improve overall system responsivness, it is recommended to increase the `swapiness` kernel parameter.\n```\nsysctl vm.swappiness=80\n```\n\n# Attestation\n\nRecommended reading: [Gramine attestation docs](https://gramine.readthedocs.io/en/stable/attestation.html)\n\n## How does the attestation work?\nGramine's RA-TLS implementation is used to provide a convenient way of attestation and secure communication with the Geth node. RA-TLS integrates Intel SGX remote attestation into the TLS connection setup. Conceptually, it extends the standard X.509 certificate with SGX-related information (SGX quote). The additional information allows the remote user (verifier) of the certificate to verify that it is indeed communicating with an SGX enclave (attester).\n\nThe RA-TLS certificate is created by default on startup. Attestation functionality can be disabled via the `RA_TYPE` environment variable.\n\nOnly DCAP attestation is supported, legacy EPID attestation is **not supported**.\n\n## Attestation verification\n\nAttestation verification requires `4` values: `mrenclave` ,`mrsigner`, `isv_prod_id`, `isv_svn`. They can be extracted from the Gramine signature with the help of the `gramine-sgx-sigstruct-view` tool.\n```\ngramine-sgx-sigstruct-view geth.sig\n```\n\nBuild the `attest` tool to verify a `RA-TLS` connection.\n```\nmake attest\n```\n\nAttest the enclave via a connection that serves the `RA-TLS` certificate.\n```\nAPPLICATION_HOST=\u003cenclave host\u003e APPLICATION_PORT=\u003cenclave port\u003e ./attest dcap \\\n        \u003cexpected mrenclave\u003e \u003cexpected mrsigner\u003e \u003cexpected isv_prod_id\u003e \u003cexpected isv_svn\u003e\n```\n\n## Using the RA-TLS certificate within Geth\n\nGeth does not have https endpoints by default. The `TLS` environment variable can be set at build time to apply a simple patch to `go-ethereum`, which will wrap a TLS layer around Geths `http` and `ws` endpoints. The TLS connection is automatically configured to serve the `RA-TLS` certificate created on startup.\n\nIt is also possible to create a TLS implementation from scratch. `RATLS_CRT_PATH` and `RATLS_KEY_PATH` environment variables control the file locations and are passed to `geth_init`, which creates key and certificate at said location. The environment variables are subsequently passed on when starting Geth.\n\n# Attribution\nGeth-SGX is developed and tested by [konVera](https://konvera.io) in collaboration with Flashbots\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflashbots%2Fgeth-sgx-gramine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflashbots%2Fgeth-sgx-gramine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflashbots%2Fgeth-sgx-gramine/lists"}