{"id":20433896,"url":"https://github.com/flashbots/hindsight","last_synced_at":"2026-02-15T19:03:53.067Z","repository":{"id":197053436,"uuid":"661497043","full_name":"flashbots/hindsight","owner":"flashbots","description":"Retroactively estimate Uniswap-ish MEV on Flashbots MEV-Share by simulating backrun-arbitrages.","archived":false,"fork":false,"pushed_at":"2024-04-26T07:09:22.000Z","size":1051,"stargazers_count":126,"open_issues_count":14,"forks_count":18,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-11-15T00:05:36.269Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flashbots.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-03T02:26:17.000Z","updated_at":"2025-09-11T09:23:35.000Z","dependencies_parsed_at":null,"dependency_job_id":"f34bf746-50f0-4d79-be50-15988efa4bb8","html_url":"https://github.com/flashbots/hindsight","commit_stats":null,"previous_names":["flashbots/hindsight"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/flashbots/hindsight","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flashbots%2Fhindsight","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flashbots%2Fhindsight/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flashbots%2Fhindsight/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flashbots%2Fhindsight/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flashbots","download_url":"https://codeload.github.com/flashbots/hindsight/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flashbots%2Fhindsight/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29487397,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-15T15:33:17.885Z","status":"ssl_error","status_checked_at":"2026-02-15T15:32:53.698Z","response_time":118,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T08:22:11.814Z","updated_at":"2026-02-15T19:03:53.035Z","avatar_url":"https://github.com/flashbots.png","language":"Rust","funding_links":[],"categories":["PBS - Proposer-Builder Separation"],"sub_categories":[],"readme":"# Hindsight\n\n\u003c!-- show ./header-img.png --\u003e\n![hindsight visual algorithm](header-img.png)\n\n_Hindsight is an arbitrage simulator written in Rust which estimates the historical value of (Uniswap) MEV from Flashbots MEV-Share events._\n\nThe simulation core uses [revm](https://github.com/bluealloy/revm) to simulate arbs locally by first getting state diffs from an Ethereum archive node that supports the `trace_callMany` API (see [requirements](#requirements) for node recommendations).\n\n\u003e Just a warning: ⚠️ running Hindsight on a hosted node may require a high rate limit, which can be expensive.\n\nThe arbitrage strategy implemented here is a relatively simple two-step arb: after simulating the user's trade, we simulate swapping WETH for tokens on the exchange with the best rate (with the user's trade accounted for) and then simulate selling them on whichever other supported exchange gives us the best rate. Currently, Uniswap V2/V3 and SushiSwap are supported. More exchanges may be added to improve odds of profitability.\n\nSimulated arbitrage attempts are saved in a MongoDB database by default, for dead-simple storage that allows us to change our data format as needed with no overhead. Postgres is also supported, but does not currently save all the same fields that Mongo does.\n\n## ⚠️ limitations ⚠️\n\nHindsight is still in development, and is not stable. If you find a bug, please [open an issue](https://github.com/zeroXbrock/hindsight/issues).\n\nThis project is an experiment. The profits estimated by this system are by no means definitive; they more accurately represent a **lower bound** for the total addressable MEV on MEV-Share. With more complex strategies and more exchanges supported, total profits which could be realized on MEV-Share should far exceed those which are estimated by this system.\n\nThis system implements a decidedly simple strategy to estimate a baseline amount of MEV exposed by a few well-known exchanges in the context of MEV-Share. It does not account for many factors that would affect the profitability of an arb, such as gas prices or placement in the block. This system also ignores multiple-hop arbitrage paths, which would improve profits considerably. It also ignores Balancer and Curve trades, which are supported by MEV-Share.\n\nThe system currently only supports Uniswap V2/V3 and SushiSwap. More exchanges may be added in the future, which should improve profitability.\n\nThe system currently only supports WETH as the input token, so that the arbitrage is always WETH -\u003e TOKEN -\u003e WETH.\n\nThe system (the `scan` command specifically) is set up to retry indefinitely when the main loop crashes. This is because every once in a while, the system encounters a critical error, usually related to a bad API response. This is not ideal, but a retry usually fixes it. However, this means that your instance might spam your node with requests if it encounters an unrecoverable error. If you're running on a hosted node, this could waste your rate limit. Make sure to check on it while it's running. 👁️\n\n## setup\n\nMake sure to clone the repo with `--recurse-submodules`. At least for now, we depend on a specific commit of rusty-sando, for its very-useful ForkDB.\n\n```bash\ngit clone --recurse-submodules https://github.com/flashbots/hindsight\n```\n\nor if you already cloned without recursing submodules:\n\n```bash\n# in hindsight/\ngit submodule update --init\n```\n\n### 🚧 DB implementation incomplete 🚧\n\nThe system defaults to using mongo as the database to store arb simulation results. Postgres can be used (add `--help` to any command for details) but currently it only stores `tx_hash`, `event_block`, `event_timestamp`, and `profit`, whereas mongo stores all event and arbitrage trade data. Postgres functionality may be improved later on.\n\n### requirements\n\n- ethereum archive node supporting [`trace_callMany`](https://openethereum.github.io/JSONRPC-trace-module#trace_callmany) API (Reth or Erigon or Infura)\n  - [Erigon](https://github.com/ledgerwatch/erigon) and [Reth](https://github.com/paradigmxyz/reth) are good self-hosted options.\n  - [Infura](https://www.infura.io/solutions/archive-access) and [QuickNode](https://www.quicknode.com/core-api) offer hosted solutions (make sure you get an \"archive node\" plan if prompted for it).\n\n  \u003e The default environment (specified in [`.env.example`](.env.example)) assumes that you have an Ethereum node accessible on `ws://localhost:8545`.\n\n### To build and run locally\n\n_Either/Or:_\n\n- [rust](https://www.rust-lang.org/learn/get-started) (tested with rustc 1.70.0)\n- [docker](https://www.docker.com/get-started/) (tested with v24.0.3)\n\n### populate environment variables\n\nIf you want to set your environment variables in a file, copy the template file `.env.example` to `.env` and update as needed.\n\n```sh\ncp .env.example .env\n# modify in your preferred editor\nvim .env\n```\n\nThe values present in `.env.example` will work if you run hindsight locally, but if you're using docker, you'll have to change the values to reflect the host in the context of the container.\n\nWith the DB and Ethereum RPC accessible on the host machine:\n\n*Docker .env config:*\n\n```txt\nRPC_URL_WS=ws://host.docker.internal:8545\nMONGO_URL=mongodb://root:example@host.docker.internal:27017\nPOSTGRES_URL=postgres://postgres:adminPassword@host.docker.internal:5432\n\n```\n\nSome docker installations on linux don't support `host.docker.internal`; you may try this instead:\n\n```txt\nRPC_URL_WS=ws://172.17.0.1:8545\nMONGO_URL=mongodb://root:example@172.17.0.1:27017\nPOSTGRES_URL=postgres://postgres:adminPassword@172.17.0.1:5432\n```\n\n#### .env vs environment variables\n\n`.env` is optional. If you prefer, you can set environment variables directly in your shell:\n\n```sh\nexport RPC_URL_WS=ws://127.0.0.1:8545\nexport MONGO_URL=mongodb://root:example@localhost:27017\nexport POSTGRES_URL=postgres://postgres:adminPassword@localhost:5432\ncargo run -- scan\n\n# alternatively, to pass the variables directly to hindsight rather than setting them in the shell\nRPC_URL_WS=ws://127.0.0.1:8545 \\\nMONGO_URL=mongodb://root:example@localhost:27017 \\\nPOSTGRES_URL=postgres://postgres:adminPassword@localhost:5432 \\\ncargo run -- scan\n```\n\n### system dependencies\n\n```sh\n# Debian/Ubuntu\nsudo apt install build-essential libssl-dev pkg-config\n```\n\n### TLS for AWS DocumentDB (optional; only used for cloud DBs)\n\nGet the CA file:\n\n```sh\n./get-ca.sh\n```\n\nEnable TLS for the db by modifying your `.env` file:\n\n- uncomment and set `TLS_CA_FILE_MONGO`\n- add `?tls=true` to your existing `MONGO_URL`\n\n`.env`\n\n```txt\nTLS_CA_FILE_MONGO=global-bundle.pem\nMONGO_URL=mongodb://root:example@localhost:27017/?tls=true\n```\n\n### run DB locally w/ docker\n\n```sh\ndocker compose up -d\n```\n\nIf you like, you can browse the database in your web browser here: [http://localhost:8081/](http://localhost:8081). Note that there won't be any interesting data in it until you run the [`scan`](#scan) command.\n\n### build and run\n\n**Locally:**\n\n```sh\n# compile it\ncargo build\n\n# run with cargo\ncargo run -- --help\n\n# or run the binary directly\n./target/debug/hindsight --help\n```\n\n**With Docker:**\n\n```sh\ndocker build -t hindsight .\ndocker run -it -e RPC_URL_WS=ws://host.docker.internal:8545 -e MONGO_URL=mongodb://host.docker.internal:27017 hindsight --help\n```\n\n\u003e :information_source: From this point on, I'll use `hindsight` to refer to whichever method you choose to run the program. So `hindsight scan --help` would translate to `cargo run -- scan --help` or `docker run -it hindsight --help` or `./target/debug/hindsight --help`.\n\n### (optional) test\n\nAll the tests are integration tests, so you'll have to have your environment (DB \u0026 ETH provider) set up to run them successfully.\n\n```sh\nexport RPC_URL_WS=ws://127.0.0.1:8545\nexport MONGO_URL=mongodb://localhost:27017\ncargo test\n```\n\n## `scan`\n\nThe `scan` command is the heart of Hindsight. It scans events from the MEV-Share Event History API, then fetches the full transactions of those events from the blockchain to use in simulations. The system then forks the blockchain at the block in which each transaction landed, and runs an [arbitrarily](./src/sim/core.rs#L28)-[juiced quadratic search](https://research.ijcaonline.org/volume65/number14/pxc3886165.pdf) to find the optimal amount of WETH to execute a backrun-arbitrage. The results are then saved to the database.\n\nTo scan the last week's events for arbs:\n\n```sh\nhindsight scan -t $(echo \"$(date +%s) - (86400 * 7)\" | bc)\n\n# or if you don't have `bc` and can accept these ugly parenthesis\nhindsight scan -t $(echo $(($(date +%s) - ((86400 * 7)))))\n```\n\nThe timestamp arguments accept unix-style integer timestamps, represented in seconds.\n\n## `export`\n\nThe `export` command is a simple way to filter and export results from the database into a JSON file.\n\nTo export arbs for events from the last week:\n\n```sh\nhindsight export -t $(echo \"$(date +%s) - (86400 * 7)\" | bc)\n\n# or\nhindsight export -t $(echo $(($(date +%s) - ((86400 * 7)))))\n```\n\nTo filter out unprofitable results:\n\n```sh\n# only export arbs that returned a profit of at least 0.0001 WETH\nhindsight export -p 0.0001\n```\n\n### exporting with docker\n\nHindsight exports all files into a directory `./arbData`, relative to wherever the program is executed. To get these files out of the docker container and on to your host machine, you'll need to map the volume to a local directory.\n\nIn the directory where you want to put the files (we make an `arbData` directory but you don't have to):\n\n```sh\nmkdir -p arbData\ndocker run -it -v $(pwd)/arbData:/app/arbData -e RPC_URL_WS=ws://host.docker.internal:8545 -e MONGO_URL=mongodb://host.docker.internal:27017 hindsight export -p 0.0001\n```\n\n## common errors\n\n### error: \"too many open files\"\n\nThe `scan` command can spawn a lot of threads. You may need to increase the open file limit on your system to ensure reliable operation.\n\n```sh\n# check current open file limit\nulimit -Sn\n\n# raise the limit if needed\nulimit -n 4000\n\n# be careful, this can cause system-wide issues if you set it too high\nulimit -n 9001\n```\n\nAlternatively, you can run the `scan` command with less parallel operations:\n\n```sh\n# only process two txs at a time\nhindsight scan -n 2\n```\n\n### Error: Kind: Server selection timeout: No available servers\n\n... `Topology: { Type: Unknown, Servers: [ { Address: host.docker.internal:27017, Type: Unknown, Error: Kind: I/O error: failed to lookup address information: Name or service not known, labels: {} } ] }, labels: {}`\n\nThis means that your system doesn't support the `host.docker.internal` mapping. Try replacing `host.docker.internal` with `172.17.0.1`.\n\n### Error: IO error: Connection refused (os error 111)\n\nThis means that either your ETH node or your DB is not properly connected.\n\nMake sure your DB is running with `docker ps` (you should see `mongo` and `mongo-express` running). If they're not running, try this:\n\n```sh\ndocker compose down\ndocker compose up\n```\n\nIf your DB is running, make sure your node is running and accessible from your host. A [simple JSON-RPC request with curl](https://ethereum.org/en/developers/docs/apis/json-rpc/#net_version) is the simplest way to test this.\n\n```sh\ncurl -X POST --data '{\"jsonrpc\":\"2.0\",\"method\":\"net_version\",\"params\":[],\"id\":42}'\n```\n\nIf that doesn't work, try double-checking your URLs. Refer back to the [environment instructions](#populate-environment-variables) if you're lost.\n\n## acknowledgements\n\n- [rusty-sando](https://github.com/mouseless-eth/rusty-sando)\n- [mev-inspect-rs](https://github.com/flashbots/mev-inspect-rs)\n- [mev-inspect-py](https://github.com/flashbots/mev-inspect-py)\n- [nitepunk](https://soundcloud.com/nitepunk/sets/slices)\n\n## future improvements\n\nSee [issues](https://github.com/flashbots/hindsight/issues) for the most up-to-date status, or to propose an improvement!\n\n- [ ] support all the fields in postgres, then make postgres the default\n- [ ] replace [ForkDB dependency](./src/sim/core.rs#L22-L23) (possibly with [Arbiter](https://github.com/primitivefinance/arbiter))\n- [ ] add more protocols (currently only support UniV2, UniV3, and Sushiswap)\n- [ ] maybe: add more complex strategies\n  - multi-hop arbs\n  - multi-tx backruns (using mempool txs)\n  - stat arb\n  - so many more...\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflashbots%2Fhindsight","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflashbots%2Fhindsight","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflashbots%2Fhindsight/lists"}