{"id":25459759,"url":"https://github.com/paritytech/evm-test-suite","last_synced_at":"2025-11-03T05:30:21.865Z","repository":{"id":260644330,"uuid":"881547283","full_name":"paritytech/evm-test-suite","owner":"paritytech","description":null,"archived":false,"fork":false,"pushed_at":"2025-02-14T02:58:22.000Z","size":259676,"stargazers_count":2,"open_issues_count":12,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-14T03:30:36.134Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Solidity","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/paritytech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-31T19:36:06.000Z","updated_at":"2025-02-10T17:10:59.000Z","dependencies_parsed_at":"2024-11-01T15:30:49.979Z","dependency_job_id":"36c9c640-565a-4d6a-bd3c-498b5e2c550e","html_url":"https://github.com/paritytech/evm-test-suite","commit_stats":null,"previous_names":["bee344/evm-tests","paritytech/evm-test-suite"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paritytech%2Fevm-test-suite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paritytech%2Fevm-test-suite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paritytech%2Fevm-test-suite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paritytech%2Fevm-test-suite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/paritytech","download_url":"https://codeload.github.com/paritytech/evm-test-suite/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239412485,"owners_count":19634016,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-18T04:35:43.908Z","updated_at":"2025-11-03T05:30:21.834Z","avatar_url":"https://github.com/paritytech.png","language":"Solidity","funding_links":[],"categories":[],"sub_categories":[],"readme":"### EVM Tests [WIP]\n\nSet of generic tests to check for chain compatibility. It covers higher level \ntests derived from well-known smart contracts (such as uniswap v4, WIP), standard testing (such as [Open Zeppelin tests](https://github.com/OpenZeppelin/openzeppelin-contracts/tree/master/test)) and lower level testing inspired by [Matter Labs](https://github.com/matter-labs).\n\n## Execution\n\nThis is a hybrid test suite that uses both [Hardhat](https://hardhat.org/) (and in the future\n[Foundry](https://book.getfoundry.sh/)).\n\nTo run the tests, simply run the script `init.sh`:\n\n```sh\n~ ./init.sh \u003cCHAIN\u003e \u003cURL\u003e \u003cTEST\u003e \u003cFORKING\u003e \u003cNODE_PATH\u003e \u003cADADPTER_PATH\u003e \u003cCOMPILER_PATH\u003e \u003cTEST_FILTER\u003e \u003cVERBOSE_LOGGING\u003e\n```\n\n* CHAIN\n* URL\n* TEST\n* FORKING\n* NODE_PATH\n* ADAPTER_PATH\n* COMPILER_PATH\n* TEST_FILTER\n* VERBOSE_LOGGING\n\n\n| 🔧 Parameter            | 📄 Description                                              |\n|-------------------------|-------------------------------------------------------------|\n| CHAIN                   | Chain preset that will be used.                             |\n| URL                     | URL to connect to when forking.                             |\n| TEST                    | Set of tests to run.                                        |\n| FORKING                 | Boolean that defines whether to work with a forked chain.   |\n| NODE_PATH               | Path to the `substrate-node` binary.                        |\n| ADAPTER_PATH            | Path to the `eth-rpc` binary.                               |\n| COMPILER_PATH           | Path to the `resolc` binary.                                |\n| TEST_FILTER             | Value used to filter the contracts to be run.               |\n| VERBOSE_LOGGING         | Boolean that defines whether to verbosely log tests.        |\n\nCurrently the available options for `CHAIN` to test against are:\n* `--acala`\n* `--arbitrum`\n* `--astar`\n* `--kitchensink`\n* `--ethereum`\n* `--moonbeam`\n* `--polygon`\n* `--westend`\n* `--endpoint` or `-e`\n\nWith `--ethereum` being the default option.\n\nWhen passing `--endpoint` or `-e`, the second argument must be the `http` endpoint\nof the node you are connecting to. You must also set the\n`USE_REVIVE` env var to either `true` or `false`, in order to enable compilation\nto `PolkaVM` rather than `EVM`. If `USE_REVIVE` is set to `false`, it will use the\nconfiguration inside of the project's `hardhat.config.ts`.\n\nIf choosing from a preset chain and specifying\na test, `--` should be passed for the `\u003cURL\u003e` argument, and the `USE_REVIVE` env\nvar is set automatically depending on the chain.\n\nAs for `TEST`, you can specify either `--matter-labs` to run the `matter-labs`\ntests or `--open-zeppelin` to run the openzeppelin standard tests. Not\nspecifying the test will result in both suites being executed consecutively.\n\nWhen running the Kitchensink node, you must specify the `NODE_PATH`, `ADAPTER_PATH` and\nthe `COMPILER_PATH`. Setting `FORKING=true` and running a local node are not compatible.\n\nThis script will install the necessary packages and run the tests in order.\n\nThe test logs will be saved to `/test-logs/`, in order to allow the user to review\nthem after they are completed, since terminals may have a limited historical display.\n\nIn order for the commands to be parsed correctly, if you have parameters you are not using\nbefore one you are using, you must set them as `--`, in order to keep the spacing correct.\n\nWhen handling tests with a considerable amount of contracts to compile while\nin resource-restricted environments such as Docker, it is recommended to define\nthe environment variable `BATCH_SIZE` in order to define the amount of contracts\nthat are compiled at once. This tells the plugin to split the contracts in subsets\nof `length=BATCH_SIZE` and compile the batches sequentially.\n\n**Example**\n\nFor example,\n\n```bash\n~ ./init.sh --kitchensink -- --matter-labs -- \\\n../../../polkadot-sdk/target/release/substrate-node \\\n../../../polkadot-sdk/target/release/eth-rpc \\\n../../revive/target/release/resolc\n```\n\nwill run the Matter Labs test suite for a local Kitchensink node and store the output in the \"output-logs\" folder.\n\nWhen running tests for `PolkaVM`, we recommend using a local Kitchensink node instead\nof forking the live chain, since doing so may result in errors like\n`CodeRejected` or `Metadata error: The generated code is not compatible with the node`\nif either the local `resolc` or `eth-rpc` versions don't match the `PolkaVM` version\nin the chain.\n\nTo make sure the versions match, we recommend building the compiler first and then\ncheck inside pallet revive's [`Cargo.toml`](https://github.com/paritytech/revive/blob/fe1b3258d2956e51e2edd86f2e77898e6b142729/Cargo.toml#L76)\nin order to see which commit of the polkadot-sdk you should use to build the\n`substrate-node` and `eth-rpc` binaries.\n\n## Matter Labs Test Options\n\n### Filtering Contracts\n\nWhen running matter labs tests with the `--matter-labs` option you can set the `TEST_FILTER` option in order to compile and run only a subset of the contracts.\n\nExample:\n```bash\n./init.sh --ethereum -- --matter-labs -- -- -- -- /destructuring \n```\nThis will match for any matter-labs contract that includes the string `/destructuring` and only compile and run the given tests of those contracts. \n\n### Log Verbosity\n\nWhen running matter labs tests with the `--matter-labs` option you can set the `VERBOSE_LOGGING` option to true in order to log additional test output which includes the method of the contract being called, its expected output and the actual output which was received.\n\nExample:\n```bash\n./init.sh --ethereum -- --matter-labs -- -- -- -- -- true \n```\n\nNote: These options are currently only available to use with Matter Labs tests with plans to expand them to additional tests.\n\n## Open Zeppelin (OZ) Test Output\n\nOpen Zeppelin Test output consists of passing and failing tests as defined in the open-zeppelin suite of tests.\n\n## OZ Passing Tests\n\nAn example of `passing` open-zeppelin test output looks like the following:\n\n```bash\naddress\n      ✔ starts empty\n      add\n        ✔ adds a value\n        ✔ adds several values\n        ✔ returns false when adding values already in the set\n      at\n        ✔ reverts when retrieving non-existent elements\n        ✔ retrieves existing element\n      remove\n        ✔ removes added values\n        ✔ returns false when removing values not in the set\n        ✔ adds and removes multiple values\n```\nThis log indicates the description of the group of tests being run (e.g. `address`), the description for different tests being run that belong to the described grouping of tests (e.g. `add`, `at` and `remove`) and the individual test cases of those tests (e.g. `adds a value`, `adds and removes multiple values` etc). Each case with a check represents a passing test.\n\n## OZ Failing Tests\n\nAn example of `failing` open-zeppelin test output: \n```bash\n1) Environment sanity\n       snapshot\n         cache and mine:\n\n      AssertionError: expected 249 to equal 250\n      + expected - actual\n\n      -249\n      +250\n      \n      at Context.\u003canonymous\u003e (test/sanity.test.js:19:57)\n```\nThis log indicates the description of the group of tests (e.g. `Environment sanity`), the description of the different tests being run that belong to that group (e.g. `snapshot`), the individual test cases for those tests (e.g. `cache and mine`) and the assertion error (and corresponding file/line where the error was thrown) which occurred for the test case.\n\n## Matter Labs (ML) Test Output\n\nMatter Labs Test output differs based on whether the test cases are considered passing, failing or  whitelisted.\n\n## ML Passing Tests\n\nPassing test output indicates that the results of deploying a contract or calling a given method in a contract returns the expected value. An example being:\n```bash\n    ✔ Tests for test case complex1 (126ms)\n    Method complex: expected: [\"12\",\"128\",\"255\",\"0\",\"123\",\"12\",\"68\",\"192\",\"2\",\"19\"] - actual: 12,128,255,0,123,12,68,192,2,19\n```\nThis log indicates the test case that was run (e.g. `complex1`), the method that was called (e.g. `complex`) as well as the expected and actual output that was received.\n\n## ML Failing Tests\n\nFailing test output indicates that an expected result was not received. An example of failing test output looks like the following:\n```bash\n  Tests for test case complex1:\n     AssertionError [ERR_ASSERTION]: Failed Test Case complex1 from contracts/era-compiler-tests/solidity/simple/algorithm/cryptography/book_cypher.sol calling method complex with inputs 1,12,55,53,22,34,45,21,12,7 - expected: [\"1\",\"12\",\"55\",\"53\",\"22\",\"34\",\"45\",\"21\",\"12\",\"7\"], actual: ProviderError: Failed to instantiate contract: Module(ModuleError { index: 80, error: [10, 0, 0, 0], message: Some(\"ContractTrapped\") })\n      at Context.\u003canonymous\u003e (test/MatterLabsTests.ts:447:49)\n      at processTicksAndRejections (node:internal/process/task_queues:105:5)\n```\nThis log indicates the test case being run (e.g. `complex1`), the contract the test case belongs to (`contracts/era-compiler-tests/solidity/simple/algorithm/cryptography/book_cypher.sol`), the method that was called (e.g. `complex`) as well as the expected and actual output that was received. The actual value for a failing test can be an error or some other unexpected value received from the contract call.\n\n## ML Whitelisted Tests\n\nThere are some cases where there are tests which have been whitelisted and are currently expected to fail. These are limited and typically represent tests that are either expected to fail due to differences in runtime environments (e.g. geth vs polygon chainIds being different), contain methods not currently supported in the virtual machine or would otherwise break compilation for a given VM causing the test suite to fail (e.g. contracts with too many arguments).\n```bash\n✔ Tests for test case main (5167ms)\nDeployed contracts/era-compiler-tests/solidity/simple/yul_instructions/basefee.sol:Test\nwhitelisted: Test Case default from contracts/era-compiler-tests/solidity/simple/yul_instructions/basefee.sol calling method main with inputs  - expected: [\"7\"], actual: AssertionError: expected +0 to equal 7\n```\nWhitelisted test logs are similar to passing/failing outputs except they do not influence the number of failing tests in the test suite.\n\n## Contribute\n\nNeed help or want to contribute ideas or code? Head over to our [CONTRIBUTING](CONTRIBUTING.md) doc for more information.\n\n**NOTE**\n\nSome tests are expected to fail, and being actively worked on. Specifically, there\nwill be errors such as `the initcode size of this transaction is too large`,\nthis is because the `substrate-node` still isn't fully compatible with the `hardhat`\nnetwork specs, due to lacking some required rpc calls. This is being worked on\nand once we have full compatibility, we can configure the tests to ignore this limit.\nThis size limit is the standard in `EVM` but is not the same as with `PolkaVM`.\n\nThe openzeppelin tests take a while to run against the Kitchensink node, around ~3hs.\nThis is due to the previously mentioned lack of full compatibility with the `hardhat`\nnetwork we cannot take advantage of hardhat-only helpers like `mine()` and `loadFixture()`,\nand have to wait for the block to be produced and redeploy the contract each time\nto replace these.\n\nTo run the tests with the default `--arbitrum` or `--polygon` options, you need\nto set the env variable `PRIVATE_KEY` with your Infura API key. \n\nTests take a while to run, and using the predetermined public endpoints may \nresult in them taking longer timeouts or being banned due to the amount of requests,\nso we highly recommend using the `-e` option.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparitytech%2Fevm-test-suite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fparitytech%2Fevm-test-suite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparitytech%2Fevm-test-suite/lists"}