{"id":13869753,"url":"https://github.com/data-apis/array-api-tests","last_synced_at":"2026-03-04T19:33:58.766Z","repository":{"id":39862325,"uuid":"291148032","full_name":"data-apis/array-api-tests","owner":"data-apis","description":"Test suite for Python array API standard compliance","archived":false,"fork":false,"pushed_at":"2026-02-17T20:02:44.000Z","size":1712,"stargazers_count":70,"open_issues_count":62,"forks_count":47,"subscribers_count":16,"default_branch":"master","last_synced_at":"2026-02-18T01:24:59.239Z","etag":null,"topics":["array-computing","testing-tools"],"latest_commit_sha":null,"homepage":"https://data-apis.org/array-api-tests/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/data-apis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-08-28T21:19:58.000Z","updated_at":"2026-02-17T20:02:49.000Z","dependencies_parsed_at":"2023-12-07T13:28:57.433Z","dependency_job_id":"5db08e40-1734-4115-8a50-1844a0cb20ca","html_url":"https://github.com/data-apis/array-api-tests","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/data-apis/array-api-tests","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/data-apis%2Farray-api-tests","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/data-apis%2Farray-api-tests/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/data-apis%2Farray-api-tests/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/data-apis%2Farray-api-tests/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/data-apis","download_url":"https://codeload.github.com/data-apis/array-api-tests/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/data-apis%2Farray-api-tests/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30090596,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-04T18:31:08.343Z","status":"ssl_error","status_checked_at":"2026-03-04T18:31:07.708Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["array-computing","testing-tools"],"created_at":"2024-08-05T20:01:15.499Z","updated_at":"2026-03-04T19:33:58.742Z","avatar_url":"https://github.com/data-apis.png","language":"Python","readme":"# Test Suite for Array API Compliance\n\nThis is the test suite for array libraries adopting the [Python Array API\nstandard](https://data-apis.org/array-api/latest).\n\nKeeping full coverage of the spec is an on-going priority as the Array API evolves.\nFeedback and contributions are welcome!\n\n## Quickstart\n\n### Setup\n\nCurrently we pin the Array API specification repo [`array-api`](https://github.com/data-apis/array-api/)\nas a git submodule. This might change in the future to better support vendoring\nuse cases (see [#107](https://github.com/data-apis/array-api-tests/issues/107)),\nbut for now be sure submodules are pulled too, e.g.\n\n```bash\n$ git submodule update --init\n```\n\nTo run the tests, install the testing dependencies.\n\n```bash\n$ pip install -r requirements.txt\n```\n\nEnsure you have the array library that you want to test installed.\n\n### Specifying the array module\n\nYou need to specify the array library to test. It can be specified via the\n`ARRAY_API_TESTS_MODULE` environment variable, e.g.\n\n```bash\n$ export ARRAY_API_TESTS_MODULE=array_api_strict\n```\n\nTo specify a runtime-defined module, define `xp` using the `exec('...')` syntax:\n\n```bash\n$ export ARRAY_API_TESTS_MODULE=\"exec('import quantity_array, numpy; xp = quantity_array.quantity_namespace(numpy)')\"\n```\n\nAlternately, import/define the `xp` and `xp_name` variables in `array_api_tests/__init__.py`.\n\n### Specifying the API version\n\nYou can specify the API version to use when testing via the\n`ARRAY_API_TESTS_VERSION` environment variable, e.g.\n\n```bash\n$ export ARRAY_API_TESTS_VERSION=\"2023.12\"\n```\n\nCurrently this defaults to the array module's `__array_api_version__` value, and\nif that attribute doesn't exist then we fallback to `\"2021.12\"`.\n\n### Run the suite\n\nSimply run `pytest` against the `array_api_tests/` folder to run the full suite.\n\n```bash\n$ pytest array_api_tests/\n```\n\nThe suite tries to logically organise its tests. `pytest` allows you to only run\na specific test case, which is useful when developing functions.\n\n```bash\n$ pytest array_api_tests/test_creation_functions.py::test_zeros\n```\n\n## What the test suite covers\n\nWe are interested in array libraries conforming to the\n[spec](https://data-apis.org/array-api/latest/API_specification/index.html).\nIdeally this means that if a library has fully adopted the Array API, the test\nsuite passes. We take great care to _not_ test things which are out-of-scope,\nso as to not unexpectedly fail the suite.\n\n### Primary tests\n\nEvery function—including array object methods—has a respective test\nmethod\u003csup\u003e1\u003c/sup\u003e. We use\n[Hypothesis](https://hypothesis.readthedocs.io/en/latest/)\nto generate a diverse set of valid inputs. This means array inputs will cover\ndifferent dtypes and shapes, as well as contain interesting elements. These\nexamples generate with interesting arrangements of non-array positional\narguments and keyword arguments.\n\nEach test case will cover the following areas if relevant:\n\n* **Smoking**: We pass our generated examples to all functions. As these\n  examples solely consist of *valid* inputs, we are testing that functions can\n  be called using their documented inputs without raising errors.\n\n* **Data type**: For functions returning/modifying arrays, we assert that output\n  arrays have the correct data types. Most functions\n  [type-promote](https://data-apis.org/array-api/latest/API_specification/type_promotion.html)\n  input arrays and some functions have bespoke rules—in both cases we simulate\n  the correct behaviour to find the expected data types.\n\n* **Shape**: For functions returning/modifying arrays, we assert that output\n  arrays have the correct shape. Most functions\n  [broadcast](https://data-apis.org/array-api/latest/API_specification/broadcasting.html)\n  input arrays and some functions have bespoke rules—in both cases we simulate\n  the correct behaviour to find the expected shapes.\n\n* **Values**: We assert output values (including the elements of\n  returned/modified arrays) are as expected. Except for manipulation functions\n  or special cases, the spec allows floating-point inputs to have inexact\n  outputs, so with such examples we only assert values are roughly as expected.\n\n### Additional tests\n\nIn addition to having one test case for each function, we test other properties\nof the functions and some miscellaneous things.\n\n* **Special cases**: For functions with special case behaviour, we assert that\n  these functions return the correct values.\n\n* **Signatures**: We assert functions have the correct signatures.\n\n* **Constants**: We assert that\n  [constants](https://data-apis.org/array-api/latest/API_specification/constants.html)\n  behave expectedly, are roughly the expected value, and that any related\n  functions interact with them correctly.\n\nBe aware that some aspects of the spec are impractical or impossible to actually\ntest, so they are not covered in the suite. \u003c!-- TODO: note what these are --\u003e\n\n## Interpreting errors\n\nFirst and foremost, note that most tests have to assume that certain aspects of\nthe Array API have been correctly adopted, as fundamental APIs such as array\ncreation and equalities are hard requirements for many assertions. This means a\ntest case for one function might fail because another function has bugs or even\nno implementation.\n\nThis means adopting libraries at first will result in a vast number of errors\ndue to cascading errors. Generally the nature of the spec means many granular\ndetails such as type promotion is likely going to also fail nearly-conforming\nfunctions.\n\nWe hope to improve user experience in regards to \"noisy\" errors in\n[#51](https://github.com/data-apis/array-api-tests/issues/51). For now, if an\nerror message involves `_UndefinedStub`, it means an attribute of the array\nlibrary (including functions) and it's objects (e.g. the array) is missing.\n\nThe spec is the suite's source of truth. If the suite appears to assume\nbehaviour different from the spec, or test something that is not documented,\nthis is a bug—please [report such\nissues](https://github.com/data-apis/array-api-tests/issues/) to us.\n\n\n## Running on CI\n\nSee our existing [GitHub Actions workflow for `array-api-strict`](https://github.com/data-apis/array-api-tests/blob/master/.github/workflows/test.yml)\nfor an example of using the test suite on CI. Note [`array-api-strict`](https://github.com/data-apis/array-api-strict)\nis an implementation of the array API that uses NumPy under the hood.\n\n### Releases\n\nWe recommend pinning against a [release tag](https://github.com/data-apis/array-api-tests/releases)\nwhen running on CI.\n\nWe use [calender versioning](https://calver.org/) for the releases. You should\nexpect that any version may be \"breaking\" compared to the previous one, in that\nnew tests (or improvements to existing tests) may cause a previously passing\nlibrary to fail.\n\n### Configuration\n\n#### Data-dependent shapes\n\nUse the `--disable-data-dependent-shapes` flag to skip testing functions which have\n[data-dependent shapes](https://data-apis.org/array-api/latest/design_topics/data_dependent_output_shapes.html).\n\n#### Extensions\n\nBy default, tests for the optional Array API extensions such as\n[`linalg`](https://data-apis.org/array-api/latest/extensions/linear_algebra_functions.html)\nwill be skipped if not present in the specified array module. You can purposely\nskip testing extension(s) via the `--disable-extension` option.\n\n#### Skip or XFAIL test cases\n\nTest cases you want to skip can be specified in a skips or XFAILS file. The\ndifference between skip and XFAIL is that XFAIL tests are still run and\nreported as XPASS if they pass.\n\nBy default, the skips and xfails files are `skips.txt` and `fails.txt` in the root\nof this repository, but any file can be specified with the `--skips-file` and\n`--xfails-file` command line flags.\n\nThe files should list the test ids to be skipped/xfailed. Empty lines and\nlines starting with `#` are ignored. The test id can be any substring of the\ntest ids to skip/xfail.\n\n```\n# skips.txt or xfails.txt\n# Line comments can be denoted with the hash symbol (#)\n\n# Skip specific test case, e.g. when argsort() does not respect relative order\n# https://github.com/numpy/numpy/issues/20778\narray_api_tests/test_sorting_functions.py::test_argsort\n\n# Skip specific test case parameter, e.g. you forgot to implement in-place adds\narray_api_tests/test_add[__iadd__(x1, x2)]\narray_api_tests/test_add[__iadd__(x, s)]\n\n# Skip module, e.g. when your set functions treat NaNs as non-distinct\n# https://github.com/numpy/numpy/issues/20326\narray_api_tests/test_set_functions.py\n```\n\nHere is an example GitHub Actions workflow file, where the xfails are stored\nin `array-api-tests.xfails.txt` in the base of the `your-array-library` repo.\n\nIf you want, you can use `-o xfail_strict=True`, which causes XPASS tests (XFAIL\ntests that actually pass) to fail the test suite. However, be aware that\nXFAILures can be flaky (see below, so this may not be a good idea unless you\nuse some other mitigation of such flakyness).\n\nIf you don't want this behavior, you can remove it, or use `--skips-file`\ninstead of `--xfails-file`.\n\n```yaml\n# ./.github/workflows/array_api.yml\njobs:\n  tests:\n    runs-on: ubuntu-latest\n    strategy:\n      matrix:\n        python-version: ['3.8', '3.9', '3.10', '3.11']\n\n    steps:\n    - name: Checkout \u003cyour array library\u003e\n      uses: actions/checkout@v3\n      with:\n        path: your-array-library\n\n    - name: Checkout array-api-tests\n      uses: actions/checkout@v3\n      with:\n        repository: data-apis/array-api-tests\n        submodules: 'true'\n        path: array-api-tests\n\n    - name: Run the array API test suite\n      env:\n        ARRAY_API_TESTS_MODULE: your.array.api.namespace\n      run: |\n        export PYTHONPATH=\"${GITHUB_WORKSPACE}/your-array-library\"\n        cd ${GITHUB_WORKSPACE}/array-api-tests\n        pytest -v -rxXfE --ci --xfails-file ${GITHUB_WORKSPACE}/your-array-library/array-api-tests-xfails.txt array_api_tests/\n```\n\n\u003e **Warning**\n\u003e\n\u003e XFAIL tests that use Hypothesis (basically every test in the test suite except\n\u003e those in test_has_names.py) can be flaky, due to the fact that Hypothesis\n\u003e might not always run the test with an input that causes the test to fail.\n\u003e There are several ways to avoid this problem:\n\u003e\n\u003e - Increase the maximum number of examples, e.g., by adding `--max-examples\n\u003e   200` to the test command (the default is `20`, see below). This will\n\u003e   make it more likely that the failing case will be found, but it will also\n\u003e   make the tests take longer to run.\n\u003e - Don't use `-o xfail_strict=True`. This will make it so that if an XFAIL\n\u003e   test passes, it will alert you in the test summary but will not cause the\n\u003e   test run to register as failed.\n\u003e - Use skips instead of XFAILS. The difference between XFAIL and skip is that\n\u003e   a skipped test is never run at all, whereas an XFAIL test is always run\n\u003e   but ignored if it fails.\n\u003e - Save the [Hypothesis examples\n\u003e   database](https://hypothesis.readthedocs.io/en/latest/database.html)\n\u003e   persistently on CI. That way as soon as a run finds one failing example,\n\u003e   it will always re-run future runs with that example. But note that the\n\u003e   Hypothesis examples database may be cleared when a new version of\n\u003e   Hypothesis or the test suite is released.\n\n#### Max examples\n\nThe tests make heavy use\n[Hypothesis](https://hypothesis.readthedocs.io/en/latest/). You can configure\nhow many examples are generated using the `--max-examples` flag, which\ndefaults to `20`. Lower values can be useful for quick checks, and larger\nvalues should result in more rigorous runs. For example, `--max-examples\n10_000` may find bugs where default runs don't but will take much longer to\nrun.\n\n#### Skipping Dtypes\n\nThe test suite will automatically skip testing of inessential dtypes if they\nare not present on the array module namespace, but dtypes can also be skipped\nmanually by setting the environment variable `ARRAY_API_TESTS_SKIP_DTYPES` to\na comma separated list of dtypes to skip. For example\n\n```\nARRAY_API_TESTS_SKIP_DTYPES=uint16,uint32,uint64 pytest array_api_tests/\n```\n\nNote that skipping certain essential dtypes such as `bool` and the default\nfloating-point dtype is not supported.\n\n#### Turning xfails into skips\n\nKeeping a large number of ``xfails`` can have drastic effects on the run time. This is due\nto the way `hypothesis` works: when it detects a failure, it does a large amount\nof work to simplify the failing example.\nIf the run time of the test suite becomes a problem, you can use the\n``ARRAY_API_TESTS_XFAIL_MARK`` environment variable: setting it to ``skip`` skips the\nentries from the ``xfail.txt`` file instead of xfailing them. Anecdotally, we saw\nspeed-ups by a factor of 4-5---which allowed us to use 4-5 larger values of\n``--max-examples`` within the same time budget.\n\n#### Limiting the array sizes\n\nThe test suite generates random arrays as inputs to functions it tests. \"unvectorized\"\ntests iterate over elements of arrays, which might be slow. If the run time becomes\na problem, you can limit the maximum number of elements in generated arrays by\nsetting the environment variable ``ARRAY_API_TESTS_MAX_ARRAY_SIZE`` to the\ndesired value. By default, it is set to 1024.\n\n\n## Contributing\n\n### Remain in-scope\n\nIt is important that every test only uses APIs that are part of the standard.\nFor instance, when creating input arrays you should only use the [array creation\nfunctions](https://data-apis.org/array-api/latest/API_specification/creation_functions.html)\nthat are documented in the spec. The same goes for testing arrays—you'll find\nmany utilities that parralel NumPy's own test utils in the `*_helpers.py` files.\n\n### Tools\n\nHypothesis should almost always be used for the primary tests, and can be useful\nelsewhere. Effort should be made so drawn arguments are labeled with their\nrespective names. For\n[`st.data()`](https://hypothesis.readthedocs.io/en/latest/data.html#hypothesis.strategies.data),\ndraws should be accompanied with the `label` kwarg i.e. `data.draw(\u003cstrategy\u003e,\nlabel=\u003clabel\u003e)`.\n\n[`pytest.mark.parametrize`](https://docs.pytest.org/en/latest/how-to/parametrize.html)\nshould be used to run tests over multiple arguments. Parameterization should be\npreferred over using Hypothesis when there are a small number of possible\ninputs, as this allows better failure reporting. Note using both parametrize and\nHypothesis for a single test method is possible and can be quite useful.\n\n### Error messages\n\nAny assertion should be accompanied with a descriptive error message, including\nthe relevant values. Error messages should be self-explanatory as to why a given\ntest fails, as one should not need prior knowledge of how the test is\nimplemented.\n\n### Generated files\n\nSome files in the suite are automatically generated from the spec, and should\nnot be edited directly. To regenerate these files, run the script\n\n    ./generate_stubs.py path/to/array-api\n\nwhere `path/to/array-api` is the path to a local clone of the [`array-api`\nrepo](https://github.com/data-apis/array-api/). Edit `generate_stubs.py` to make\nchanges to the generated files.\n\n\n### Release\n\nTo make a release, first make an annotated tag with the version, e.g.:\n\n```\ngit tag -a 2022.01.01\n```\n\nBe sure to use the calver version number for the tag name. Don't worry too much\non the tag message, e.g. just write \"2022.01.01\".\n\nVersioneer will automatically set the version number of the `array_api_tests`\npackage based on the git tag. Push the tag to GitHub:\n\n```\ngit push --tags upstream 2022.1\n```\n\nThen go to the [tags page on\nGitHub](https://github.com/data-apis/array-api-tests/tags) and convert the tag\ninto a release. If you want, you can add release notes, which GitHub can\ngenerate for you.\n\n\n---\n\n\u003csup\u003e1\u003c/sup\u003eThe only exceptions to having just one primary test per function are:\n\n* [`asarray()`](https://data-apis.org/array-api/latest/API_specification/generated/signatures.creation_functions.asarray.html),\n  which is tested by `test_asarray_scalars` and `test_asarray_arrays` in\n  `test_creation_functions.py`. Testing `asarray()` works with scalars (and\n  nested sequences of scalars) is fundamental to testing that it works with\n  arrays, as said arrays can only be generated by passing scalar sequences to\n  `asarray()`.\n\n* Indexing methods\n  ([`__getitem__()`](https://data-apis.org/array-api/latest/API_specification/generated/signatures.array_object.array.__getitem__.html)\n  and\n  [`__setitem__()`](https://data-apis.org/array-api/latest/API_specification/generated/signatures.array_object.array.__setitem__.html)),\n  which respectively have both a test for non-array indices and a test for\n  boolean array indices. This is because [masking is\n  opt-in](https://data-apis.org/array-api/latest/API_specification/indexing.html#boolean-array-indexing)\n  (and boolean arrays need to be generated by indexing arrays anyway).\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdata-apis%2Farray-api-tests","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdata-apis%2Farray-api-tests","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdata-apis%2Farray-api-tests/lists"}