Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/brson/ctrs

A Rust language conformance test suite
https://github.com/brson/ctrs

Last synced: about 1 year ago
JSON representation

A Rust language conformance test suite

Awesome Lists containing this project

README 

          
[![Build Status](https://travis-ci.org/brson/ctrs.svg?branch=master)](https://travis-ci.org/brson/ctrs)



# Conformance Tests for Rust (CTRS)



This is an implementation-independent test suite for The Rust

Programming Language. It is intended as a regression test for

the reference implementation, as well as a tool for testing

conformance of alternate implementations.



It differs from the RI test suite in several ways:



* It only covers stable features.

* It has a simple implementation-independent test runner.

* It draws test cases from a wide range of sources.

* It categorizes test cases by language revision, so that multiple

  levels of conformance may be established.



CTRS is not a curated test of the specification - this probably isn't

code you want to read for fun. Rather, it is simply a massive corpus

of valid Rust code that is known to work against the reference

compiler. The test suite will evolve over time as bugs are shaken

out of the RI.



Execute with `./run.py` to test your Rust compiler.

Set the `RUSTC` environment variable if you must.



## Criteria for inclusion



Criteria will expand over time.

Currently:



* Must pass 'rustc -F deprecated -F unstable_features'

* Must pass 'rustc -F deprecated -F unstable_features --test'

* No 'extern crate' (i.e. only std)

* Single-file test cases only



Run `./stab.py` on a Rust file to see if it is a candidate.



## Organization



All tests are under the `test` directory. Immediately under `test` are

directories for Rust language/compiler versions. Under each version

are directories for test 'groups'. Test groups are drawn from many

sources, and are licensed individually.



Note that even tests categorized under a later language revision may

work under previous revisions. When tests are added for a new language

version they are not first checked against previous versions.



## Current coverage



Current focus is pulling tests out of the reference implementation that

meet the criteria, as well as sourcing single-file examples from

other places.



- 1.4.0

  - doc - Extracts from the RI 'doc' directory

  - doc-collections - Extracts from the RI collections API docs

  - doc-core - Extracts from the RI core library API docs

  - doc-std - Extracts from the RI std library API docs

  - run-pass - Tests from the RI test suite's 'run-pass' directory

- 1.3.0

  - doc - Extracts from the RI 'doc' directory

  - doc-collections - Extracts from the RI collections API docs

  - doc-core - Extracts from the RI core library API docs

  - doc-std - Extracts from the RI std library API docs

  - too-many-lists - Extracts from *[Learn Rust by writing Entirely Too Many Linked Lists][lists]*

  - run-pass - Tests from the RI test suite's 'run-pass' directory

  - rust-by-example - Extracts from [www.rustbyexample.com]

- 1.2.0

  - doc - Extracts from the RI 'doc' directory

  - doc-collections - Extracts from the RI collections API docs

  - doc-core - Extracts from the RI core library API docs

  - doc-std - Extracts from the RI std library API docs

  - pretty - Tests from the RI test suite's 'pretty' directory

  - run-pass - Tests from the RI test suite's 'run-pass' directory

  - rust-by-example - Extracts from [www.rustbyexample.com]

- 1.1.0

  - doc - Extracts from the RI 'doc' directory

  - doc-collections - Extracts from the RI collections API docs

  - doc-core - Extracts from the RI core library API docs

  - doc-std - Extracts from the RI std library API docs

  - pretty - Tests from the RI test suite's 'pretty' directory

  - run-pass - Tests from the RI test suite's 'run-pass' directory

  - rust-by-example - Extracts from [www.rustbyexample.com]



[lists]: https://github.com/Gankro/too-many-lists

[www.rustbyexample.com]: http://www.rust-by-example.com



## Compiler interface requirements



Running the test suite requires a Rust compiler that generally behaves

like the reference compiler. Currently the test suite expects the

compiler to support the `-o` flag. In the future it will expect more

flags to work.



## Licensing



For test cases, any OSI-approved license accepted. Non-free licenses

also welcome, but in another repo.



The way this is tracked currently is in a LICENSE file in each

directory of code that didn't originate from The Rust Project itself,

along with an ORIGIN file indicating where it came from.



Other code in CTRS is MIT/Apache-2.0.



## Scripts



This test suite is a bunch of python scripts for wrangling Rust code

from various places and into a form that is runnable.



* `dupes.py` - Lists files in the repo with identical hashes.

* `delete_dupes.py` - Deletes duplicates from git.

* `run.py` - The test suite runner. Just needs a `rustc` command

  available or the `RUSTC` environment variable to be set.

* `slurp.py` - Copies all Rust files that pass `stab.py` from one dir to another.

* `slurp_docs.py` - Extracts stable docs from one dir to another.

* `undoc_dir.py` - Recursively extracts doc tests from all .rs/.md files to a dir.

* `stab.py` - Checks whether a crate meets the criteria for inclusion.

* `undoc.py` - Extracts doc tests from a single file.



## Future directions



* Improve accuracy of staby.py

* Add support for reference compile-fail tests

* Support aux-build tests

* Support should-panic doc tests

* Support run-fail tests

* Add support for Makefile tests?

* FFI tests, like rust_test_helpers

* Sources

  * compile-fail, run-fail, run-make tests

  * In-tree std crate docs

  * Popular blog posts

  * rustforrubyists.com

  * euler-rust https://github.com/gifnksm/ProjectEulerRust

  * http://www.programming-idioms.org/about#about-block-language-coverage

  * https://github.com/geraldstanje/rust_snippets

  * guidelines

  * http://blog.burntsushi.net/rust-error-handling/

  * Dependency-free libs e.g. math, quickcheck

  * coretest

  * automatic extraction of #[test] cases

  * tool to extract build plans from cargo graphs

  * https://www.ralfj.de/projects/rust-101/main.html

  * error index examples!