Testing

Long tests

There are a series of test snippets located in the test/samples folder. If you want to run these, you can use this:

$ LONGTESTS=all python -m pytest test/

Valid values for the LONGTESTS variable (note that multiple values can be seperated by comma’s):

value meaning
all Go all in, run all tests possible
any Run some extra tests which take somewhat longer
python Convert sample snippets to python
jit Run a tricky test in which code is jitted
wasm Convert sample snippets to wasm
riscv,msp430, avr,microblaze, xtensa,arm Convert sample snippets to code for the given architecture

Some targets need iverilog to emulate a certain processor. If you want to run these, use this:

$ LONGTESTS=all IVERILOG=1 python -m pytest test/

3rd party test suites

There exist many different compiler validation suites. Some of them are pure validation sets, others are part of a compiler toolchain. In order to use these test suites, a series of test suite adapter files exists.

Available test adapters:

  • mcpp (set MCPP_DIR) test/lang/c/test_mcpp_test_suite.py
  • fortran compiler validation system 2.1 (set FCVS_DIR) test/lang/fortran/test_fortran_test_suite.py

WebAssembly spec

The WebAssembly specification contains a validation suite.

To use these tests, clone https://github.com/WebAssembly/spec and set the environment variable WASM_SPEC_DIR to the location where the code was cloned.

To run the test spec tests:

$ export WASM_SPEC_DIR=~/GIT/spec
$ python -m pytest test/wasm/test_suite_full -v

C testsuite

The c-testsuite is a collection of C test cases.

Usage with pytest:

$ export C_TEST_SUITE_DIR=/path/to/GIT/c-testsuite $ python -m pytest test/lang/c/test_c_test_suite.py -v

Usage as a script:

$ python test_c_test_suite.py /path/to/GIT/c-testsuite

See also:

https://github.com/c-testsuite/c-testsuite

Compiler testing

There are a number of ways to stress test the compiler.

One way is to compile existing C sourcecode, and see if the code compiles and runs.

Current results:

test sample Compiles Runs
libmad yes ?
8cc yes ?
lcc no ?
micropython no ?

libmad

The sourcecode for this test can be found here: https://www.underbit.com/products/mad/

To compile libmad, use the script tools/compile_libmad.py. This will compile the libmad sourcecode.

Compilation takes 45 seconds.

8cc

8cc is a small C compiler which can compile itself. PPCI can also compile it, running it remains a challenge.

Sourcecode is located here: https://github.com/rui314/8cc

To compile 8cc, use the script tools/compile_8cc.py

lcc

lcc is a C compiler written in C. Sourcecode can be found here: https://github.com/drh/lcc

To compile this sourcecode, use the script tools/compile_lcc.py

micropython

Micropython is a python implementation for microcontrollers. Website: http://micropython.org/

To compile this sourcecode, use the script tools/compile_micropython.py