GSoC Project Updates, Progress with MicroPython Tests

The first item we decided to focus on after the mid-term was getting the MicroPython test suite to work with the RTEMS Port. In the previous post I wrote about the test “framework” (run-tests.py) and how it works, along with features required on the MicroPython side that allow the test framework to communicate with it (a “raw” REPL, and a so-called “raw paste mode”). There was a problem with raw paste mode where MicroPython would freeze up and not send the ACKs back to the test runner that are expected in this mode. I wanted to focus on getting the tests to run first, so I decided to look into this later and commented out the lines in the pyboard module that ask the device if raw paste mode is supported and flip a flag to use that mode.

So I focused on making the tests pass. I have been recording the progress on that in this issue.

This is still just the “basics” tests. We currently ignore the “extmod”, “micropython” and “float” test groups because they require dependencies (probably not float, I haven’t changed these since my early trials with run-tests.py where I wanted to get at least something to run, and I kept adding test groups to the “exclude” flag until I got there), like the unittest module to be installed at runtime (using a tool like mpremote, that once again needs Raw Paste mode).

I have made steady progress and currently, 501 out of 504 executed tests (around 16119 individual testcases) pass, 46 are skipped, and 3 are failing. As guided by my mentors, I wanted to get to this point with the “basics” tests where I could no longer just enable a feature in mpconfigport.h and make the test pass, and it is here (at least with the current set of excludes). Currently I am looking closely at the three failing testcases: builtin_compile, int_big_div and int_big_mul.

int_big_div stands out here as it mostly fails. There is no random number generation in the test, here’s the code:

for lhs in (1000000000000000000000000, 10000000000100000000000000, 10012003400000000000000007, 12349083434598210349871029923874109871234789):
    for rhs in range(1, 555):
        print(lhs // rhs)

# these check an edge case on 64-bit machines where two mpz limbs
# are used and the most significant one has the MSB set
x = 0x8000000000000000
print((x + 1) // x)
x = 0x86c60128feff5330
print((x + 1) // x)

# these check edge cases where borrow overflows
print((2 ** 48 - 1) ** 2 // (2 ** 48 - 1))
print((2 ** 256 - 2 ** 32) ** 2 // (2 ** 256 - 2 ** 32))

Looking at a diff of expected and logged outputs (expected ouptut on the right):

28258772161551968764006933464242814350651                       28258772161551968764006933464242814350651
28194254416890891209751209871858698336152                       28194254416890891209751209871858698336152
281300306027294008541847448573745124991423                    | 28130030602729408541847448573745124991423
280660987149959326133432498269866133                          | 28066098714995932613343249826986613343715
28002456767796395351181473750281428279443                       28002456767796395351181473750281428279443
27939102793208620701065678560801153554829                       27939102793208620701065678560801153554829

We see that one of the numbers seems larger than it should be, and the next one has a lot of digits missing. This might be related to the port using the MPZ module as the longint implementation, but I am currently not sure. I did try switching to MICROPY_LONGINT_IMPL_LONGLONG, but that just causes a crash, so it is also worth looking into.

I solved builtin_compile after I started writing this post. That testcase uses the __globals__ attribute on a function object, and I didn’t have that functionality enabled. So now we have 2 failing tests: int_big_div and int_big_mul. I am currently trying to fix int_big_div as I think it could be something related to the serial output. I did try calling fflush before printing stuff to the console but that didn’t fix it. And there’s of course the fact that it randomly fails. (so we have 1.5 passing testcases? :P)

I also compared the output of int_big_mul to the expected output but the diff tool just highlights the whole file so that is interesting.

more passing tests!

I also tried not ignoring the float group and got 55 more tests to pass! Here’s a snippet from the end of my latest run:

$ ./run-tests.py -t /dev/pts/2 -e "(extmod|micropython)/*"

...

559 tests performed (18668 individual testcases)
557 tests passed
55 tests skipped: async_await async_await2 async_def async_for async_for2 async_syntaxerror async_with async_with2 async_with_break async_with_return attrtuple2 builtin_help builtin_next_arg2 cexample_class cexample_module cexample_subclass class_inplace_op cmath_dunder cmath_fun cmath_fun_special exception_chain float2int_doubleprec_intbig float_divmod float_format_ints_doubleprec float_parse_doubleprec fun_code io_buffered_writer io_bytesio_cow io_bytesio_ext io_bytesio_ext2 io_iobase io_stringio1 io_stringio_base io_stringio_with io_write_ext math_constants_extra math_domain_special math_factorial_intbig math_fun_special math_isclose memoryview_slice_size nanbox_smallint print_exception rge_sm subclass_native_call subclass_native_init sys_atexit sys_exc_info sys_path sys_settrace_features sys_settrace_generator sys_settrace_loop sys_stdio sys_stdio_buffer sys_tracebacklimit
2 tests failed: int_big_div int_big_mul

Other than the test groups I explicitly ignore, the tests that are being executed are determined by run-tests.py based on the detected “minimal” (the default) configuration of the port. Before executing any tests it sends a little script to MicroPython that checks for certain features and returns a detected platform type (pre-defined). The RTEMS MicroPython port could, in the future, have its own set of test groups/individual testcases to ignore specified directly in the test runner (like the other “platforms”/ports), but the minimal configuration seems like a good set of tests for now.

raw-paste mode now works

I also mentioned raw-paste mode being a problem and how I had to comment out some lines from tools/pyboard.py to get the tests to run. Well now that is also fixed!

I started using write in my implementation of mp_hal_stdout_tx_strn instead of printf (I thought it was only used for printing characters to the screen but apparently it is also relevant for stuff like raw-paste mode) as using printf was leading to poorly formatted output that was making a lot of tests fail in the beginning. I removed the comments in tools/pyboard.py to see if the problem had changed now, but it just worked.

Raw Paste mode also brings niceties like being able to use mpremote run and mpremote connect. We can now also start looking into what’s required to be able to install mip packages to MicroPython at runtime.

I am currently working on getting the remaining tests to pass and simultaneously looking into how threads can be implemented. I looked at other ports’ implementations of this a few weeks ago so I already have some idea of it. They are not finalized yet but threads would be nice to have in the final deliverables. Networking is also something worth looking into. There are other available modules listed at docs.micropython.org that can simply be enabled, so I will be going through these and see if there’s anything there that needs implementation at the port level.

Thank you for your time!