Shutdown crash with GLIP USB3 backend

[koenenwmn]

Python currently crashes after executing my script to connect to an FPGA, program the cores, and trace the software output. However, it only crashes when I use the stm loggers.
Here's the output when I execute the script with:
gdb -return-child-result -batch -ex r -ex bt --args ./venv/bin/python3 src/host/demonstrator_runner/demonstrator_runner.py

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Starting demonstrator runner...
[New Thread 0x7ffff3edd700 (LWP 10419)]
[New Thread 0x7ffff36dc700 (LWP 10420)]
[New Thread 0x7ffff2edb700 (LWP 10421)]
[New Thread 0x7ffff26da700 (LWP 10422)]
[New Thread 0x7ffff1ed9700 (LWP 10423)]
[New Thread 0x7ffff12d6700 (LWP 10424)]
[New Thread 0x7ffff0ad5700 (LWP 10425)]
[New Thread 0x7fffe3fff700 (LWP 10426)]
INFO osd 2019-07-16 16:44:27,247 backend_cypressfx3.c:347 glip: Detected FX3 firmware with 16 bit FIFO width.
[New Thread 0x7fffe37fe700 (LWP 10427)]
INFO osd 2019-07-16 16:44:27,796 backend_cypressfx3.c:823 glip: Unable to transfer data to USB device. Error code: -1
Modules in Demonstrator
  0.0: OSD SCM v0 (1.1)
  0.1: OSD MAM v0 (1.3)
  0.2: OSD STM v0 (1.4)
  0.3: OSD CTM v0 (1.5)
  0.4: OSD MAM v0 (1.3)
  0.5: OSD STM v0 (1.4)
  0.6: OSD CTM v0 (1.5)
  0.7: OSD STM v0 (1.4)
  0.8: OSD CTM v0 (1.5)
  0.9: OSD MAM v0 (1.3)
  0.10: OSD STM v0 (1.4)
  0.11: OSD CTM v0 (1.5)
  0.12: OSD STM v0 (1.4)
  0.13: OSD CTM v0 (1.5)
  0.14: TUMLIS NOC_BRIDGE v0 (4.4)
  0.15: OSD MAM v0 (1.3)
  0.16: OSD STM v0 (1.4)
  0.17: OSD CTM v0 (1.5)
  0.18: OSD MAM v0 (1.3)
  0.19: OSD STM v0 (1.4)
  0.20: OSD CTM v0 (1.5)
  0.21: OSD MAM v0 (1.3)
  0.22: OSD STM v0 (1.4)
  0.23: OSD CTM v0 (1.5)
  0.24: OSD MAM v0 (1.3)
  0.25: OSD STM v0 (1.4)
  0.26: OSD CTM v0 (1.5)
  0.27: OSD STM v0 (1.4)
  0.28: OSD CTM v0 (1.5)
  0.29: OSD MAM v0 (1.3)
  0.30: OSD STM v0 (1.4)
  0.31: OSD CTM v0 (1.5)
Setting up system trace loggers
  DI addr 2
[New Thread 0x7fffe2ffd700 (LWP 10433)]
  DI addr 5
[New Thread 0x7fffe27fc700 (LWP 10434)]
  DI addr 7
[New Thread 0x7fffe1ffb700 (LWP 10435)]
  DI addr 10
[New Thread 0x7fffe17fa700 (LWP 10436)]
  DI addr 12
[New Thread 0x7fffe0ff9700 (LWP 10437)]
  DI addr 16
[New Thread 0x7fffbbfff700 (LWP 10438)]
  DI addr 19
[New Thread 0x7fffbb7fe700 (LWP 10439)]
  DI addr 22
[New Thread 0x7fffbaffd700 (LWP 10440)]
  DI addr 25
[New Thread 0x7fffba7fc700 (LWP 10441)]
  DI addr 27
[New Thread 0x7fffb9ffb700 (LWP 10442)]
  DI addr 30
[New Thread 0x7fffb97fa700 (LWP 10443)]
[New Thread 0x7fffb8ff9700 (LWP 10444)]
Loading memories
  Memory of tile 0
INFO osd 2019-07-16 16:44:29,447 memaccess.c:210 Load program header 0
INFO osd 2019-07-16 16:44:29,448 memaccess.c:210 Load program header 1
  Memory of tile 1
INFO osd 2019-07-16 16:44:29,611 memaccess.c:210 Load program header 0
INFO osd 2019-07-16 16:44:29,611 memaccess.c:210 Load program header 1
  Memory of tile 2
INFO osd 2019-07-16 16:44:29,696 memaccess.c:210 Load program header 0
INFO osd 2019-07-16 16:44:29,696 memaccess.c:210 Load program header 1
  Memory of tile 3
INFO osd 2019-07-16 16:44:29,789 memaccess.c:210 Load program header 0
INFO osd 2019-07-16 16:44:29,789 memaccess.c:210 Load program header 1
  Memory of tile 4
INFO osd 2019-07-16 16:44:29,885 memaccess.c:210 Load program header 0
INFO osd 2019-07-16 16:44:29,885 memaccess.c:210 Load program header 1
  Memory of tile 5
INFO osd 2019-07-16 16:44:29,975 memaccess.c:210 Load program header 0
INFO osd 2019-07-16 16:44:29,975 memaccess.c:210 Load program header 1
  Memory of tile 6
INFO osd 2019-07-16 16:44:30,069 memaccess.c:210 Load program header 0
INFO osd 2019-07-16 16:44:30,069 memaccess.c:210 Load program header 1
  Memory of tile 7
INFO osd 2019-07-16 16:44:30,157 memaccess.c:210 Load program header 0
INFO osd 2019-07-16 16:44:30,157 memaccess.c:210 Load program header 1
Starting CPUs
[Thread 0x7fffb8ff9700 (LWP 10444) exited]
Executing software for 5 seconds
Ending observation, shutting down
[Thread 0x7fffe3fff700 (LWP 10426) exited]
[Thread 0x7ffff12d6700 (LWP 10424) exited]
[Thread 0x7ffff0ad5700 (LWP 10425) exited]
Demonstrator runner finished.
[Thread 0x7fffb97fa700 (LWP 10443) exited]
[Thread 0x7fffb9ffb700 (LWP 10442) exited]
[Thread 0x7fffba7fc700 (LWP 10441) exited]
[Thread 0x7fffbaffd700 (LWP 10440) exited]
[Thread 0x7fffbb7fe700 (LWP 10439) exited]
[Thread 0x7fffbbfff700 (LWP 10438) exited]
[Thread 0x7fffe0ff9700 (LWP 10437) exited]
*** Error in `/data/home/gu95wey/workspace/aramis/demonstrator/venv/bin/python3': corrupted double-linked list: 0x0000000000cd5c80 ***
[Thread 0x7fffe17fa700 (LWP 10436) exited]
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7ffff78677e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x80baf)[0x7ffff7870baf]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7ffff787453c]
/usr/lib/x86_64-linux-gnu/libzmq.so.5(+0x1d23e)[0x7ffff490923e]
/usr/lib/x86_64-linux-gnu/libzmq.so.5(+0x488ba)[0x7ffff49348ba]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7ffff7bc16ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7ffff78f741d]
======= Memory map: ========
00400000-007aa000 r-xp 00000000 fc:00 1197004                            /usr/bin/python3.5
009a9000-009ab000 r--p 003a9000 fc:00 1197004                            /usr/bin/python3.5
009ab000-00a42000 rw-p 003ab000 fc:00 1197004                            /usr/bin/python3.5
00a42000-00d55000 rw-p 00000000 00:00 0                                  [heap]
7fff94000000-7fff94021000 rw-p 00000000 00:00 0 
7fff94021000-7fff98000000 ---p 00000000 00:00 0 
7fff9c000000-7fff9c021000 rw-p 00000000 00:00 0 
7fff9c021000-7fffa0000000 ---p 00000000 00:00 0 
7fffa0000000-7fffa0034000 rw-p 00000000 00:00 0 
7fffa0034000-7fffa4000000 ---p 00000000 00:00 0 
7fffa4000000-7fffa4021000 rw-p 00000000 00:00 0 
7fffa4021000-7fffa8000000 ---p 00000000 00:00 0 
7fffa8000000-7fffa8021000 rw-p 00000000 00:00 0 
7fffa8021000-7fffac000000 ---p 00000000 00:00 0 
7fffac000000-7fffac021000 rw-p 00000000 00:00 0 
7fffac021000-7fffb0000000 ---p 00000000 00:00 0 
7fffb0000000-7fffb0021000 rw-p 00000000 00:00 0 
7fffb0021000-7fffb4000000 ---p 00000000 00:00 0 
7fffb4000000-7fffb4021000 rw-p 00000000 00:00 0 
7fffb4021000-7fffb8000000 ---p 00000000 00:00 0 
7fffba7fd000-7fffba7fe000 ---p 00000000 00:00 0 
7fffba7fe000-7fffbaffe000 rw-p 00000000 00:00 0 
7fffbaffe000-7fffbafff000 ---p 00000000 00:00 0 
7fffbafff000-7fffbb7ff000 rw-p 00000000 00:00 0 
7fffbb7ff000-7fffbb800000 ---p 00000000 00:00 0 
7fffbb800000-7fffbc000000 rw-p 00000000 00:00 0 
7fffbc000000-7fffbc021000 rw-p 00000000 00:00 0 
7fffbc021000-7fffc0000000 ---p 00000000 00:00 0 
7fffc0000000-7fffc0021000 rw-p 00000000 00:00 0 
7fffc0021000-7fffc4000000 ---p 00000000 00:00 0 
7fffc4000000-7fffc4021000 rw-p 00000000 00:00 0 
7fffc4021000-7fffc8000000 ---p 00000000 00:00 0 
7fffc8000000-7fffc8021000 rw-p 00000000 00:00 0 
7fffc8021000-7fffcc000000 ---p 00000000 00:00 0 
7fffcc000000-7fffcc021000 rw-p 00000000 00:00 0 
7fffcc021000-7fffd0000000 ---p 00000000 00:00 0 
7fffd0000000-7fffd0021000 rw-p 00000000 00:00 0 
7fffd0021000-7fffd4000000 ---p 00000000 00:00 0 
7fffd4000000-7fffd4021000 rw-p 00000000 00:00 0 
7fffd4021000-7fffd8000000 ---p 00000000 00:00 0 
7fffd8000000-7fffd8021000 rw-p 00000000 00:00 0 
7fffd8021000-7fffdc000000 ---p 00000000 00:00 0 
7fffdc000000-7fffdc021000 rw-p 00000000 00:00 0 
7fffdc021000-7fffe0000000 ---p 00000000 00:00 0 
7fffe07f9000-7fffe07fa000 ---p 00000000 00:00 0 
7fffe07fa000-7fffe0ffa000 rw-p 00000000 00:00 0 
7fffe0ffa000-7fffe0ffb000 ---p 00000000 00:00 0 
7fffe0ffb000-7fffe17fb000 rw-p 00000000 00:00 0 
7fffe17fb000-7fffe17fc000 ---p 00000000 00:00 0 
7fffe17fc000-7fffe1ffc000 rw-p 00000000 00:00 0 
7fffe1ffc000-7fffe1ffd000 ---p 00000000 00:00 0 
7fffe1ffd000-7fffe27fd000 rw-p 00000000 00:00 0 
7fffe27fd000-7fffe27fe000 ---p 00000000 00:00 0 
7fffe27fe000-7fffe2ffe000 rw-p 00000000 00:00 0 
7fffe2ffe000-7fffe2fff000 ---p 00000000 00:00 0 
7fffe2fff000-7fffe37ff000 rw-p 00000000 00:00 0 
7fffe4000000-7fffe4148000 rw-p 00000000 00:00 0 
7fffe4148000-7fffe8000000 ---p 00000000 00:00 0 
7fffe8000000-7fffe8021000 rw-p 00000000 00:00 0 
7fffe8021000-7fffec000000 ---p 00000000 00:00 0 
7fffec000000-7fffec028000 rw-p 00000000 00:00 0 
7fffec028000-7ffff0000000 ---p 00000000 00:00 0 
7ffff16d9000-7ffff16da000 ---p 00000000 00:00 0 
7ffff16da000-7ffff1eda000 rw-p 00000000 00:00 0 
7ffff1eda000-7ffff1edb000 ---p 00000000 00:00 0 
7ffff1edb000-7ffff26db000 rw-p 00000000 00:00 0 
7ffff26db000-7ffff26dc000 ---p 00000000 00:00 0 
7ffff26dc000-7ffff2edc000 rw-p 00000000 00:00 0 
7ffff2edc000-7ffff2edd000 ---p 00000000 00:00 0 
7ffff2edd000-7ffff36dd000 rw-p 00000000 00:00 0 
7ffff36dd000-7ffff36de000 ---p 00000000 00:00 0 
7ffff36de000-7ffff3ede000 rw-p 00000000 00:00 0 
7ffff3ede000-7ffff3ef4000 r-xp 00000000 fc:00 654146                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7ffff3ef4000-7ffff40f3000 ---p 00016000 fc:00 654146                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7ffff40f3000-7ffff40f4000 rw-p 00015000 fc:00 654146                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7ffff40f4000-7ffff4266000 r-xp 00000000 fc:00 1179570                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7ffff4266000-7ffff4466000 ---p 00172000 fc:00 1179570                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7ffff4466000-7ffff4470000 r--p 00172000 fc:00 1179570                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7ffff4470000-7ffff4472000 rw-p 0017c000 fc:00 1179570                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7ffff4472000-7ffff4476000 rw-p 00000000 00:00 0 
7ffff4476000-7ffff44d3000 r-xp 00000000 fc:00 1202411                    /usr/lib/x86_64-linux-gnu/libsodium.so.18.0.1
7ffff44d3000-7ffff46d2000 ---p 0005d000 fc:00 1202411                    /usr/lib/x86_64-linux-gnu/libsodium.so.18.0.1
7ffff46d2000-7ffff46d3000 r--p 0005c000 fc:00 1202411                    /usr/lib/x86_64-linux-gnu/libsodium.so.18.0.1
7ffff46d3000-7ffff46d4000 rw-p 0005d000 fc:00 1202411                    /usr/lib/x86_64-linux-gnu/libsodium.so.18.0.1
7ffff46d4000-7ffff46eb000 r-xp 00000000 fc:00 654376                     /lib/x86_64-linux-gnu/libusb-1.0.so.0.1.0
7ffff46eb000-7ffff48ea000 ---p 00017000 fc:00 654376                     /lib/x86_64-linux-gnu/libusb-1.0.so.0.1.0
7ffff48ea000-7ffff48eb000 r--p 00016000 fc:00 654376                     /lib/x86_64-linux-gnu/libusb-1.0.so.0.1.0
7ffff48eb000-7ffff48ec000 rw-p 00017000 fc:00 654376                     /lib/x86_64-linux-gnu/libusb-1.0.so.0.1.0
7ffff48ec000-7ffff494e000 r-xp 00000000 fc:00 1198658                    /usr/lib/x86_64-linux-gnu/libzmq.so.5.0.0
7ffff494e000-7ffff4b4d000 ---p 00062000 fc:00 1198658                    /usr/lib/x86_64-linux-gnu/libzmq.so.5.0.0
7ffff4b4d000-7ffff4b51000 r--p 00061000 fc:00 1198658                    /usr/lib/x86_64-linux-gnu/libzmq.so.5.0.0
7ffff4b51000-7ffff4b52000 rw-p 00065000 fc:00 1198658                    /usr/lib/x86_64-linux-gnu/libzmq.so.5.0.0
7ffff4b52000-7ffff4b64000 r-xp 00000000 fc:02 11802060                   /data/home/gu95wey/workspace/aramis/demonstrator/external/optimsoc-src/objdir/dist/host/lib/libglip.so.0.0.0
7ffff4b64000-7ffff4d64000 ---p 00012000 fc:02 11802060                   /data/home/gu95wey/workspace/aramis/demonstrator/external/optimsoc-src/objdir/dist/host/lib/libglip.so.0.0.0
7ffff4d64000-7ffff4d65000 r--p 00012000 fc:02 11802060                   /data/home/gu95wey/workspace/aramis/demonstrator/external/optimsoc-src/objdir/dist/host/lib/libglip.so.0.0.0
7ffff4d65000-7ffff4d66000 rw-p 00013000 fc:02 11802060                   /data/home/gu95wey/workspace/aramis/demonstrator/external/optimsoc-src/objdir/dist/host/lib/libglip.so.0.0.0
7ffff4d66000-7ffff4d7d000 r-xp 00000000 fc:00 1181792                    /usr/lib/x86_64-linux-gnu/libelf-0.165.so
7ffff4d7d000-7ffff4f7c000 ---p 00017000 fc:00 1181792                    /usr/lib/x86_64-linux-gnu/libelf-0.165.so
7ffff4f7c000-7ffff4f7d000 r--p 00016000 fc:00 1181792                    /usr/lib/x86_64-linux-gnu/libelf-0.165.so
7ffff4f7d000-7ffff4f7e000 rw-p 00017000 fc:00 1181792                    /usr/lib/x86_64-linux-gnu/libelf-0.165.so
7ffff4f7e000-7ffff501f000 r-xp 00000000 fc:00 1201605                    /usr/lib/x86_64-linux-gnu/libczmq.so.3.0.0
7ffff501f000-7ffff521f000 ---p 000a1000 fc:00 1201605                    /usr/lib/x86_64-linux-gnu/libczmq.so.3.0.0
7ffff521f000-7ffff5220000 r--p 000a1000 fc:00 1201605                    /usr/lib/x86_64-linux-gnu/libczmq.so.3.0.0
7ffff5220000-7ffff5221000 rw-p 000a2000 fc:00 1201605                    /usr/lib/x86_64-linux-gnu/libczmq.so.3.0.0
7ffff5221000-7ffff5228000 r-xp 00000000 fc:00 655491                     /lib/x86_64-linux-gnu/librt-2.23.so
7ffff5228000-7ffff5427000 ---p 00007000 fc:00 655491                     /lib/x86_64-linux-gnu/librt-2.23.so
7ffff5427000-7ffff5428000 r--p 00006000 fc:00 655491                     /lib/x86_64-linux-gnu/librt-2.23.so
7ffff5428000-7ffff5429000 rw-p 00007000 fc:00 655491                     /lib/x86_64-linux-gnu/librt-2.23.so
7ffff5429000-7ffff5443000 r-xp 00000000 fc:02 11404659                   /data/home/gu95wey/workspace/aramis/demonstrator/external/optimsoc/framework/host/lib/libosd.so.0.0.1
7ffff5443000-7ffff5642000 ---p 0001a000 fc:02 11404659                   /data/home/gu95wey/workspace/aramis/demonstrator/external/optimsoc/framework/host/lib/libosd.so.0.0.1
7ffff5642000-7ffff5643000 r--p 00019000 fc:02 11404659                   /data/home/gu95wey/workspace/aramis/demonstrator/external/optimsoc/framework/host/lib/libosd.so.0.0.1
7ffff5643000-7ffff5644000 rw-p 0001a000 fc:02 11404659                   /data/home/gu95wey/workspace/aramis/demonstrator/external/optimsoc/framework/host/lib/libosd.so.0.0.1
7ffff5644000-7ffff5695000 r-xp 00000000 fc:02 12977969                   /data/home/gu95wey/workspace/aramis/demonstrator/venv/lib/python3.5/site-packages/osd.cpython-35m-x86_64-linux-gnu.so
7ffff5695000-7ffff5894000 ---p 00051000 fc:02 12977969                   /data/home/gu95wey/workspace/aramis/demonstrator/venv/lib/python3.5/site-packages/osd.cpython-35m-x86_64-linux-gnu.so
7ffff5894000-7ffff5895000 r--p 00050000 fc:02 12977969                   /data/home/gu95wey/workspace/aramis/demonstrator/venv/lib/python3.5/site-packages/osd.cpython-35m-x86_64-linux-gnu.so
7ffff5895000-7ffff589c000 rw-p 00051000 fc:02 12977969                   /data/home/gu95wey/workspace/aramis/demonstrator/venv/lib/python3.5/site-packages/osd.cpython-35m-x86_64-linux-gnu.so
7ffff589c000-7ffff58dd000 rw-p 00000000 00:00 0 
7ffff58dd000-7ffff58e0000 r-xp 00000000 fc:02 12977987                   /data/home/gu95wey/workspace/aramis/demonstrator/venv/lib/python3.5/site-packages/psutil/_psutil_posix.cpython-35m-x86_64-linux-gnu.so
7ffff58e0000-7ffff5adf000 ---p 00003000 fc:02 12977987                   /data/home/gu95wey/workspace/aramis/demonstrator/venv/lib/python3.5/site-packages/psutil/_psutil_posix.cpython-35m-x86_64-linux-gnu.so
7ffff5adf000-7ffff5ae0000 r--p 00002000 fc:02 12977987                   /data/home/gu95wey/workspace/aramis/demonstrator/venv/lib/python3.5/site-packages/psutil/_psutil_posix.cpython-35m-x86_64-linux-gnu.so
7ffff5ae0000-7ffff5ae1000 rw-p 00003000 fc:02 12977987                   /data/home/gu95wey/workspace/aramis/demonstrator/venv/lib/python3.5/site-packages/psutil/_psutil_posix.cpython-35m-x86_64-linux-gnu.so
7ffff5ae1000-7ffff5ae6000 r-xp 00000000 fc:02 12977986                   /data/home/gu95wey/workspace/aramis/demonstrator/venv/lib/python3.5/site-packages/psutil/_psutil_linux.cpython-35m-x86_64-linux-gnu.so
7ffff5ae6000-7ffff5ce5000 ---p 00005000 fc:02 12977986                   /data/home/gu95wey/workspace/aramis/demonstrator/venv/lib/python3.5/site-packages/psutil/_psutil_linux.cpython-35m-x86_64-linux-gnu.so
7ffff5ce5000-7ffff5ce6000 r--p 00004000 fc:02 12977986                   /data/home/gu95wey/workspace/aramis/demonstrator/venv/lib/python3.5/site-packages/psutil/_psutil_linux.cpython-35m-x86_64-linux-gnu.so
7ffff5ce6000-7ffff5ce7000 rw-p 00005000 fc:02 12977986                   /data/home/gu95wey/workspace/aramis/demonstrator/venv/lib/python3.5/site-packages/psutil/_psutil_linux.cpython-35m-x86_64-linux-gnu.so
7ffff5ce7000-7ffff5d27000 rw-p 00000000 00:00 0 
7ffff5d27000-7ffff5d48000 r-xp 00000000 fc:00 654154                     /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7ffff5d48000-7ffff5f47000 ---p 00021000 fc:00 654154                     /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7ffff5f47000-7ffff5f48000 r--p 00020000 fc:00 654154                     /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7ffff5f48000-7ffff5f49000 rw-p 00021000 fc:00 654154                     /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7ffff5f49000-7ffff5f50000 r-xp 00000000 fc:00 1197576                    /usr/lib/python3.5/lib-dynload/_lzma.cpython-35m-x86_64-linux-gnu.so
7ffff5f50000-7ffff614f000 ---p 00007000 fc:00 1197576                    /usr/lib/python3.5/lib-dynload/_lzma.cpython-35m-x86_64-linux-gnu.so
7ffff614f000-7ffff6150000 r--p 00006000 fc:00 1197576                    /usr/lib/python3.5/lib-dynload/_lzma.cpython-35m-x86_64-linux-gnu.so
7ffff6150000-7ffff6152000 rw-p 00007000 fc:00 1197576                    /usr/lib/python3.5/lib-dynload/_lzma.cpython-35m-x86_64-linux-gnu.so
7ffff6152000-7ffff6161000 r-xp 00000000 fc:00 655353                     /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7ffff6161000-7ffff6360000 ---p 0000f000 fc:00 655353                     /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7ffff6360000-7ffff6361000 r--p 0000e000 fc:00 655353                     /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7ffff6361000-7ffff6362000 rw-p 0000f000 fc:00 655353                     /lib/x86_64-linux-gnu/libbz2.so.1.0.4
7ffff6392000-7ffff6396000 r-xp 00000000 fc:00 1197590                    /usr/lib/python3.5/lib-dynload/_bz2.cpython-35m-x86_64-linux-gnu.so
7ffff6396000-7ffff6595000 ---p 00004000 fc:00 1197590                    /usr/lib/python3.5/lib-dynload/_bz2.cpython-35m-x86_64-linux-gnu.so
7ffff6595000-7ffff6596000 r--p 00003000 fc:00 1197590                    /usr/lib/python3.5/lib-dynload/_bz2.cpython-35m-x86_64-linux-gnu.so
7ffff6596000-7ffff6597000 rw-p 00004000 fc:00 1197590                    /usr/lib/python3.5/lib-dynload/_bz2.cpython-35m-x86_64-linux-gnu.so
7ffff6597000-7ffff67f8000 rw-p 00000000 00:00 0 
7ffff67f8000-7ffff6c9d000 r--p 00000000 fc:00 1178698                    /usr/lib/locale/locale-archive
7ffff6c9d000-7ffff6da5000 r-xp 00000000 fc:00 655620                     /lib/x86_64-linux-gnu/libm-2.23.so
7ffff6da5000-7ffff6fa4000 ---p 00108000 fc:00 655620                     /lib/x86_64-linux-gnu/libm-2.23.so
7ffff6fa4000-7ffff6fa5000 r--p 00107000 fc:00 655620                     /lib/x86_64-linux-gnu/libm-2.23.so
7ffff6fa5000-7ffff6fa6000 rw-p 00108000 fc:00 655620                     /lib/x86_64-linux-gnu/libm-2.23.so
7ffff6fa6000-7ffff6fbf000 r-xp 00000000 fc:00 654479                     /lib/x86_64-linux-gnu/libz.so.1.2.8
7ffff6fbf000-7ffff71be000 ---p 00019000 fc:00 654479                     /lib/x86_64-linux-gnu/libz.so.1.2.8
7ffff71be000-7ffff71bf000 r--p 00018000 fc:00 654479                     /lib/x86_64-linux-gnu/libz.so.1.2.8
7ffff71bf000-7ffff71c0000 rw-p 00019000 fc:00 654479                     /lib/x86_64-linux-gnu/libz.so.1.2.8
7ffff71c0000-7ffff71e6000 r-xp 00000000 fc:00 654256                     /lib/x86_64-linux-gnu/libexpat.so.1.6.0
7ffff71e6000-7ffff73e6000 ---p 00026000 fc:00 654256                     /lib/x86_64-linux-gnu/libexpat.so.1.6.0
7ffff73e6000-7ffff73e8000 r--p 00026000 fc:00 654256                     /lib/x86_64-linux-gnu/libexpat.so.1.6.0
7ffff73e8000-7ffff73e9000 rw-p 00028000 fc:00 654256                     /lib/x86_64-linux-gnu/libexpat.so.1.6.0
7ffff73e9000-7ffff73eb000 r-xp 00000000 fc:00 655617                     /lib/x86_64-linux-gnu/libutil-2.23.so
7ffff73eb000-7ffff75ea000 ---p 00002000 fc:00 655617                     /lib/x86_64-linux-gnu/libutil-2.23.so
7ffff75ea000-7ffff75eb000 r--p 00001000 fc:00 655617                     /lib/x86_64-linux-gnu/libutil-2.23.so
7ffff75eb000-7ffff75ec000 rw-p 00002000 fc:00 655617                     /lib/x86_64-linux-gnu/libutil-2.23.so
7ffff75ec000-7ffff75ef000 r-xp 00000000 fc:00 655493                     /lib/x86_64-linux-gnu/libdl-2.23.so
7ffff75ef000-7ffff77ee000 ---p 00003000 fc:00 655493                     /lib/x86_64-linux-gnu/libdl-2.23.so
7ffff77ee000-7ffff77ef000 r--p 00002000 fc:00 655493                     /lib/x86_64-linux-gnu/libdl-2.23.so
7ffff77ef000-7ffff77f0000 rw-p 00003000 fc:00 655493                     /lib/x86_64-linux-gnu/libdl-2.23.so
7ffff77f0000-7ffff79b0000 r-xp 00000000 fc:00 655612                     /lib/x86_64-linux-gnu/libc-2.23.so
7ffff79b0000-7ffff7bb0000 ---p 001c0000 fc:00 655612                     /lib/x86_64-linux-gnu/libc-2.23.so
7ffff7bb0000-7ffff7bb4000 r--p 001c0000 fc:00 655612                     /lib/x86_64-linux-gnu/libc-2.23.so
7ffff7bb4000-7ffff7bb6000 rw-p 001c4000 fc:00 655612                     /lib/x86_64-linux-gnu/libc-2.23.so
7ffff7bb6000-7ffff7bba000 rw-p 00000000 00:00 0 
7ffff7bba000-7ffff7bd2000 r-xp 00000000 fc:00 655495                     /lib/x86_64-linux-gnu/libpthread-2.23.so
7ffff7bd2000-7ffff7dd1000 ---p 00018000 fc:00 655495                     /lib/x86_64-linux-gnu/libpthread-2.23.so
7ffff7dd1000-7ffff7dd2000 r--p 00017000 fc:00 655495                     /lib/x86_64-linux-gnu/libpthread-2.23.so
7ffff7dd2000-7ffff7dd3000 rw-p 00018000 fc:00 655495                     /lib/x86_64-linux-gnu/libpthread-2.23.so
7ffff7dd3000-7ffff7dd7000 rw-p 00000000 00:00 0 
7ffff7dd7000-7ffff7dfd000 r-xp 00000000 fc:00 655494                     /lib/x86_64-linux-gnu/ld-2.23.so
7ffff7e01000-7ffff7fc7000 rw-p 00000000 00:00 0 
7ffff7fcf000-7ffff7fd0000 rw-p 00000000 00:00 0 
7ffff7fd0000-7ffff7fee000 r-xp 00000000 fc:00 655876                     /lib/x86_64-linux-gnu/libudev.so.1.6.4
7ffff7fee000-7ffff7fef000 r--p 0001d000 fc:00 655876                     /lib/x86_64-linux-gnu/libudev.so.1.6.4
7ffff7fef000-7ffff7ff0000 rw-p 0001e000 fc:00 655876                     /lib/x86_64-linux-gnu/libudev.so.1.6.4
Thread 2 "python3" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff3edd700 (LWP 10419)]
0x00007ffff7825428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
#0  0x00007ffff7825428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007ffff782702a in __GI_abort () at abort.c:89
#2  0x00007ffff78677ea in __libc_message (do_abort=2, fmt=fmt@entry=0x7ffff7980ed8 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007ffff7870baf in malloc_printerr (ar_ptr=0x7ffff7bb4b20 <main_arena>, ptr=<optimized out>, str=0x7ffff797dc92 "corrupted double-linked list", action=3) at malloc.c:5006
#4  _int_free (av=0x7ffff7bb4b20 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:4005
#5  0x00007ffff787453c in __GI___libc_free (mem=<optimized out>) at malloc.c:2968
#6  0x00007ffff490923e in zmq::epoll_t::loop (this=0xcb0850) at src/epoll.cpp:176
#7  0x00007ffff49348ba in thread_routine (arg_=0xcb08d0) at src/thread.cpp:96
#8  0x00007ffff7bc16ba in start_thread (arg=0x7ffff3edd700) at pthread_create.c:333
#9  0x00007ffff78f741d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

[imphil]

Two things you can try:

1. Try to use the debug version of Python to get more information.
   • Install the python3-dbg Ubuntu package
   • Use python3-dbg instead of python3 when creating the venv, and then when running the simulation. (delete the old venv first)
2. Run the simulation through valgrind. To do that,
   • install valgrind
   • recompile osd-sw with --enable-debug --enable-valgrind --disable-asan passed to configure
   • run valgrind --tool=memcheck ./venv/bin/python3 src/host/demonstrator_runner/demonstrator_runner.py and see what it has to offer.

[koenenwmn]

Thanks for the reply!
Unfortunately the problem does not seem to be deterministic. With both options the script will sometimes run through without issues, sometimes crash. When using python-dbg, I still don't get backtraces with function names and lines in the source code. Instead it prints out a large part of the python code.
Valgrind shows a number of errors even if it runs through. However, when it crashes it shows the following summary (I included the invalid read right before the summary):

...
==3397== Invalid read of size 8
==3397==    at 0x94A0CF0: cbuf_reserve (cbuf.c:413)
==3397==    by 0x94A86DF: usb_read_thread (backend_cypressfx3.c:917)
==3397==    by 0x4E416B9: start_thread (pthread_create.c:333)
==3397==  Address 0xc5c1d18 is 104 bytes inside a block of size 176 free'd
==3397==    at 0x4C2EDEB: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3397==    by 0x94A0AD1: cbuf_free (cbuf.c:279)
==3397==    by 0x94A7CF6: gb_cypressfx3_close (backend_cypressfx3.c:518)
==3397==    by 0x94A1D74: glip_close (glip.c:551)
==3397==    by 0x8BD3322: osd_gateway_glip_disconnect (gateway_glip.c:347)
==3397==    by 0x8976EDD: __pyx_pf_3osd_11GatewayGlip_6disconnect (osd.c:8866)
==3397==    by 0x8976F89: __pyx_pw_3osd_11GatewayGlip_7disconnect (osd.c:8845)
==3397==    by 0x54A9A4: call_function (ceval.c:4684)
==3397==    by 0x547F42: PyEval_EvalFrameEx (ceval.c:3236)
==3397==    by 0x54A74D: fast_function (ceval.c:4803)
==3397==    by 0x54ADA4: call_function (ceval.c:4730)
==3397==    by 0x547F42: PyEval_EvalFrameEx (ceval.c:3236)
==3397==  Block was alloc'd at
==3397==    at 0x4C2FB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3397==    by 0x94A0947: cbuf_init (cbuf.c:223)
==3397==    by 0x94A77FC: gb_cypressfx3_open (backend_cypressfx3.c:418)
==3397==    by 0x94A1CE3: glip_open (glip.c:521)
==3397==    by 0x8BD30BD: osd_gateway_glip_connect (gateway_glip.c:314)
==3397==    by 0x8976F9B: __pyx_pf_3osd_11GatewayGlip_4connect (osd.c:8805)
==3397==    by 0x8977047: __pyx_pw_3osd_11GatewayGlip_5connect (osd.c:8784)
==3397==    by 0x54A9A4: call_function (ceval.c:4684)
==3397==    by 0x547F42: PyEval_EvalFrameEx (ceval.c:3236)
==3397==    by 0x54A74D: fast_function (ceval.c:4803)
==3397==    by 0x54ADA4: call_function (ceval.c:4730)
==3397==    by 0x547F42: PyEval_EvalFrameEx (ceval.c:3236)
==3397== 
==3397== Invalid read of size 8
==3397==    at 0x99202A6: libusb_submit_transfer (in /lib/x86_64-linux-gnu/libusb-1.0.so.0.1.0)
==3397==    by 0x9922658: do_sync_bulk_transfer (in /lib/x86_64-linux-gnu/libusb-1.0.so.0.1.0)
==3397==    by 0x9922A2E: libusb_bulk_transfer (in /lib/x86_64-linux-gnu/libusb-1.0.so.0.1.0)
==3397==    by 0x94A8747: usb_read_thread (backend_cypressfx3.c:926)
==3397==    by 0x4E416B9: start_thread (pthread_create.c:333)
==3397==  Address 0x40 is not stack'd, malloc'd or (recently) free'd
==3397== 
==3397== 
==3397== Process terminating with default action of signal 11 (SIGSEGV)
==3397==  Access not within mapped region at address 0x40
==3397==    at 0x99202A6: libusb_submit_transfer (in /lib/x86_64-linux-gnu/libusb-1.0.so.0.1.0)
==3397==    by 0x9922658: do_sync_bulk_transfer (in /lib/x86_64-linux-gnu/libusb-1.0.so.0.1.0)
==3397==    by 0x9922A2E: libusb_bulk_transfer (in /lib/x86_64-linux-gnu/libusb-1.0.so.0.1.0)
==3397==    by 0x94A8747: usb_read_thread (backend_cypressfx3.c:926)
==3397==    by 0x4E416B9: start_thread (pthread_create.c:333)
==3397==  If you believe this happened as a result of a stack
==3397==  overflow in your program's main thread (unlikely but
==3397==  possible), you can try to increase the size of the
==3397==  main thread stack using the --main-stacksize= flag.
==3397==  The main thread stack size used in this run was 8388608.
==3397== 
==3397== HEAP SUMMARY:
==3397==     in use at exit: 4,092,031 bytes in 4,752 blocks
==3397==   total heap usage: 4,797,294 allocs, 4,792,542 frees, 330,228,116 bytes allocated
==3397== 
==3397== LEAK SUMMARY:
==3397==    definitely lost: 40 bytes in 1 blocks
==3397==    indirectly lost: 0 bytes in 0 blocks
==3397==      possibly lost: 2,861,639 bytes in 3,656 blocks
==3397==    still reachable: 1,230,352 bytes in 1,095 blocks
==3397==         suppressed: 0 bytes in 0 blocks
==3397== Rerun with --leak-check=full to see details of leaked memory
==3397== 
==3397== For counts of detected and suppressed errors, rerun with: -v
==3397== ERROR SUMMARY: 1172 errors from 36 contexts (suppressed: 0 from 0)
Killed

[imphil]

Ah, now I see it. You're having a shutdown crash. In the GDB trace, you have "Demonstrator runner finished." and then the threads are exiting. That's confirmed by the Valgrind output, where we have

==3397==    at 0x4C2EDEB: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3397==    by 0x94A0AD1: cbuf_free (cbuf.c:279)
==3397==    by 0x94A7CF6: gb_cypressfx3_close (backend_cypressfx3.c:518)
==3397==    by 0x94A1D74: glip_close (glip.c:551)
==3397==    by 0x8BD3322: osd_gateway_glip_disconnect (gateway_glip.c:347)

The shutdown process is a bit fragile especially with the USB GLIP backend, as libusb has slightly unexpected/unclean behavior around thread shutdown sometimes.

I'll look into it later today. If you only want to avoid the crash, simply remove the disconnect statements from your Python code and force an "unclean" shutdown this way.

[koenenwmn]

Thanks for looking into that!
Removing the disconnect statements mostly helps, although I now sometimes get a "(core dumped)" at the end (which, however, never happens when I run it through Valgrind). I noticed that this only happens when I use the stm loggers. If I only send/receive data to/from my new debug module everything seems to work fine.
In any case I can continue the development this way for now.

[imphil]

STM loggers stream data at random times, so they expose the problem best. But it could equally happen with any other debug module which still sends data while the host side is in the process of being torn down.