SIGSEGV when using asan-cores in qemu_launcher fuzzer

[jthorpe6]

IMPORTANT

1. You have verified that the issue to be present in the current main branch
   yes

% git log | head -n 1
commit a212d66afef398cefb5ab7d1844e935efb702d23

Describe the bug
I get a SIGSEGV when trying to run the qemu_launcher example with asan

To Reproduce
Steps to reproduce the behavior:

add cargo-make to the docker file

% echo -e "RUN cargo install cargo-make" >> Dockerfile

build the docker container image

% docker build --platform linux/amd64 -t libafl .

run the docker container, clone LibAFL and run qemu_launcher with asan

% docker run --rm --platform linux/amd64 -it libafl
root@f072d778bf3f:/libafl# cd /tmp
root@f072d778bf3f:/tmp# git clone https://github.com/AFLplusplus/LibAFL.git && cd LibAFL/fuzzers/binary_only/qemu_launcher && cargo make asan

Cloning into 'LibAFL'...
remote: Enumerating objects: 49623, done.
remote: Counting objects: 100% (10040/10040), done.
remote: Compressing objects: 100% (2608/2608), done.
remote: Total 49623 (delta 8480), reused 8273 (delta 7371), pack-reused 39583 (from 1)
Receiving objects: 100% (49623/49623), 22.69 MiB | 5.70 MiB/s, done.
Resolving deltas: 100% (34798/34798), done.
[cargo-make] INFO - cargo make 0.37.18
[cargo-make] INFO - Calling cargo metadata to extract project info
[cargo-make] INFO - Cargo metadata done
[cargo-make] INFO - Project: qemu_launcher
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: asan
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: target_dir
[cargo-make] INFO - Running Task: arch_target_dir
[cargo-make] INFO - Running Task: deps_dir
[cargo-make] INFO - Running Task: zlib_unix_wget

SNIP

[cargo-make] INFO - Running Task: harness
[cargo-make] INFO - Execute Command: "cargo" "build" "--profile" "release" "--features" "x86_64" "--target-dir" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64"
   Compiling proc-macro2 v1.0.86
   Compiling unicode-ident v1.0.13
   Compiling rustversion v1.0.17
   
SNIP

Compiling qemu_launcher v0.13.2 (/tmp/LibAFL/fuzzers/binary_only/qemu_launcher)
   Compiling getset v0.1.3
   Compiling clap_derive v4.5.18
   Compiling thread_local v1.1.8
   Compiling num-derive v0.4.2
   Compiling syscall-numbers v4.0.0
   Compiling clap v4.5.19
   Compiling readonly v0.2.12
   Compiling capstone v0.12.0
warning: [email protected]: Qemu not found, cloning with git (d6637939526f453c69f4c6bfe4635feb5dc5c0be)...
warning: [email protected]: Version of llvm-config is 18 but needs to be at least rustc's version (19)! We will (try to) continue to build. Continue at your own risk, or rebuild with a set LLVM_CONFIG_PATH env variable, pointing to a newer version.
warning: [email protected]: wrapper.h has been regenerated.
warning: field `page_filter` is never read
   --> /tmp/LibAFL/libafl_qemu/src/modules/edges.rs:360:5
    |
357 | pub struct EdgeCoverageModule<AF, PF, V> {
    |            ------------------ field in this struct
...
360 |     page_filter: PF,
    |     ^^^^^^^^^^^
    |
    = note: `EdgeCoverageModule` has a derived impl for the trait `Debug`, but this is intentionally ignored during dead code analysis
    = note: `#[warn(dead_code)]` on by default

warning: `libafl_qemu` (lib) generated 1 warning
    Finished `release` profile [optimized + debuginfo] target(s) in 6m 18s
    
[cargo-make] INFO - Running Task: fuzzer
[cargo-make] INFO - Execute Command: "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/release/qemu_launcher-development" "--input" "./corpus" "--output" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/" "--log" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/log.txt" "--cores" "0" "--asan-cores" "0" "--" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/libpng-harness-development"
Fuzzing stopped by user. Good bye.
[cargo-make] INFO - Build Done in 456.63 seconds.
root@f3d6e2040375:/tmp/LibAFL/fuzzers/binary_only/qemu_launcher#       

Try run the fuzzer with the --verbose flag

root@f3d6e2040375:/tmp/LibAFL/fuzzers/binary_only/qemu_launcher# /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/release/qemu_launcher-development --input ./corpus/ --output /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/ --log /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/log.txt --cores 0 --asan-cores 0 --verbose -- /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/libpng-harness-development
qemu_launcher-development: QEMU internal SIGSEGV {code=MAPERR, addr=0x200071506a0b}
thread 'main' panicked at /tmp/LibAFL/libafl/src/events/llmp/restarting.rs:637:21:
Fuzzer-respawner: Storing state in crashed fuzzer instance did not work, no point to spawn the next client! This can happen if the child calls `exit()`, in that case make sure it uses `abort()`, if it got killed unrecoverable (OOM), or if there is a bug in the fuzzer itself. (Child exited with: 0)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Fuzzing stopped by user. Good bye.

Add the debug / log environment variables

root@f3d6e2040375:/tmp/LibAFL/fuzzers/binary_only/qemu_launcher# export QASAN_LOG=1
root@f3d6e2040375:/tmp/LibAFL/fuzzers/binary_only/qemu_launcher# export QASAN_DEBUG=1
root@f3d6e2040375:/tmp/LibAFL/fuzzers/binary_only/qemu_launcher# /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/release/qemu_launcher-development --input ./corpus/ --output /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/ --log /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/log.txt --cores 0 --asan-cores 0 --verbose -- /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/libpng-harness-development
==13629== QEMU-AddressSanitizer (v0.3)
==13629== Copyright (C) 2019-2021 Andrea Fioraldi <[email protected]>
==13629== 
qemu_launcher-development: QEMU internal SIGSEGV {code=MAPERR, addr=0x200071c12c0b}
thread 'main' panicked at /tmp/LibAFL/libafl/src/events/llmp/restarting.rs:637:21:
Fuzzer-respawner: Storing state in crashed fuzzer instance did not work, no point to spawn the next client! This can happen if the child calls `exit()`, in that case make sure it uses `abort()`, if it got killed unrecoverable (OOM), or if there is a bug in the fuzzer itself. (Child exited with: 0)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Fuzzing stopped by user. Good bye.

Environment info

root@f3d6e2040375:/tmp/LibAFL/fuzzers/binary_only/qemu_launcher#  lsb_release -a && \
    arch && \
    llvm-config --version && \
    rustup toolchain list && \
    rustc -V

No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm
x86_64
18.1.8
nightly-x86_64-unknown-linux-gnu (default)
1.76.0-x86_64-unknown-linux-gnu
rustc 1.83.0-nightly (06bb8364a 2024-10-01)

Expected behavior
when using asan cores not to cause a SIGSEGV

Additional context
I've also tried llvm-19 because of that warning during the compilation of libafl_qemu_sys

first remove all docker images/container

% docker rmi -f $(docker images -aq) ; docker rm -f $(docker ps -aq); docker system prune -f ; docker rmi $(docker images -aq); docker rm $(docker images -aq)

Then change LLVM version from 18 to 19

% sed -i s/18/19/g ./Dockerfile
% cat Dockerfile | grep 19   
# Install clang 19, common build tools
ENV LLVM_VERSION=19

build the container image again

docker build --platform linux/amd64 -t libafl .

clone LibAFL and attempt the qemu_launcher example with asan again

docker run --rm --platform linux/amd64 -it libafl
cd /tmp
git clone https://github.com/AFLplusplus/LibAFL.git && cd LibAFL/fuzzers/binary_only/qemu_launcher && cargo make asan

No llvm version warning, however asan cores still crashes.

export QASAN_LOG=1
export QASAN_DEBUG=1
/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/release/qemu_launcher-development --input ./corpus/ --output /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/ --log /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/log.txt --cores 0 --asan-cores 0 --verbose -- /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/libpng-harness-development

==13784== QEMU-AddressSanitizer (v0.3)
==13784== Copyright (C) 2019-2021 Andrea Fioraldi <[email protected]>
==13784== 
qemu_launcher-development: QEMU internal SIGSEGV {code=MAPERR, addr=0x20007532860b}
thread 'main' panicked at /tmp/LibAFL/libafl/src/events/llmp/restarting.rs:637:21:
Fuzzer-respawner: Storing state in crashed fuzzer instance did not work, no point to spawn the next client! This can happen if the child calls `exit()`, in that case make sure it uses `abort()`, if it got killed unrecoverable (OOM), or if there is a bug in the fuzzer itself. (Child exited with: 0)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Fuzzing stopped by user. Good bye.

[jthorpe6]

I'd like to add I did try adding the --privileged flag when running the docker container, which did not appear to change any of the results.

[rmalmain]

thank you for the report, i'll take a look at this during the week