How to use riscof run with testlist argument?

[Barabas5532]

I'm trying to pass a testlist to run a single test rather than the entire suite for faster iteration while I'm setting up the tests and debugging failures.

When I run riscof run --config=config.ini --suite=riscv-arch-test/riscv-test-suite/ --env=riscv-arch-test/riscv-test-suite/env --testfile test_list.yaml, some errors are emitted from riscof_sail_cSim.py related to directories already existing from these lines:

os.mkdir(os.path.join(self.work_dir,"sail_work"))
os.mkdir(os.path.join(self.work_dir,"test"))

After fixing these, more errors are coming from inside riscof scripts:

$ riscof run --config=config.ini --suite=riscv-arch-test/riscv-test-suite/ --env=riscv-arch-test/riscv-test-suite/env --testfile test_list.yaml
    INFO | ****** RISCOF: RISC-V Architectural Test Framework 1.25.3 *******
    INFO | using riscv_isac version : 0.18.0
    INFO | using riscv_config version : 3.7.2
    INFO | Reading configuration from: /home/barabas/projects/hardware/risc-v/tools/compliance/config.ini
    INFO | Preparing Models
    INFO | Read fusesoc name: ::crush:0.0.1
    INFO | Current working directory: /home/barabas/projects/hardware/risc-v/tools/compliance
    INFO | Running Build for DUT
    INFO | Build simulator: fusesoc run --target sim_cpu --build crush
    INFO | Running Build for Reference
Traceback (most recent call last):
  File "/home/barabas/projects/hardware/risc-v/.venv/bin/riscof", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/barabas/projects/hardware/risc-v/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/barabas/projects/hardware/risc-v/.venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/barabas/projects/hardware/risc-v/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/barabas/projects/hardware/risc-v/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/barabas/projects/hardware/risc-v/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/barabas/projects/hardware/risc-v/.venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/barabas/projects/hardware/risc-v/.venv/lib/python3.11/site-packages/riscof/cli.py", line 321, in run
    report_objects['results'] = framework.run(dut, base, ctx.obj.isa_file,
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/barabas/projects/hardware/risc-v/.venv/lib/python3.11/site-packages/riscof/framework/main.py", line 205, in run
    results = test.run_tests(dut, base, ispec['hart0'], pspec, work_dir, cntr_args)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/barabas/projects/hardware/risc-v/.venv/lib/python3.11/site-packages/riscof/framework/test.py", line 402, in run_tests
    os.mkdir(dut_test_list[entry]['work_dir'])
FileExistsError: [Errno 17] File exists: '/home/barabas/projects/hardware/risc-v/tools/compliance/riscof_work/rv32i_m/I/src/add-01.S/dut'

I get either FileExistsError or FileNotFoundError depending on whether I have run riscof with the default test list or not.

How can I make this work?

[cr1901]

@Barabas5532 I had to do something like this to make "inject a single test" into a RISCOF run, where my_test_list.yaml is your abbreviated set of tests:

shopt -s globstar
riscof testlist --config=config.ini --suite=riscv-arch-test/riscv-test-suite/ --env=riscv-arch-test/riscv-test-suite/env --work-dir=riscof_work
rm -rf riscof_work/**/dut/
rm -rf riscof_work/**/ref/
cp my_test_list.yaml riscof_work/test_list.yaml
riscof run --config=config.ini --suite=riscv-arch-test/riscv-test-suite/ --env=riscv-arch-test/riscv-test-suite/env --testfile=riscof_work/test_list.yaml --no-browser --no-clean
shopt -u globstar

Guessing this isn't quite intended, but it does work for my use case of "separating test generation from running the tests."

[Barabas5532]

#108 is related