Pytest Uses OpenMC Source Folder Instead of Installed Version

[ahnaf-tahmid-chowdhury]

Description

I am encountering an issue where, during the execution of pytest, the tests are using the OpenMC source folder rather than the installed version of OpenMC from the wheel package. As a result, libopenmc.so is not found, since it is no longer available in the source directory (after removing it in the recent updates to copy it from the build directory). I expected pytest to run against the installed wheel version of OpenMC, which includes all dependencies (including libopenmc.so).

Reproduction Steps

1. Build OpenMC with scikit-build-core and manylinux to create a wheel package that includes all dependencies (including libopenmc.so). PR wheel building with scikit build core #3087
2. Install the OpenMC wheel package using pip install.
3. Run pytest for testing.

Expected Behavior

The tests should use the installed version of OpenMC from the wheel package, which includes libopenmc.so.

Observed Behavior

Instead, pytest uses the OpenMC source directory, where libopenmc.so is missing. This leads to an OSError as shown below:

tests/regression_tests/cmfd_feed/test.py:2: in <module>
    from openmc import cmfd
openmc/cmfd.py:24: in <module>
    import openmc.lib
openmc/lib/__init__.py:30: in <module>
    _dll = CDLL(str(_filename))  # TODO: Remove str() when Python 3.12+
/opt/python/cp312-cp312/lib/python3.12/ctypes/__init__.py:379: in __init__
    self._handle = _dlopen(self._name, mode)
E   OSError: /root/openmc/openmc/lib/libopenmc.so: cannot open shared object file: No such file or directory

Investigation

I have confirmed that the installed wheel package is working correctly by running OpenMC notebooks, which operate with the installed version of OpenMC without any issues. However, when running tests with pytest, it always defaults to the OpenMC source folder.

Renaming the openmc source folder to something else causes the tests to stop working, which indicates that pytest is not using the installed version but the source directory.

regression_tests/cmfd_feed/test.py Fatal Python error: Segmentation fault

Current thread 0x00007f34ee8e6080 (most recent call first):
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/openmc/lib/core.py", line 344 in init
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/openmc/lib/core.py", line 610 in run_in_memory
  File "/usr/lib/python3.12/contextlib.py", line 137 in __enter__
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/openmc/cmfd.py", line 811 in run_in_memory
  File "/usr/lib/python3.12/contextlib.py", line 137 in __enter__
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/openmc/cmfd.py", line 778 in run
  File "/mnt/Data/Projects/OpenMC/tests/regression_tests/cmfd_feed/test.py", line 32 in test_cmfd_physical_adjoint
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/_pytest/python.py", line 159 in pytest_pyfunc_call
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/_pytest/python.py", line 1627 in runtest
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/_pytest/runner.py", line 174 in pytest_runtest_call
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/_pytest/runner.py", line 242 in <lambda>
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/_pytest/runner.py", line 241 in call_and_report
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/_pytest/runner.py", line 132 in runtestprotocol
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/_pytest/runner.py", line 113 in pytest_runtest_protocol
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/_pytest/main.py", line 362 in pytest_runtestloop
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/_pytest/main.py", line 337 in _main
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/_pytest/main.py", line 283 in wrap_session
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/_pytest/main.py", line 330 in pytest_cmdline_main
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/home/tahmid/venvs/openmc/lib/python3.12/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/home/tahmid/venvs/openmc/bin/pytest", line 8 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, lxml._elementpath, lxml.etree, h5py._errors, h5py.defs, h5py._objects, h5py.h5, h5py.utils, h5py.h5t, h5py.h5s, h5py.h5ac, h5py.h5p, h5py.h5r, h5py._proxy, h5py._conv, h5py.h5z, h5py.h5a, h5py.h5d, h5py.h5ds, h5py.h5g, h5py.h5i, h5py.h5o, h5py.h5f, h5py.h5fd, h5py.h5pl, h5py.h5l, h5py._selector, scipy._lib._ccallback_c, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.special._ellip_harm_2, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._direct, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.signal._sigtools, scipy._lib._uarray._uarray, scipy.signal._max_len_seq_inner, scipy.signal._upfirdn_apply, scipy.signal._spline, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, scipy.signal._sosfilt, scipy.signal._spectral, scipy.special.cython_special, scipy.stats._stats, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy.stats._ansari_swilk_statistics, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.stats._unuran.unuran_wrapper, scipy.signal._peak_finding_utils (total: 168)
[1]    49574 segmentation fault (core dumped)  pytest regression_tests/cmfd_feed/test.py

Additional Information

Any additional guidance on configuring pytest to properly use the installed version of OpenMC would be appreciated.

[shimwell]

I think this is one of the reasons we should move to a src/openmc layout instead of the current openmc folder layout. I think python imports from the folder if found before looking for packages with that name

[paulromano]

Thanks @ahnaf-tahmid-chowdhury for reporting. This is a known issue and duplicate of #1226 so I'll close this out but please continue to discuss there. As @shimwell said, the easy solution is to move to a src layout but that may be a bit disruptive with the current number of pending PRs we have.

[shimwell]

@ahnaf-tahmid-chowdhury if it makes the scikit core pip install easier then we can do this src folder layout on that branch and see how it goes