Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open3D Python cannot find DLLs in add_dll_directory-locations #7104

Open
3 tasks done
fegorsch opened this issue Dec 19, 2024 · 1 comment · May be fixed by #7108
Open
3 tasks done

Open3D Python cannot find DLLs in add_dll_directory-locations #7104

fegorsch opened this issue Dec 19, 2024 · 1 comment · May be fixed by #7108
Labels
bug Not a build issue, this is likely a bug.

Comments

@fegorsch
Copy link

fegorsch commented Dec 19, 2024

Checklist

Describe the issue

I try to import custom-build Open3D with CUDA on Windows. In __init__.py when open3d/cuda/pybind.cp310-win_amd64.pyd is loaded, open3d/tbb12.dll is not found, although its location has been added to the DLL-search-directories with os.add_dll_directory before.

Workarounds:

  1. Copying tbb12.dll into Windows/system32 resolves the problem.
  2. Loading tbb12.dll explicitly with CDLL(str(Path(__file__).parent / "tbb12.dll")) in __init__.py before loading cuda/pybind.cp310-win_amd64.pyd resolves the problem.
  3. Adding tbb12.dll's location to PATH resolves the problem. I.e. doing os.environ['PATH'] = str(Path(__file__).parent) + ';' + os.environ.get('PATH', '') in __init__.py.

Main question: can anyone reproduce the problem or has any insights into why add_dll_directory seems not to be respected here? If this is a general problem, I'd open a PR that applies workaround 3.

Steps to reproduce the bug

Build Open3D with the software versions stated below and try to import it with `import open3d`.

Error message

I get the warning below. Note that the hint about the CUDA libraries is wrong in my case and only the "Reported error" is relevant.

"Open3D was built with CUDA support, but an error ocurred while loading the Open3D CUDA Python bindings. This is usually because the CUDA libraries could not be found. Check your CUDA installation. Falling back to the CPU pybind library. Reported error: Could not find module 'c:\workspace\visionkit\venv310\lib\site-packages\open3d\cuda\pybind.cp310-win_amd64.pyd' (or one of its dependencies). Try using the full path with constructor syntax."

Expected behavior

pybind.cp310-win_amd64.pyd is loaded without error. I.e. tbb12.dll can be located and loaded during the call to CDLL().

Open3D, Python and System information

- Operating system: Windows 11
- Python version: Python 3.10.11
- Open3D version: 0.18.0+8e434558a
- System architecture: x86
- Is this a remote workstation?: no
- How did you install Open3D?: build from source
- Compiler version (if built from source): msvc 14.3

Additional information

If I understand this correctly, the problem could perhaps also be that tbb12.dll is not finding its dependencies and not that tbb12.dll is not found. I did not follow this particular rabbit hole, further, though. If this is the problem, Open3D should perhaps implement a workaround or switch to a different tbb-version.

[edit: with some further research I found that this problem also exists for CUDA-DLLs, so I'm pretty sure the issue above has nothing to do with this issue.]

@fegorsch fegorsch added the bug Not a build issue, this is likely a bug. label Dec 19, 2024
@fegorsch fegorsch changed the title Open3D Python cannot find tbb12.dll (or a dependency thereof) Open3D Python cannot find DLLs in add_dll_directory-locations Dec 19, 2024
@fegorsch
Copy link
Author

fegorsch commented Dec 20, 2024

I reproduced the experiment from here with the open3d-package, i.e. tried all combinations of winmode=0/winmode=None and add_dll_directory / PATH. I did this for Python 3.10 and 3.13, all on Windows 11 with my custom build and came to the same conclusion as in the thread: only the following combinations work:

  • winmode=0 and adding the DLL-search-directory to PATH,
  • winmode=None and adding the DLL-search-directory via os.add_dll_directory.

Open3D currently uses winmode=0 + os.add_dll_directory, this seems to be the problem.

I'll open a PR with a change to winmode=None + os.add_dll_directory, as this is the documented (i.e. winmode=0 is undocumented) and recommended way of setting DLL-search-directories.

I guess winmode=0worked originally, because it causes PATH to be respected and the author of the PR had the CUDA-DLL-directory listed in there. At the same time he probably did not build with anything, that puts DLLs into the open3d-package directory (e.g. tbb). To also support this use-case, I propose to explicitly get the CUDA-DLL-directory from PATH and add it via add_dll_directory.

fegorsch pushed a commit to fegorsch/Open3D that referenced this issue Dec 20, 2024
This PR introduced `winmode=0` for loading DLLs during import:
isl-org#5259

This however leads to directories added via `os.add_dll_directory` not
being considered, when loading DLLs (see
isl-org#7104 (comment)).
In particular this leads to open3d/tbb12.dll not being found.

Fix this by going back to the default DLL-search mode (`winmode=None`).

In order to still be able to find CUDA-DLLs, that are not part of the
open3d-package, search their directory in PATH and explicitly add it to
the DLL-search-locations for CUDA-builds.
@fegorsch fegorsch linked a pull request Dec 20, 2024 that will close this issue
9 tasks
fegorsch pushed a commit to fegorsch/Open3D that referenced this issue Dec 20, 2024
This PR introduced `winmode=0` for loading DLLs during import:
isl-org#5259

This however leads to directories added via `os.add_dll_directory` not
being considered, when loading DLLs (see
isl-org#7104 (comment)).
In particular this leads to open3d/tbb12.dll not being found.

Fix this by going back to the default DLL-search mode (`winmode=None`).

In order to still be able to find CUDA-DLLs, that are not part of the
open3d-package, search their directory in PATH and explicitly add it to
the DLL-search-locations for CUDA-builds.
fegorsch pushed a commit to fegorsch/Open3D that referenced this issue Dec 20, 2024
This PR introduced `winmode=0` for loading DLLs during import:
isl-org#5259

This however leads to directories added via `os.add_dll_directory` not
being considered, when loading DLLs (see
isl-org#7104 (comment)).
In particular this leads to open3d/tbb12.dll not being found.

Fix this by going back to the default DLL-search mode (`winmode=None`).

In order to still be able to find CUDA-DLLs, that are not part of the
open3d-package, search their directory in PATH and explicitly add it to
the DLL-search-locations for CUDA-builds.
fegorsch pushed a commit to fegorsch/Open3D that referenced this issue Dec 20, 2024
This PR introduced `winmode=0` for loading DLLs during import:
isl-org#5259

This however leads to directories added via `os.add_dll_directory` not
being considered, when loading DLLs (see
isl-org#7104 (comment)).
In particular this leads to open3d/tbb12.dll not being found.

Fix this by going back to the default DLL-search mode (`winmode=None`).

In order to still be able to find CUDA-DLLs, that are not part of the
open3d-package, search their directory in PATH and explicitly add it to
the DLL-search-locations for CUDA-builds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Not a build issue, this is likely a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant