Homebrew package unable to find the file "ggml-metal.metal"

[mislav]

Hi, reporting something that is not necessarily a bug with this codebase, but a packaging problem. I'm not familiar with this project enough to determine that, so I'm leaving it here:

I was trying out this project by installing it via Homebrew:

brew install whisper-cpp

It seems to successfully transcribe audio, but it's very slow on an M1 MacBook. The output suggests that it wasn't able to use Metal backend:

whisper_backend_init: using Metal backend
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1
ggml_metal_init: picking default device: Apple M1
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: error: could not use bundle path to find ggml-metal.metal, falling back to trying cwd
ggml_metal_init: loading 'ggml-metal.metal'
ggml_metal_init: error: Error Domain=NSCocoaErrorDomain Code=260 "The file “ggml-metal.metal” couldn’t be opened because there is no such file." UserInfo={NSFilePath=ggml-metal.metal, NSURL=ggml-metal.metal -- file:///Users/mislav/Projects/whisper/, NSUnderlyingError=0x60000014c360 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}
whisper_backend_init: ggml_backend_metal_init() failed

The homebrew formula does include this file during installation and its location is share/whisper-cpp/ggml-metal.metal under its installation prefix: https://github.com/Homebrew/homebrew-core/blob/418ec8d5274aa16e99bdbb96dfc9c88f01a3b0ac/Formula/w/whisper-cpp.rb#L29

But I'm guessing at runtime whisper-cpp does not discover this file at this location. It can be helped by setting an environment variable:

GGML_METAL_PATH_RESOURCES="$(brew --prefix whisper-cpp)/share/whisper-cpp" whisper-cpp ...

Now it uses Metal and it seems noticeably faster. I think that ideally, setting GGML_METAL_PATH_RESOURCES should not be necessary as that the metal file should have been auto-discovered, but this might be a problem with how whisper-cpp is packaged within Homebrew. It that seems to be the case, please close this issue with a suggestion of a better location for the file, and I will go ahead and submit a PR to Homebrew. Thanks!

[ggerganov]

The homebrew formula is a bit outdated. It should be updated in a similar way to llama.cpp.rb. When this is done, the Metal library will be embedded in the binary and the env path would not be required. I just haven't had the time to look into this. If you are interested in giving it a try, please go ahead and feel free to ping me with any questions.

[ggerganov]

Ah and it will likely require updating whisper.cpp to the latest v1.7.3. I've currently release a pre-release version: https://github.com/ggerganov/whisper.cpp/releases/tag/v1.7.3-pre and planning to push the official v1.7.3 release soon.

[mislav]

Thanks for chiming in. After whisper.cpp v1.7.3 is out I can try improving the Homebrew formula for whisper-cpp.

BTW, did I open this issue in the wrong repo? I was meaning to open it in https://github.com/ggerganov/whisper.cpp; please feel free to move it there. Thank you!

[ggerganov]

It's ok to have the issue here - no problem.

https://github.com/ggerganov/whisper.cpp/releases/tag/v1.7.3 has just been released.

[mislav]

Here is the Homebrew PR to bump the version: Homebrew/homebrew-core#201851

It installs the "main" example as the whisper-cpp binary. This is the full directory tree of the resulting install:

$ tree /opt/homebrew/Cellar/whisper-cpp/*    
/opt/homebrew/Cellar/whisper-cpp/1.7.3
├── AUTHORS
├── INSTALL_RECEIPT.json
├── LICENSE
├── README.md
├── bin
│   └── whisper-cpp
├── ggml
│   └── src
│       ├── ggml-blas
│       │   └── libggml-blas.dylib
│       ├── ggml-metal
│       │   └── libggml-metal.dylib
│       ├── libggml-base.dylib
│       ├── libggml-cpu.dylib
│       └── libggml.dylib
├── sbom.spdx.json
├── src
│   ├── libwhisper.1.7.3.dylib
│   ├── libwhisper.1.dylib -> libwhisper.1.7.3.dylib
│   └── libwhisper.dylib -> libwhisper.1.dylib
└── test
    ├── for-tests-ggml-tiny.bin
    └── jfk.wav

To figure out the right build incantation was very much trial-and-error on my side, so please correct me if I'm doing something obviously wrong. Notice that the installation now produces dylib files, whereas the build system for v1.5.4 (a simple make invocation, it seems) did not produce any. The compiled main binary depends on these dylibs at runtime. However, the Metal support is successfully embedded, like you promised, so an external ggml-metal.metal file isn't needed anymore. Finally, I keep for-tests-ggml-tiny.bin and jfk.wav there merely for the brew test whisper-cpp functionality. This all seems to work, so I've submitted it to Homebrew in this format.

---

P.S. here is some feedback about the build system from someone who has zero experience with cmake, but who has a solid grasp of the ./configure && make && make install combo from other projects.

(BTW, everything that follows is in the context of the https://github.com/ggerganov/whisper.cpp repo.)

On a fresh download of this project's source code, I've tried this (simplified for brevity):

cmake -B "$BUILD_DIR" -DCMAKE_INSTALL_PREFIX="$INSTALL_DIR" [args...]
cmake --build "$BUILD_DIR" --target main
cmake --install "$BUILD_DIR"

What I expected was that the main binary that I specifically asked for with --target main would be a part of the final installation. But that wasn't the case, as this was the resulting INSTALL_DIR:

/tmp/whisper-install
├── bin
│   └── ggml-metal.metal
├── include
│   ├── ggml-alloc.h
│   ├── ggml-backend.h
│   ├── ggml-blas.h
│   ├── ggml-cann.h
│   ├── ggml-cpu.h
│   ├── ggml-cuda.h
│   ├── ggml-kompute.h
│   ├── ggml-metal.h
│   ├── ggml-opt.h
│   ├── ggml-rpc.h
│   ├── ggml-sycl.h
│   ├── ggml-vulkan.h
│   ├── ggml.h
│   └── whisper.h
└── lib
    ├── cmake
    │   └── whisper
    │       ├── whisper-config.cmake
    │       └── whisper-version.cmake
    ├── libggml-base.dylib
    ├── libggml-blas.dylib
    ├── libggml-cpu.dylib
    ├── libggml-metal.dylib
    ├── libggml.dylib
    ├── libwhisper.1.7.3.dylib
    ├── libwhisper.1.dylib -> libwhisper.1.7.3.dylib
    ├── libwhisper.dylib -> libwhisper.1.dylib
    └── pkgconfig
        └── whisper.pc

7 directories, 26 files

It looks like the product of the install command are header files and dylibs, which is fine for other programs to use when linking to the whisper library, but I'm mainly interested in the main executable (which I now understand is only an example program for the libwhisper library).

The main program is still in the build directory, but is configured to load dylibs not from INSTALL_DIR, but from BUILD_DIR:

$ DYLD_PRINT_LIBRARIES=1 DYLD_PRINT_LIBRARIES_POST_LAUNCH=1 DYLD_PRINT_RPATHS=1 /tmp/whisper-build/bin/main 2>&1 | grep whisper-build 
dyld[30327]: <E1FE1DB8-B103-34C9-BB31-795823E3D463> /private/tmp/whisper-build/bin/main
dyld[30327]: <176162D3-B7E0-3B15-903C-E64C517C2547> /private/tmp/whisper-build/src/libwhisper.1.7.3.dylib
dyld[30327]: <17593FD3-96D1-36AC-8438-BE1EF34278F1> /private/tmp/whisper-build/ggml/src/libggml.dylib
dyld[30327]: <F743383C-518A-3DF5-9D3A-6B6916A8DBD2> /private/tmp/whisper-build/ggml/src/libggml-cpu.dylib
dyld[30327]: <1D570838-3F60-3400-BCC6-CA8D135BEC4A> /private/tmp/whisper-build/ggml/src/ggml-blas/libggml-blas.dylib
dyld[30327]: <E0996A4F-E71C-3072-8E52-E5E66D2F1D07> /private/tmp/whisper-build/ggml/src/ggml-metal/libggml-metal.dylib
dyld[30327]: <47234C84-317E-3276-9871-427F0A46CF61> /private/tmp/whisper-build/ggml/src/libggml-base.dylib
usage: /tmp/whisper-build/bin/main [options] file0.wav file1.wav ...

I find this to be a problem, because if I move main to a desired location and then rm -rf $BUILD_DIR as usual part of post-installation cleanup, main cannot find required libraries anymore. I would expect that it searches for them in INSTALL_DIR instead. Could this be an oversight?

And finally, a small feature request: I wonder if my system already has libggml dylibs, the build system for this repo could be instructed to link to those at runtime instead of building fresh ones? For example, I can use Homebrew to install llama.cpp:

$ brew install llama.cpp && ls -l /opt/homebrew/lib/libggml*
...
lrwxr-xr-x  1 mislav  admin  47 Dec 20 01:17 /opt/homebrew/lib/libggml-base.dylib -> ../Cellar/llama.cpp/4367/lib/libggml-base.dylib
lrwxr-xr-x  1 mislav  admin  47 Dec 20 01:17 /opt/homebrew/lib/libggml-blas.dylib -> ../Cellar/llama.cpp/4367/lib/libggml-blas.dylib
lrwxr-xr-x  1 mislav  admin  46 Dec 20 01:17 /opt/homebrew/lib/libggml-cpu.dylib -> ../Cellar/llama.cpp/4367/lib/libggml-cpu.dylib
lrwxr-xr-x  1 mislav  admin  48 Dec 20 01:17 /opt/homebrew/lib/libggml-metal.dylib -> ../Cellar/llama.cpp/4367/lib/libggml-metal.dylib
lrwxr-xr-x  1 mislav  admin  42 Dec 20 01:17 /opt/homebrew/lib/libggml.dylib -> ../Cellar/llama.cpp/4367/lib/libggml.dylib

This gives me libggml*.dylib that are symlinked into /opt/homebrew/lib and presumeably available for other programs to link to. I imagine that at build time, the future whisper-cpp Homebrew formula could be made to depend on llama.cpp and then reuse these libraries instead of having to build them? I could not figure out how to reuse existing libggml*.dylib with the current build system.

Please forgive me if I'm not making sense; I just have a basic understanding of dylibs in general, which is that they're needed at runtime and that they are discovered in a list of locations that is dictated by the program's embedded configuration (and possibly some environment variables).

And finally, thank you for this awesome project. It certainly solved my problem (a personal tool to grep podcasts I've listened to) and I hope this can help you improve the build system for everyone who comes next. 💫

[ggerganov]

Thank you for the very detailed explanation! I will attempt to address the install and link problems in ggerganov/whisper.cpp#2648.

Regarding the shared libggml - not sure we can do it at this point because llama.cpp and whisper.cpp often depend on a different version of the library. Also having one package depend on the other does not seem ideal. Maybe in the future ggml can become a separate package that installs the library and the other packages (such as whisper.cpp and llama.cpp) depend on it.

[ggerganov]

Btw, I only now realize that the libggml binaries from the llama.cpp package are symlinked into /opt/homebrew/lib. I wonder if this is a good idea due to the explanation in my previous comment.

[mislav]

Btw, I only now realize that the libggml binaries from the llama.cpp package are symlinked into /opt/homebrew/lib. I wonder if this is a good idea due to the explanation in my previous comment.

Ah, I thought that was intentional for the llama.cpp Homebrew formula. The way Homebrew works is that if the install block of the formula places any headers into #{prefix}/include or libraries in #{prefix}/lib, those will automatically get symlinked into /opt/homebrew. I don't know if there is a way to disable that apart from marking a formula as "keg only", in which case nothing ever gets symlinked, not even the binaries in bin. If you don't want libraries from llama.cpp to get exposed, I think the best way would be for the installer to place them in a place different than #{prefix}/lib.

[ggerganov]

Yeah, so as far as I understand, the correct way to distribute both the llama.cpp and whisper.cpp packages on Homebrew would be to have a common package ggml that would deploy the shared libggml* binaries to be used by both projects (and potentially others in the future). At the moment, there would be a clash of the libggml binaries from the 2 projects and I am not sure if Homebrew has some mechanism to resolve this.

So I am thinking that there are 2 possible solutions:

• We make a ggml Homebrew package
• Find some way in which both llama.cpp and whisper.cpp ship their own versions of libggml and link to those respectively

In the long term, the first option is what we should aim for, but I am not sure if we are ready for it at the moment. Maybe there is some way to make the second option work somehow?

I will cc @slaren as he has very good knowledge about the build system and might have some additional insights on the topic.

[slaren]

I am looking into packaging now, I am just not sure yet what's the best way to handle it. Generally I think we should have separate packages for the libraries that other developers can also use, but that will probably require at least us to start versioning ggml, which I am not sure that would be compatible with the way we currently have a duplicate of ggml in every repository.

Some notes that may fix the immediate issues:

• Running cmake --install should clear the dynamic library search path
• You can also change the search path by setting CMAKE_INSTALL_RPATH
• If there is a single executable, which I believe is the case for the whisper.cpp brew package, then a static build should also work. This can be done by disabling BUILD_SHARED_LIBS. In the long term we should avoid this in favor of separate packages for the libraries and backends, but it should solve the immediate problem.

[mislav]

• If there is a single executable, which I believe is the case for the whisper.cpp brew package, then a static build should also work. This can be done by disabling BUILD_SHARED_LIBS

@slaren Thanks for that tip; I was able to simplify the Homebrew formula using that flag. https://github.com/mislav/homebrew-core/blob/e4dbcb99e2ce4ab6cbd167126578afe2fa795b9f/Formula/w/whisper-cpp.rb#L28-L42

It was apparently needed too, because with shared libs, the Homebrew formula seemed have been failing on Linux due to a dylib not being found: https://github.com/Homebrew/homebrew-core/actions/runs/12422801076/job/34685263836. I've only ever tested the formula on macOS, so I'm not sure what is different on Linux. By doing BUILD_SHARED_LIBS=OFF, I was able to avoid having to debug the Linux failure and just end up with a single self-contained binary. 👍

[ggerganov]

I pushed a v1.7.4-pre-0 tag that renames the binaries for consistency with llama.cpp. For example, the main binary now is called whisper-cli.

I also fixed the install rule for the binary so it should appear in the bin folder.

You can also change the search path by setting CMAKE_INSTALL_RPATH

@mislav What if we keep the shared libraries build ON and try to set the RPATH relative to the binaries, such as: ../lib. Not sure if this is correct, but my idea is to try to make the whisper-cli first look in the local lib folder for libggml and libwhisper. This way it would not matter which version of libggml is symlinked in /opt/homebrew/lib.

[mislav]

Update: Homebrew v1.7.3 formula has been merged and the original issue in this thread fixed, so as far as I'm concerned, this can be closed and followup work tracked in separate issues. 🎉

What if we keep the shared libraries build ON and try to set the RPATH relative to the binaries, such as: ../lib

I'm not familiar with RPATH at all, but it sounds like that might work. However, now that we have a self-contained binary in Homebrew, what incentive would we have to break out dylibs into separate files anymore? As previously concluded in this thread, if both llama.cpp and whisper-cpp tried to publish libggml dylibs, Homebrew would deem these formulae incompatible with each other due to filename clash.

But in any case, feel free to share suggestions here and if I can help apply them to future versions of the Homebrew formula, I will do my best. Note that am not a machine learning developer of any sort, so I am not a target audience for libggml libraries and cannot weigh in on how they should be used. However, I support the previously expressed idea that in the future where libggml is more "stable" and is versioned on its own, both llama.ccp and whisper-cpp could be made to depend on it externally.

I pushed a v1.7.4-pre-0 tag that renames the binaries for consistency with llama.cpp. For example, the main binary now is called whisper-cli.

Great, thank you! BTW, in future Homebrew formula upgrades for this, should the Homebrew-installed binary be continued to be called whisper-cpp (to match the formula name) or renamed to whisper-cli (to match what the installer generates)?

[ggerganov]

However, now that we have a self-contained binary in Homebrew, what incentive would we have to break out dylibs into separate files anymore?

The static builds are not ideal because ggml has a mechanism to load a dynamic backend library that is optimized for the specific user CPU that is going to run whisper.cpp. This would allow to take advantage of all the CPU features and get the optimal performance and we won't be able to use this mechanism with BUILD_SHARED_LIBS=OFF.

However, I support the previously expressed idea that in the future where libggml is more "stable" and is versioned on its own, both llama.ccp and whisper-cpp could be made to depend on it externally.

Yes, this seems like the ultimate solution.

BTW, in future Homebrew formula upgrades for this, should the Homebrew-installed binary be continued to be called whisper-cpp (to match the formula name) or renamed to whisper-cli (to match what the installer generates)?

For consistency with the llama.cpp package where we have llama-cli, llama-server, etc, it would be better to call it whisper-cli.

[mislav]

Okay, I am preparing the change to the Homebrew formula to adjust it to the latest build system in this repo. I've reenabled BUILD_SHARED_LIBS and this is what I got so far:

    system "cmake", "-S", ".", "-B", "build", *args, *std_cmake_args
    if head?
      system "cmake", "--build", "build"
      system "cmake", "--install", "build"
      (bin/"whisper-cpp").write <<~EOS
      #!/bin/bash
      here="${BASH_SOURCE[0]}"
      echo "${BASH_SOURCE[0]}: warning: whisper-cpp is deprecated. Use whisper-cli instead."
      exec "$(dirname "$here")/whisper-cli" "$@"
      EOS
      (bin/"whisper-cpp").chmod 0755
    else
      # the "main" target is our whisper-cpp binary
      system "cmake", "--build", "build", "--target", "main"
      bin.install "build/bin/main" => "whisper-cpp"
    end

As you can see, I've added a whisper-cpp shim that prints the warning to use whisper-cli and executes whisper-cli in turn.

However, the Homebrew linking step now fails because I also have llama.cpp installed:

Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /opt/homebrew
Could not symlink include/ggml-alloc.h
Target /opt/homebrew/include/ggml-alloc.h
is a symlink belonging to llama.cpp. You can unlink it:
  brew unlink llama.cpp

To force the link and overwrite all conflicting files:
  brew link --overwrite whisper-cpp

To list all files that would be deleted:
  brew link --overwrite whisper-cpp --dry-run

Possible conflicting files are:
/opt/homebrew/include/ggml-alloc.h -> /opt/homebrew/Cellar/llama.cpp/4375/include/ggml-alloc.h
/opt/homebrew/include/ggml-backend.h -> /opt/homebrew/Cellar/llama.cpp/4375/include/ggml-backend.h
/opt/homebrew/include/ggml-blas.h -> /opt/homebrew/Cellar/llama.cpp/4375/include/ggml-blas.h
/opt/homebrew/include/ggml-cann.h -> /opt/homebrew/Cellar/llama.cpp/4375/include/ggml-cann.h
/opt/homebrew/include/ggml-cpu.h -> /opt/homebrew/Cellar/llama.cpp/4375/include/ggml-cpu.h
/opt/homebrew/include/ggml-cuda.h -> /opt/homebrew/Cellar/llama.cpp/4375/include/ggml-cuda.h
/opt/homebrew/include/ggml-kompute.h -> /opt/homebrew/Cellar/llama.cpp/4375/include/ggml-kompute.h
/opt/homebrew/include/ggml-metal.h -> /opt/homebrew/Cellar/llama.cpp/4375/include/ggml-metal.h
/opt/homebrew/include/ggml-opt.h -> /opt/homebrew/Cellar/llama.cpp/4375/include/ggml-opt.h
/opt/homebrew/include/ggml-rpc.h -> /opt/homebrew/Cellar/llama.cpp/4375/include/ggml-rpc.h
/opt/homebrew/include/ggml-sycl.h -> /opt/homebrew/Cellar/llama.cpp/4375/include/ggml-sycl.h
/opt/homebrew/include/ggml-vulkan.h -> /opt/homebrew/Cellar/llama.cpp/4375/include/ggml-vulkan.h
/opt/homebrew/include/ggml.h -> /opt/homebrew/Cellar/llama.cpp/4375/include/ggml.h
/opt/homebrew/lib/libggml-base.dylib -> /opt/homebrew/Cellar/llama.cpp/4375/lib/libggml-base.dylib
/opt/homebrew/lib/libggml-blas.dylib -> /opt/homebrew/Cellar/llama.cpp/4375/lib/libggml-blas.dylib
/opt/homebrew/lib/libggml-cpu.dylib -> /opt/homebrew/Cellar/llama.cpp/4375/lib/libggml-cpu.dylib
/opt/homebrew/lib/libggml-metal.dylib -> /opt/homebrew/Cellar/llama.cpp/4375/lib/libggml-metal.dylib
/opt/homebrew/lib/libggml.dylib -> /opt/homebrew/Cellar/llama.cpp/4375/lib/libggml.dylib

Could I pass options to the build system to install header files and libraries to locations different to include and lib?

And finally, there is a .metal file that ended up in the bin directory of the install:

$ ls -l `brew --prefix whisper-cpp`/bin             
total 2640
-r--r--r--  1 mislav  staff  263028 Dec 22 23:14 ggml-metal.metal
-r-xr-xr-x  1 mislav  admin   36512 Dec 22 23:17 whisper-bench
-r-xr-xr-x  1 mislav  admin  334824 Dec 22 23:17 whisper-cli
-r-xr-xr-x  1 mislav  admin     167 Dec 22 23:17 whisper-cpp
-r-xr-xr-x  1 mislav  admin  707808 Dec 22 23:17 whisper-server

The file isn't even executable, so it should not end up in bin I think.

[slaren]

I don't see a good way to deal with the library conflicts until we have versioned ggml. Maybe you could install the libraries into a different location, and change the shared library search path with CMAKE_INSTALL_RPATH so that the executable files can still find them. I think that's going to depend on what restrictions brew puts on the install location of the files.

I don't think ggml-metal.metal should installed at all in the default build, since with GGML_METAL_EMBED_LIBRARY enabled (the default) it is bundled directly into the library file. That's something that we need to fix.

[ggerganov]

I don't think ggml-metal.metal should installed at all in the default build, since with GGML_METAL_EMBED_LIBRARY enabled (the default) it is bundled directly into the library file. That's something that we need to fix.

I opened #1054 to fix this. Not sure what should we do when GGML_METAL_EMBED_LIBRARY=OFF, but for now I think this path is not very important since embedding the Metal code always seems to be the better option.

[mislav]

@slaren To enable dylibs but to place them in a different location than "lib", I've tried this:

1. -DCMAKE_INSTALL_LIBDIR=libinternal
2. -DCMAKE_INSTALL_RPATH=@loader_path/../libinternal - this syntax is apparently macOS-specific but I'll figure out Linux later

The dylibs are now installed to "libinternal" but the whisper-cli executable still tries to find them in "lib":

dyld[46804]: Library not loaded: @rpath/libwhisper.1.dylib
  Referenced from: <01DC58FA-4DDA-3D36-A1EB-33F73C34DC90> /opt/homebrew/Cellar/whisper-cpp/HEAD-ed09075/bin/whisper-cli
  Reason: tried: '/opt/homebrew/Cellar/whisper-cpp/HEAD-ed09075/lib/libwhisper.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/Cellar/whisper-cpp/HEAD-ed09075/lib/libwhisper.1.dylib' (no such file), '/opt/homebrew/Cellar/whisper-cpp/HEAD-ed09075/lib/libwhisper.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/Cellar/whisper-cpp/HEAD-ed09075/lib/libwhisper.1.dylib' (no such file)
Error: whisper-cpp: failed

Do you know what I might be missing? I have no previous experience with RPATH.

[slaren]

I tried this with llama.cpp and worked as expected:

cmake -B build -DCMAKE_INSTALL_LIBDIR=libinternal -DCMAKE_INSTALL_RPATH=@loader_path/../libinternal 
cmake --build build
cmake --install build --prefix "install_test"
install_test/bin/llama-cli

I am not sure if whisper.cpp does something different, but I think it should work.

[ggerganov]

For some reason, the install RPATH of whisper.cpp targets was hardcoded to lib:

ggerganov/whisper.cpp@53c9a3a

I've tagged the latest version with v1.7.4-pre-1. This should probably work now.

[mislav]

Thanks; the Homebrew PR is up: Homebrew/homebrew-core#202270