Why is Homebrew so slow?

[xiruizhao]

Non-network operations took one second!

• macOS 11.2.3 with Intel i7 2.2 GHz
• /bin/bash 3.2.57(1)-release (x86_64-apple-darwin20)
• /bin/ruby 2.6.3p62 (2019-04-16 revision 67580) [universal.x86_64-darwin20]

hyperfine '/bin/bash /usr/local/bin/brew help' --warmup 1
# Benchmark #1: /bin/bash /usr/local/bin/brew help
#   Time (mean ± σ):     964.8 ms ±   7.0 ms    [User: 514.7 ms, System: 433.3 ms]
#   Range (min … max):   953.4 ms … 973.6 ms    10 runs
hyperfine '/bin/bash /usr/local/bin/brew list' --warmup 1
# Benchmark #1: /bin/bash /usr/local/bin/brew list
#   Time (mean ± σ):      1.151 s ±  0.031 s    [User: 622.7 ms, System: 509.2 ms]
#   Range (min … max):    1.125 s …  1.210 s    10 runs

Would using POSIX-compliant dash (preinstalled on macOS) improve performance? /usr/local/bin/brew already declares POSIX compliance.

[SMillerDev]

It's busy, that is why it's slow. What kind of real world use-case do you have that needs speed optimization upon this?

[SMillerDev]

Pip is a language package manager, it has different use cases and requirements then brew has as a system package manager.

[justdanyul]

Pip is a language package manager, it has different use cases and requirements then brew has as a system package manager.

Try using apt-get , pacman, etc.

Long story short, brew is handy, but it sure is slow.

[gwerbin]

A good comparison would be MacPorts, which in extreme cases can download a full set of package updates and install one or two of them, in the same time it takes Brew to show info output for a Formula.

Also, the fact that Pip happens to be a language package manager doesn't make much of a difference in my mind. It performs broadly the same functions and its CLI has similar design constraints. And Pip is an interesting example because it's not particularly fast due to the overhead of starting Python.

[MikeMcQuaid]

@xiruizhao > hyperfine 'python3.9 /usr/local/bin/pip list' --warmup 1

Note when Homebrew/brew#17317 is merged: brew list will be faster than pip list.

That's the real answer to "why is Homebrew slow": because more people are interested in complaining about Homebrew being slow than spending their evenings and weekends making it faster (like me).

[MikeMcQuaid]

And when Homebrew/brew#17710 is merged brew help will be faster than pip help, too.

[alebcay]

Would using POSIX-compliant dash (preinstalled on macOS) improve performance?

My guess would be probably not. brew list may be slower than it could be due to how some of the logic is overloaded - it's not used just for listing installed items, but also can filter by formulae only, casks only, unbrewed formulae, versions of a particular formula, pinned formulae only, etc.

Interestingly, from the numbers above, the CPU time for brew shows that System time makes up the majority of User time, while pip shows a System time that is about a third that of the User time.

You can find the code that drives brew list here. If you have a proposal on how to make the brew list command run faster, please kindly open a pull request with the requisite documentation and before/after benchmarks and the team would be more than happy to review and consider the change.

[gwerbin]

As I stated elsewhere, a much better comparison would be another package manager such as MacPorts, Pkgsrc/Pkgin, APT, Zypper, DNF, Pacman, XBPS, etc. Nearly all of these perform near-instant list filtering, info output, and other "offline" operations, and some of them support very complicated logic for filtering results.

Another interesting comparison is Conda, which is also famously slow, but its reimplementation Mamba is generally fast for offline operations such as listing packages, and the main bottleneck there is dependency resolution.

Looking at brew list specifically is maybe missing the forest for the trees a bit. There's something fundamental in the design of Brew that gives it a ton of CLI startup overhead, and generally makes commands slow. Does Brew use the filesystem differently? Does it lack certain layers of caching and recorded manifests that other package managers have? Etc.

[alebcay]

You raise good points regarding looking at the bigger picture. Nobody on the maintainer team that I know of is actively instrumenting/profiling brew holistically on a regular or rigorous basis.

For anyone that is interested in improving brew performance, the suggested course of action is still the same as in my original response. Of course, at a bigger picture scale, any sort of design change is subject to a number of other considerations aside from performance as well.

The brew prof command may be a helpful tool in collecting profiling/performance data. Run brew prof --help for more details.

[xiruizhao]

Ubuntu 20.04 with Intel J5005 CPU @ 1.50GHz

hyperfine 'apt list --installed'
# Benchmark #1: apt list --installed
#   Time (mean ± σ):     470.4 ms ±   4.4 ms    [User: 436.0 ms, System: 34.5 ms]
#   Range (min … max):   462.3 ms … 477.5 ms

[gwerbin]

I always assumed that Brew was slow because Ruby was slow. There are similar issues with CLI tools built in Python and NodeJS, where lots of little inefficiencies in the language runtime compound into really big overhead even for simple tasks. Compare NVM and Rbenv to FNM and Frum; it's night and day.

Apparently Ruby 3.0 is quite a bit faster, and there are also faster (albeit experimental) Ruby implementations that use either JIT compiling or traditional ahead-of-time compiling using Sorbet type annotations. Has anyone experimented with running Brew under those runtimes?

[rdp]

You could profile it to see why it's slow, then propose a fix. Here's one from me: convert code base to crystal (if it's cpu bound that is) ? :)

[Elly-Mandliel-Forter]

Homebrew is a piece of unusable software

[aaomidi]

It's the git fetch. If you're on a slow connection and haven't updated your git repo clone for a while, it's going to take a loooooong time.

[gwerbin]

The brew CLI command is slow to perform even basic non-network operations like brew info, which are nearly instant in other package managers.

[StephenLeafWW]

I noticed recently, while simply downloading a single package, that it opted to update my qemu, as well as my elixir stuff, as well as my erlang, as well as...... well it's updating everything. I know not the reason why it decided that. Maybe I did something wrong but I figured brew install [package] was an operation that wasn't going to be running for 45 min on a business line, 10gb/s down.

[gromgit]

That has been Homebrew policy for a long while now: https://docs.brew.sh/FAQ#why-does-brew-upgrade-formula-or-brew-install-formula-also-upgrade-a-bunch-of-other-stuff

[Loosie94]

Brew might be downloading something large.
Run for example: brew update --debug --verbose
To see its progress.

[fr0z3nfyr]

Well, Homebrew is indeed not among the fastest package managers, especially if any the operation network dependent (even if your corporate leased line is 10Gb/s. That's something the maintainers have to first admit to attempt a fix 😞

Anyway, if someone stumbles across this thread when trying to perform network dependent operations (install, upgrade etc.) and wondering why brew is compiling almost everything, make sure the version of the OS is supported. FAQ on this makes sense.

[cxpreng]

It's amazing how shitty a piece of software may get to be, after a few minutes, when I noticed brew was going nowhere "trying" to install php8.1, becaue stupid brew upgraded to 8.2 without asking me, and while it was downloading a lot of useless and non necessary "dependencies" I was able to download and install MacPorts and finish a php installation with it.
And I say brew was "trying" cause I finally got an error because of the macOS version, all good and working with MacPorts, brew I had it with you, you are going down, now I'm sure removing it is gonna be fun

[xiruizhao]

Compare Ubuntu package libapache2-mod-php8.1 dependencies and Homebrew [email protected] formula dependencies, you can find many common dependencies. MacPorts probably bundles the dependencies in the package, but that is not the standard packaging practice (as adopted by Debian and Fedora).

[cxpreng]

Installing in Debian takes about 20% of the time brew takes, I do it all the time in servers with fewer resources than my MacBook has, and I correct, the error was because brew couldn't find Python which I also installed with brew 😂
Also apt won't try to install dependencies that are already installed as brew does

[gwerbin]

https://docs.brew.sh/FAQ#why-does-brew-upgrade-formula-or-brew-install-formula-also-upgrade-a-bunch-of-other-stuff

[abhi40308]

Just tried to install watchman and its been going on for 30 mins. Keeps updating dependency after dependency, almost feels like I just installed the next mac OS version. Funny to see people normalising this DX, giving all the reasons why this is necessary / unavoidable / configurable.

[cxpreng]

Months after removing this crap I still find orphan files scattered all around my system!! I did mention uninstalling was gonna be fun, and you guys did not disappoint!
You can't even write a decent uninstaller! 🤦🏻‍♂️
Oh the humanity!!

[gwerbin]

@cris-spreng my experience with Homebrew is that I can rm -rf the main directory (eg /opt/homebrew) and that's more or less a clean uninstall.

[dondxniel]

I literally just spent 5 hours trying to install and symlink node and npm (probably because my OS is outdated). And it still failed. After 5 hours.
There has to be a way to work around the speed constraint. Anyone knows how?

[SMillerDev]

Update your OS to a supported version so you can use prebuilt binaries again.

[cxpreng]

What is your OS?

[stuaxo]

On MacOS brew --prefix could do with being faster, I use it in two places in my .zshrc (one as brew --prefix postgresql) and get to watch the title run through, ruby and clang every new terminal.

[gromgit]

brew --prefix <formula> goes through all that to give you a value that you can count on, but if all you want is a reasonable starting point to drill down into a formula's files, then ${HOMEBREW_PREFIX}/opt/<formula> is usually good enough and as fast as it gets. In the rare case when there's nothing there (renamed formula, perhaps), then run brew --prefix <formula> to get the canonical path...and consider caching it somewhere.

[stuaxo]

Cheers, I managed to get rid of those, I didn't know $HOMEBREW_PREFIX was a thing, and replaced my first use with it.

The second one I didn't need as I hadn't really considered all the binaries end up in ${HOMEBREW_PREFIX}/bin as symlinks

Everything is faster now, with the rest of my slowdown not being homebrew at least.

[gromgit]

The second one I didn't need as I hadn't really considered all the binaries end up in ${HOMEBREW_PREFIX}/bin as symlinks

That assumption fails for keg-only formulae (e.g. sqlite). As a general rule, the public binaries for every formula can be found in /opt/homebrew/opt/<formula>/bin.