IDE support for formula development

[LecrisUT]

From my experience of getting my IDE to work with formula development, I've found that there are a lot of missing environment variables in order to replicate the build process of the formula. Particularly the difference between what's available in brew sh and brew install -s, e.g. CMAKE_PREFIX_PATH=/usr/local in the former, while the latter is added by various other packages depending on the formula: /usr/local/opt/libffi:/usr/local/opt/[email protected]:/usr/local/opt/readline:/usr/local/opt/sqlite:/usr/local/opt/tcl-tk:/usr/local/opt/icu4c:/usr/local.

So I would like to start some discussion on how to best implement these missing variables, either manually or via a command. Specifically:

• brew sh: does it scrub all of the initial environment variables and generate them from scratch? E.g. if we run it from an app some variables such as DYLD_LIBRARY_PATH are wiped by Mac, so does brew sh depend on any of such initial variables?
• Implementation of brew sh {formula}: similar to brew sh and brew install -s --interactive, but simply read the formula, and initialize a shell environment depending on the depends_on similar to how the brew install -s environment is constructed. And compared to brew install -s --interactive it should not download, create tmp folders, etc. so it can better be used in IDEs.

[SMillerDev]

brew sh: does it scrub all of the initial environment variables and generate them from scratch? E.g. if we run it from an app some variables such as DYLD_LIBRARY_PATH are wiped by Mac, so does brew sh depend on any of such initial variables?

It filters almost all variables. What it keeps depends on the formula and the environment you're running things in.

Implementation of brew sh {formula}: similar to brew sh and brew install -s --interactive, but simply read the formula, and initialize a shell environment depending on the depends_on similar to how the brew install -s environment is constructed. And compared to brew install -s --interactive it should not download, create tmp folders, etc. so it can better be used in IDEs.

I'm sure that if you write the code people will happily review it and merge it if it's up to our standards. But I have to say that in 5 years of contributions to homebrew with simple text editors like nano or sublime text I don't think there has been a situation where a more advanced IDE would have helped me write a formula. Most things in formulae aren't complex functions after all, they're more like system "make", which upstream documentation will help you with much more than any IDE ever will.

[LecrisUT]

But I have to say that in 5 years of contributions to homebrew with simple text editors like nano or sublime text I don't think there has been a situation where a more advanced IDE would have helped me write a formula

Yes, I didn't have a need in the past either, but the issues with webkit-gtk showed me the nieche usecases. For example the error that I need to solve now are:

../Source/JavaScriptCore/jsc.cpp:123:10: fatal error: 'wtf/spi/darwin/ProcessMemoryFootprint.h' file not found
#include <wtf/spi/darwin/ProcessMemoryFootprint.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
../Source/WebKit/Platform/SharedMemory.h:79:38: error: use of non-static data member 'm_size' of 'SharedMemory' from nested type 'Handle'
        size_t size() const { return m_size; }
                                     ^~~~~~
1 error generated.

First one is a cmake issue which can be patched by adding the appropriate header, and the latter is something more deeply rooted in the code. These will require upstream patches, but in-order to find the patches needed we need some development environment to work on.

I'm sure that if you write the code people will happily review it and merge it if it's up to our standards.
Could you help point me to the shell initialization code when brew install is initialized, and the brew sh command? I'm not proficient in ruby, but I should be able to get a draft working and discuss it later.

It filters almost all variables. What it keeps depends on the formula and the environment you're running things in.

So far, in brew sh I have to change PKG_CONFIG_PATH because some system ones are not included the same way as brew install, and on the IDE even if I include them, so many variables are inconsistent that it is not able to import homebrew libraries properly, e.g.:

In file included from ../Source/JavaScriptCore/API/glib/JSCOptions.cpp:24:
/usr/local/Cellar/glib/2.68.0/include/glib-2.0/glib/gi18n-lib.h:23:10: fatal error: 'libintl.h' file not found
#include <libintl.h>
         ^~~~~~~~~~~
1 error generated.

Those will be harder to track down.

[SMillerDev]

The issues you're having with webkit-gtk just sound like you should try and build it outside of homebrew first. Only after it works generally on macOS you should try and package it in homebrew. Otherwise you'll spend a lot of time trying to figure where each issue comes from.

[LecrisUT]

Well the root problem with building in the IDE is the same, MacOS purges some essential environment variables so I am not able to pick up the necessary packages in order to be able to build, independent of brew sh or any environments. I don't know if this is a specific IDE issue, but I had similar issues with Matlab and intel's MKL, and I've managed so solve it by running matlab from terminal. The same trick does not work with CLion, probably because the shell for cmake is different than the IDE's. I will need to test some more on this side to see what would be the best way to setup.

[SMillerDev]

In intellij projects you can turn of the overrides they add to the terminal in settings.

[LecrisUT]

I'm afraid I can't find those settings on CLion 2020.3. In environment there is an option to include system environment variables which is enabled. Homebrew and InteliJ variables are included in that list, and for example lmod is well configured, but some might be sanitized by default by MacOS (BigSur to be precise). Other more enigmatic options are shell integration and activate virtualenv.