Make ExternalTool exportable

[lilatomic]

This MR wires ExternalTools into the export machinery.

It exposes them under a separate cli arg --bin. Although it uses some of the same machinery as --resolve, there are several differences that I think support a separate flag (and some separate internal):

1. ExternalTools don't have a resolve. They therefore must not show up for generating lockfiles. We have to implement this separation interally, so we should probably surface that.
2. They would be invoked directly (instead of source dist/export/.../activate)
3. We can put them all in a folder "dist/export/bin" so people can get them all

Closes #21251

Bikeshedding:

• Do we like --bin (yes)
• Is putting all the bins in the same dir what we want (yes, we want a "bin" folder with all the binaries. But we have to put each binary's digest in its own folder to prevent collisions of supporting files)
• [-] Do we want to make all TemplatedExternalTools exportable? We could extend Subsystem.rules and add the UnionRule there (deferred from this MR)

[huonw]

This is great! Thanks for taking it on

Do we like --bin

Fine by me!

Is putting all the bins in the same dir what we want

I wonder if we don't, due to two key downsides of putting them all in the same directory:

• this means that a tool that's more than just a single binary (e.g. READMEs, supporting data files) end up piled into the bin/ directory adjacent to all the other binaries
• binaries and/or their supporting files can have names that overlap, e.g. hypothetically someone could be using both a helm subsystem and migrating to their own fork of the plugin helm2 or something, and they both might have a helm binary. Or just overlapping on supporting files.

As such, I wonder if we want to stick the Pants option scope in the directory hierarchy as a definitely unique disambiguation, e.g. dist/export/bin/helm/helm or dist/export/helm/bin/helm.

[huonw]

Suggested change

	exporatbles = ExportableTool.filter_for_subclasses(
	exportables = ExportableTool.filter_for_subclasses(

[lilatomic]

I wonder if we don't, due to two key downsides of putting them all in the same directory

Both of these are valid.

There's also a parameter "generate_exe" which points to the actual executable. We could export everything to separate dirs and then symlink the executable itself into a bin dir

[benjyw]

Looks like a very sensible solution to me, and --bin seems fine to me. I would put them in different dirs, but the symlinking is a good idea so that it's easy to access them via a single PATH entry.

[lilatomic]

do we want to make all TemplatedExternalTools exportable? We could extend Subsystem.rules and add the UnionRule there

[huonw]

We've just branched for 2.23, so merging this pull request now will come out in 2.24, please move the release notes updates to docs/notes/2.24.x.md. Thank you!

[huonw]

I think this is really nice, thanks for working on it

[huonw]

This seems fine (i.e. I don't think we need to block merging this, but could instead refactor later), but did you consider the option of creating a digest with symlinks in it and including that in the "main" workspace.write_digest(...) call? Rather than shell out to symlinking.

[lilatomic]

yeah good point. I lifted the idea from the python backend's export implementation, which uses a PostProcessingCommand to symlink the venv in.
It probably makes more sense to collect all the symlinks and write them out, though. I'll see how hard that is

[lilatomic]

yeah that's a lot simpler, thanks for the suggestion! I've kept it as a separate write_digest to keep the locality of the processing, there's a lot going on.

[lilatomic]

also the symlinks are absolute symlinks, relative symlinks complained about the symlink not being in the subtree. Should be fine?