Feature: Apt install command to install all packages at once

[SubaruArai]

See: #940

This should make the installation faster.
If this is successful, we might need to think about other package managers as well.

[cottsay]

The original reason that rosdep behaves in this way is to ensure that packages will still get installed on a best-effort basis despite an issue with one of them.

If the install operation attempts to install 101 packages and one of them doesn't actually exist, the other 100 packages should still be installed prior to exiting. Some package managers (like dnf) support this behavior, so rosdep invokes the installation in a single command. At the time when apt support was implemented, there didn't appear to be a way to achieve that behavior.

[SubaruArai]

The original reason that rosdep behaves in this way is to ensure that packages will still get installed on a best-effort basis despite an issue with one of them.
...
At the time when apt support was implemented, there didn't appear to be a way to achieve that behavior.

Interesting, thanks for the input!
Although I'm not really sure if I like this decision, I can get it.
(I personally prefer all or nothing (atomic) - which isn't achievable since we're using multiple package managers)

Anyways, we should be able to add such a feature with something like this.

I guess the real question will be - do we want to preserve this behavior, or speed-up the installation instead (especially in CI/CD systems)?
According to @tonynajjar, a 24 apt package + 1 pip package installation was reduced from 58s to 28s - which I think is a big deal in the context of CI systems. I also prefer failing fast (it looks like apt-get fails at the lookup stage) rather than figuring out one package went wrong after a while.

@cottsay What do you say? Keep the best-effort behavior, or ditch it for faster execution?

[tonynajjar]

I would really like to keep both features if possible. In the package.xml I also define dependencies on custom packages that are not released, for the purpose of colcon building. So when running rosdep, it fails to fetch these custom packages but still continues with the rest.
As for faster rosdep, it's obvious why we would want that - imagine all the CI time you would save in total across the world 😬

[tonynajjar]

@cottsay did you have the chance to think about it?

[cottsay]

Keep the best-effort behavior, or ditch it for faster execution?

I do not believe that changing the existing best-effort behavior is a good idea, and I can't see myself approving a PR that changes that only to improve performance.

A change to a single-invocation that preserves the established behavior and keeps apt aligned with the other common package managers supported by rosdep would certainly get my attention.

[tonynajjar]

There is already the flag -r Continue installing despite errors, can't we apply the "one-shot" approach only if this flag is on?

[cottsay]

FWIW, that would still change behavior that was established over a decade ago. I don't feel particularly strong one way or the other - maybe you might be able to convince other maintainers though 😅

Have either of you looked at how we might get apt to do what we want? 30 seconds of googling led me to a --ignore-missing flag that might help, though I haven't tried it.

To be clear, rosdep won't (shouldn't) try to install packages that are already installed. The established behavior mostly revolves around incorrect rules. It's pretty common that we'll have a ubuntu: [foo] rule where foo was available in older Ubuntu versions and then isn't available in a newer one. This behaves differently from an explicitly unavailable rule, where -r allows you to continue the rosdep operation despite lacking resolution for one or more keys.

[SubaruArai]

It's been a while and I can't recall what I tried, but I remember that there wasn't a silver bullet - including the --ignore-missing flag.

[130s]

I reviewed the code. The current state of this PR would change rosdep's behavior as already discussed in the review comments section. I +1 for not changing the default behavior, but also personally +1 for adding an optional feature to faster installation.

I'm not a maintainer.