[get_depends] when a dependency is missing, 1st run returns no pkgs, but 2nd run returns some pkgs (only some).

[130s]

I call this is a bug.

Problem description

In an unmerged-closed PR #129 (comment), I covered too many issues that made a review harder. So I decided to split into multiple PRs and tickets.

As per subject, when a defined dependency is not available in the paths in ROS_PACKAGE_PATH on the platform rospkg runs on, get_depends's behavior becomes counter-intuitive. Its 1st run and 2nd (or more) run behaves differently as following:

• 1. 1st invocation raises ResourceNotFound without returning any depended pkgs. The current error message does not provide enough info about the situation.
     • Returning an exception be ok.
     • A confusing situation [get_depends] ResourceNotFound exception occurs when "build_depend"-ed pkgs not found #131 can happen.
• 2. 2nd invocation returns a list of depended pkgs, on the contrary to the 1st run. But depending on the situation, the returned pkgs can be only a subset, although in this case the command doesn't return an error.

Example reproducing the problem

Example of 1st invocation

(This example uses OSRF Docker image but involves an adhoc sequence of commands in an attempt to realize the problematic situation. While that set of commands are unusual, the situation it realizes can happen.)

$ docker pull ros:melodic-ros-core
melodic-ros-core: Pulling from library/ros
:
Digest: sha256:cd604a70d3f5df3201a7103338fb3082cf6fe36f1a792498d420bedab7146ea5
Status: Downloaded newer image for ros:melodic-ros-core

$ docker run -it ros:melodic-ros-core /bin/bash
root@d5a64a312993:/# apt-get update && apt-get install -y ipython

root@d5a64a312993:/# apt-get purge ros-melodic-roslang && apt-get autoremove
root@d5a64a312993:/# apt-get install ros-melodic-roscpp ros-melodic-rospack
root@d5a64a312993:/# rospack profile && rospack find roslang
[rospack] Error: package 'roslang' not found
root@d5a64a312993:/# rospack find roscpp
/opt/ros/melodic/share/roscpp

root@d5a64a312993:/# ipython
In [1]: import rospkg
In [2]: rp = rospkg.RosPack()
In [3]: rp.get_depends("roscpp")
---------------------------------------------------------------------------
ResourceNotFound                          Traceback (most recent call last)
<ipython-input-3-b793dd583b6b> in <module>()
----> 1 rp.get_depends("roscpp")

/usr/lib/python2.7/dist-packages/rospkg/rospack.pyc in get_depends(self, name, implicit)
    238
    239             for p in names:
--> 240                 s.update(self.get_depends(p, implicit))
    241             # add in our own deps
    242             s.update(names)

/usr/lib/python2.7/dist-packages/rospkg/rospack.pyc in get_depends(self, name, implicit)
    238
    239             for p in names:
--> 240                 s.update(self.get_depends(p, implicit))
    241             # add in our own deps
    242             s.update(names)

/usr/lib/python2.7/dist-packages/rospkg/rospack.pyc in get_depends(self, name, implicit)
    232
    233             # take the union of all dependencies
--> 234             names = [p.name for p in self.get_manifest(name).depends]
    235
    236             # assign key before recursive call to prevent infinite case

/usr/lib/python2.7/dist-packages/rospkg/rospack.pyc in get_manifest(self, name)
    165             return self._manifests[name]
    166         else:
--> 167             return self._load_manifest(name)
    168
    169     def _update_location_cache(self):

/usr/lib/python2.7/dist-packages/rospkg/rospack.pyc in _load_manifest(self, name)
    209         :raises: :exc:`ResourceNotFound`
    210         """
    --> 211         retval = self._manifests[name] = parse_manifest_file(self.get_path(name), self._manifest_name, rospack=self)
        212         return retval
    213

/usr/lib/python2.7/dist-packages/rospkg/rospack.pyc in get_path(self, name)
    201         self._update_location_cache()
        202         if name not in self._location_cache:
--> 203             raise ResourceNotFound(name, ros_paths=self._ros_paths)
    204         else:
        205             return self._location_cache[name]

ResourceNotFound: rosunit
ROS path [0]=/opt/ros/melodic/share/ros
ROS path [1]=/opt/ros/melodic/share

The situation of missing this particular pkg in the example (rosunit) is legitimately possible, due to the current design (of buildfarm? I'm not sure) explained in #131 (comment). But with the error message users have no idea about:

• whether the given package (roscpp here) has a list of dependency or not (this may sound absurd as a package not depending on anything is unrealistic).
• what ResourceNotFound: rosunit means (It is actually one of the packages defined in the dependency chain but not available on the platform).
• what to do if there's a way to fix the error.

Example of 2nd invocation

In [4]: rp.get_depends("roscpp")
Out[4]:
{'catkin',
 'cpp_common',
 'gencpp',
 'geneus',
 'genlisp',
 'genmsg',
 'gennodejs',
 'genpy',
 'message_generation',
 'message_runtime',
 'roscpp_serialization',
 'roscpp_traits',
 'rostime',
 'std_msgs'}

2nd invocation returns a list. There's some issues here:

• This is inconsistent combined with the 1st invocation behavior.
• This list of packages looks like only a subset of the entire dependency, e.g. rosconsole that is directly run/exec-depended by roscpp is missing, but the command did not return error.

Expected behavior

I'd suggest:

• Error message should explain why the error occurs.
• Command invocation behavior should be consistent, regardless if error occurs or not.

[dirk-thomas]

I have brought the same comment up in a different ticket: #162 (comment)

You currently have five PRs pending in this repository:

• Add "get_licenses" method #133
• Fix inconsistent return value of get_rosdeps #141
• WIP: Add license introspection feature. This handles Ubuntu system pkg #143
• Fix get_depends returning 1st run) none then on 2nd run) fewer pkgs #144
• WIP: Add license introspection feature #162

Please comment on each of them with the current status and what your plan is to proceed on these. Otherwise I will likely go ahead and close them in the near future due to no follow ups.

[130s]

If inactive open PRs matter for you I have no problem in having them closed. I'm willing to address review comments and improve whenever I have time, so will reopen one by one with update.

[dirk-thomas]

If you are still planning to work on them in the future I am fine keeping them open. Since some of them are very old I thought I would ask. Looking forward for updates then.

[dirk-thomas]

Due to the long time of inactivity I will go ahead and close the ticket. Please feel free to comment with updates and it can be reopened.

[130s]

I filed this issue again #214 as I still see the same issue happens and I didn't see a way to re-open this ticket.

I think this ticket got closed because (after reading #142 (comment)) I might have given a wrong impression that I meant this ticket to be a feature suggestion (with the streak of PRs I had opened back then)? In fact this ticket is a bug report.