Pipelines with multiple inputs

[lschr]

I was thinking about implementing support for multiple inputs (and maybe outputs) to pipelines. For example, it would be nice to be able to

@pipeline
def add(a1, a2):
    return a1 + a2

i1 = pims.open("img1.tif")
i2 = pims.open("img2.tif")
i12 = add(i1, i2)

Before I start I wanted to ask how to go about it. A few possibilities, decreasing in elegance (in my opinion) but increasing in API compatibility:

• Extend the Pipeline class (and pipeline decorator)
  • Rearrange __init__ arguments: def __init__(proc_func, *ancestors, propagate_attrs=None). This breaks the API.
  • Keep __init__ arguments in order: def __init__(*args, propagate_attrs=None), where args[-1] is proc_func. This turns propagate_attrs into a keyword-only and proc_func and ancestor into positional arguments but otherwise preserves the API.
  • Preserve the API: def __init__(ancestor, proc_func, propagate_attrs=None, *other_ancestors)
  • Some middle ground between the above
• Create a new class (and decorator). Complete freedom since there is no backward-compatibility to think of.
• Something else I did not think about

Which is the preferred way?

[caspervdw]

@lschr Thanks for this great initiative! I would personally start extending the Pipeline class. I am not very worried about breaking the API at this point, as there is still no PIMS release that depends on it (ref soft-matter/pims#247). So I would say, just go for the most elegant option.

[lschr]

This should be easy enough. The only problem I have come across so far is attribute propagation. A few possibilities:

• Propagate only attributes from the first ancestor
• Propagate attributes from all ancestors as long as there are no name conflicts. If there are conflicts, use the attribute from the first ancestor that has the attribute.
• Return a tuple containing the respective attribute values from all ancestors.
• Do some name mangling to avoid conflicts.

I'd strongly prefer the first or second solution. The third option is cumbersome to use in my opinion; the propagated attribute would need different treatment from the original attribute. The fourth option is messy altogether. Anyone needing non-trivial treatment of attributes can do so by subclassing Pipeline.

Any thoughts?

[nkeim]

Since there are so many options, maybe use a kwarg to specify how this is done, a la pandas’ join()? Defaults to “right” but you could also implement “left.” Then if you discover cases where others are needed you could add them. Nathan On Oct 10, 2017, 6:47 AM -0700, lschr <[email protected]>, wrote: This should be easy enough. The only problem I have come across so far is attribute propagation. A few possibilities: * Propagate only attributes from the first ancestor * Propagate attributes from all ancestors as long as there are no name conflicts. If there are conflicts, use the attribute from the first ancestor that has the attribute. * Return a tuple containing the respective attribute values from all ancestors. * Do some name mangling to avoid conflicts. I'd strongly prefer the first or second solution. The third option is cumbersome to use in my opinion; the propagated attribute would need different treatment from the original attribute. The fourth option is messy altogether. Anyone needing non-trivial treatment of attributes can do so by subclassing Pipeline. Any thoughts? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#29 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AC2NbfTEdRkEF0madYBA9YSkPPbANYBrks5sq3VqgaJpZM4PyYoz>.

[lschr]

#30 is an initial implementation. I suggest moving the discussion over there.

[lschr]

The implementation in #30 is finished now.