Better support for spatial/temporal images: ImageAxes, Unitful or others?

[johnnychen94]

I don't know much about spatial/temporal image processing especially those related to neural sciences so I'm probably not the right person to open this issue. I open this as a milestone to our roadmap 1.x (#977).

The current status of JuliaImages on this is that we have some traits in ImageCore and most supports are implemented in AxisArrays and ImageAxes. Based on the low frequencies of issues and PRs there, I bet people dislike the current status.

I'm not sure if this is already discussed before, depending on where we make the abstraction, there can be two paths to go:

• on pixel/scalar level: take advantage of Unitful.jl and define new Color types (e.g., Gray{Unitful{m}}, UGray{Unitful{m}, :depth})
• on array level: the current ImageAxes abstraction (e.g., AxisArray(1:5, Axis{:width}(1:5)))

TBH I don't know how these would imply so I can't make a decision here yet.

#44 (comment)
The mapping from a physical quantity to a unitless color representation is an important step that should be represented in the types being used. Making everything a color does not help in this respect.

A third path is to make the core functionalities not restricted to Color types. For instance, move core functionality of ImageFiltering to JuliaArrays ArrayFiltering.jl (JuliaImages/ImageFiltering.jl#228). The main difficulty of this path is that Number type and Colorant type are not in the same abstraction level; colors can live in vector spaces while numbers are just scalars. Thus if we don't think with the colorant (vector space) abstraction, there will still be some gaps here and there according to my own experiences when doing ImageDistances.jl

About traits: we definitely need traits to support the semantics, but traits are not panaceas, what we need is valid use cases and functionalities built on top of traits. Thus I personally am conservative to traits that don't contribute to the general API. ndims/axes are good traits because it's defined as the interface of array types, but many of the existing traits in ImageCore and ImageAxes are not.

Some related issues that I know of:

• Shepp Logan Phantom #44 (comment)
• How much should we support time based traits? ImageCore.jl#112
• Better axis support via name traits #908

cc: @JeffFessler @Tokazama @timholy @tknopp

[Tokazama]

My gut feeling here is that MRI needs to just have a float instead of a typed value (e.g., Gray{Unitful{m}}). Even if we can agree on a handful of unit types that are representative of the vast majority of imaging sequences, it seems equally significant to convert those units appropriately in subsequent processing steps. Even if we assumed that MRI processing was completely stagnant in its state of development, the exact same processing pipeline can have very different interpretations. It just doesn't seem pragmatic to refer to this as bunch of different types solely for the sake of bookkeeping.

[timholy]

Unitful eltype is completely orthogonal to named axes with physical units. We want both, and implementing better support for one will have almost zero impact (positive or negative) on the other.

Re axes: I intend to ditch AxisArrays in favor of one of the newer approaches, but I haven't taken the time to investigate yet. AxisArrays is just too unmaintained, and the world has moved on.

Re Unitful eltypes: someone just needs to switch to a Unitful eltype and then see what breaks, and report bugs/add tests/add generalizations. A personal goal is to add a "Photons" eltype for calibrated images. That doesn't seem to be in Unitful, so I don't yet have a concrete plan for it (either add it to Unitful or create a standalone Photons.jl package).