Treatment of NULLs in ivo_epoch_prop

[mbtaylor]

The description of ivo_epoch_prop_pos, which returns a propagated (ra,dec) position, basically says that for the parameters parallax, pmra, pmdec and radial_velocity, NULL values must be treated as zero. That makes sense.

However, ivo_epoch_prop, which returns a propagated (ra,dec,plx,pmra,pmdec,rv) array, inherits the same parameter descriptions ("This is epoch_prop_pos as described in Sect. 2.2.1, except it returns a full parameter set at out_epoch.") That is problematic, because input NULL values get propagated as output non-NULL values. Try for instance at GAVO DC:

   SELECT TOP 10
          random_index,
          ra, dec, parallax, pmra, pmdec, radial_velocity,
          ivo_epoch_prop(ra, dec, parallax, pmra, pmdec, radial_velocity, 2016, 2100) AS astrom2100
   FROM gaia.dr3lite
   ORDER BY random_index

All the input radial_velocity values are NULL, but the propagated ones (apart from random_index=5) have a small definite value. Futhermore the parallax, pmra and pmdec values for the source with random_index=5, which are NULL on input, propagate to definite values of zero.

I note that this documentation and behaviour more or less follows practice of the Gaia archive, which says "Zero is used as the default value for any missing null input parameter"' though the details are not identical (esdc_epoch_prop yields zero for parallax and rv outputs given NULL parallax, pm, rv inputs, while GAVO's ivo_epoch_prop yields NaN for those).

Since DPAC has more accomplished astrometers on board than me I wonder whether this behaviour really is respectable... but I have to say it doesn't seem like it.

Do other people agree this looks problematic?

[msdemlei]

On Thu, May 09, 2024 at 08:28:52AM -0700, Mark Taylor wrote: which says _"Zero is used as the default value for any missing null input parameter"_' though the details are not identical (`esdc_epoch_prop` yields zero for `parallax` and `rv` outputs given NULL `parallax`, `pm`, `rv` inputs, while GAVO's `ivo_epoch_prop` yields NaN for those).

I'd argue NaN (a.k.a. NULL in a VOTable context for doubles) is less bad here, but...

Do other people agree this looks problematic?

...we are certainly setting up bad traps here. I have missed this problem, and I would consider it bad enough to change the definition of ivo_epoch_prop, as in: take it out of the current batch before others implement the data-inventing behaviour you are seeing. While inventing data certainly is out of the question, it is unclear to me what behaviour we should demand instead. I could see returning NULL rather than the array when the proper motion is missing; presumably, nobody would want a non-propagated position in a six-parameter solution, where that's a lot less clear for ivo_epoch_prop_pos. However, requring people to pass in a parallax, let alone an RV, seems wrong even in the age of Gaia. I suppose we *could* make them NaN on output if they were not passed in, as a special marker that no distance information is available. It's not pretty, but at least we'd not be inventing too much data. Hm.

[mbtaylor]

Up till now the equivalent functionality in stilts/topcat epochProp/epochPropErr has propagated blank (NaN) values for parallax or (either) PM through the equations without special measures; the effect is that blank input parallax or PM gives blank values for propagated position, parallax and PM. This is respectable but kind of annoying if you're just trying to get the best estimate for position at some different epoch.

[Concerning RV, it does the bad thing I'm complaining about above: a NaN input RV turns into a definite output RV.]

So I was considering changing its behaviour, or maybe adding another function, that does the following:

• on input, check and record PM, parallax and RV for NULL
• perform calculations as if NULL values for PM, parallax, RV are zero
• on output, for each of PM, parallax, RV, if the input was NULL, set the output to NULL

I would consider that behaviour a definite improvement on the current behaviour of ivo_epoch_prop.

If you're using this function to get the best available astrometry at a given epoch (apply PM if available, otherwise don't), I'd say this is what's required. My guess is that covers most people's requirements here.

There is another usage scenario which only wants positions at a given epoch if they are known to have definite proper motions applied. You could imagine separate UDFs, or a flag on epoch_prop_pos/epoch_prop that selects this behaviour. However I find it a bit hard to imagine why you'd want this without also wanting the propagated positional errors, which are anyway not supplied by either of these UDFs. So my feeling is that such a requirement is niche enough that we could expect people to handle it manually (WHERE pmra IS NOT NULL AND pmdec IS NOT NULL [AND parallax IS NOT NULL]).

[msdemlei]

So... I have implemented what is proposed in PR #20 in a local pgsphere, or so I think. I will wait for comments here before putting this on the operational server, though, so you cannot try it just yet.

Opinions?