Handling versioning of the popgetter pipeline

[sgreenbury]

Placeholder for handling versioning of the pipeline data on Azure.

[andrewphilipsmith]

Quoting comments made by @penelopeysm below:

---

The versioning strategy feels quite tricky to get right. I think there is a correct answer, and I'm not sure if the below is it, but I hope it's not too far from it. Also I'll adopt Andy's names poppusher and popgetter because I think they are very sensible!

• popgetter/Azure synchronisation: version X.Y.Z of popgetter should read from an Azure folder called vX.Y
  • having pondered this a little bit, I would vote to not include Z in the Azure folder --- this means that we don't have to republish data if we add non-breaking changes like new features to popgetter
  • this should ideally be encoded somehow in popgetter itself (e.g. by fetching its own version number) rather than being a string in the config that needs to change
• poppusher/Azure synchronisation: version X.Y.Z of poppusher should deploy to an Azure folder called vX.Y
  • Ideally this would also be encoded as an invariant, rather than it being a string that you put in .env , but how we implement this will depend on how we choose to generate and upload the data. (having a standardised way to generate & upload data, as in my other PR, could solve this — not pushing you to look at it now, just offering it as a possibility 🙂)
• The two criteria above effectively imply that the major+minor versions of poppusher and popgetter would stay in sync.
• Patch versions can, in theory, be incremented on both sides without any breakage. For example:
  • If we expand poppusher to enrich the metadata to include extra fields, that shouldn't cause popgetter to break. We could then bump the patch version of poppusher and regenerate the data on Azure.
  • If we then update popgetter to make use of this extra metadata, that can be a patch version bump to popgetter
• Minor version bumps could be driven by changes to poppusher. For example:
  • If we remove a metadata field that popgetter requires, we should bump the minor version of popgetter and republish the data on a new folder.
  • Simultaneously, modify popgetter to work with the reduced metadata and release a new minor version of popgetter.
• Minor version bumps could also be driven by breaking changes to popgetter's API, since popgetter exposes a library that can be used as a dependency by other Rust crates. However I suppose we should try to keep this to a minimum (or aggregate lots of changes into one minor version release) to avoid needlessly republishing the same data every time this happens.***
• Major version bumps follow the same rules as minor version bumps, but chosen to mark something more significant (at your discretion ;))
• Finally, this may be a question for when we actually have more minor versions, but we should decide how many minor versions' worth of data we want to keep on Azure so that old versions of popgetter will still work. It's fine if the answer is 1 😉 but we should document it, and/or put empty v0.1 folders on Azure with something like an error.txt file which the CLI can parse and pass to someone who tries to use it.

*** In theory, we could have separate minor versions for poppusher and popgetter to avoid republishing data due to this. i.e. instead of a one-to-one mapping between poppusher and popgetter, it would be a one-to-many i.e. multiple versions of popgetter can depend on one version of poppusher. I think this was what I had in mind when I wrote the current popgetter README. However revisiting this, I think the cognitive overhead of doing this is unnecessarily high, and it also makes it impossible to enforce the version invariants in code, which I think is a huge plus of keeping the version numbers in sync.