Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable the creation of partial metadata and subsequent updating #77

Open
andrewphilipsmith opened this issue Apr 11, 2024 · 0 comments
Open
Milestone

Comments

@andrewphilipsmith
Copy link
Collaborator

The metadata models contain many fields. Of these some will be known early in the pipeline, whilst others cannot be determined until later. For example:

  • source_metric_id will be known early in the pipeline.
  • metric_parquet_file_url cannot be determined until late in the publishing pipeline.

Therefore we need to ensure it is possible to:

  • Create an incomplete metadata object (ie that is one where some fields are None) to store known metadata as it progresses in the pipeline.
  • Validitate the metadata object at the end of the pipeline, to ensure that the fields are correctly typed, non-none, etc.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Backlog:
Development

No branches or pull requests

1 participant