Add the ability to evaluate a model with different test label timespans without retraining models

[kasunamare]

Currently, if we want to use multiple test_label_timespans in the same experiment (without changing the train_label_timespan), triage creates a separate model using the same training data for each test label timespan. While this is not wrong per se, this could be inefficient. For instance, we could run only the evaluations using the new labels without retraining the whole model.

Since changing entire triage would be a big lift and since we do not necessarily need these evaluations for the whole model grid, we could tackle this at the post-modeling level? For instance, we could have a tool to add evaluations (similar to the one that adds predictions). However, current definitions of the predictions and evaluations tables do not allow that.

The primary key of the predictions table is (model_id, entity_id, as_of_date). This is correct, as the score/prediction only depends on those three factors. But, we do include the label and the label_timespan in the predictions table, and as I understand, that's the only place we persist those. We could include the label_timespan in the predictions table primary key? (even though it will repeat the predictions)

The evaluations table does capture the matrix_uuid in the primary key, but not the label timespan explicitly. If we wanted to create this as a postmodeling utility, we wouldn't want to create a new matrix with the same features, so we could include the label_timespan in the evaluations table, and add it to the primary key? (But this would create a mismatch between the 'original' label included in the matrix and the second evaluations generated)

Would be great to hear your thoughts @shaycrk @thcrock @nanounanue @rayidghani @ecsalomon

[ecsalomon]

I agree with you on including the additional field in the key for predictions tables. It probably already should be in there. When would it repeat predictions?

adding the label timespan to the evaluations table primary key is probably the right stopgap for allowing projects to move forward. It might be good to create a fork or branch for that that main is merged into since the issues you point out will be confusing for users. But also may not be worth the effort.

I still think pulling apart features and labels is way to go but won't have time to work on that for ~1 month. However, I could put aside some time in the last half of March / early April for a Triage sprint if that would be useful.