VS-1470 Have Extract Cohort to PGEN write to cost_observability table

[gbggrant]

Fix a bug where ExtractCohortToPgen does not write to the cost_observability table.

• Successful run here
• Successful integration test here

[koncheto-broad]

LGTM

[rsasch]

Looks ok to me, although I wonder why some of the intervals in the test run have 0 BigQuery Query Scanned

[gbggrant]

Thanks! This is a good question. I had seen it too and looked into it much greater detail now. For Extract (both for VCF and PGEN) the only BigQuery activity that is being recorded is on the sample_info table. Each shard in a multi-sharded extract is querying the entire table to get a map of sample id to sample name (excluding withdrawn). Since every shard is running the same query (again, not sample-specific, it's the whole table!), BigQuery returns the cached result in almost all cases and so, the scanned value is 0. For the run in question, there were two shards which returned values of 54. I think they ran close enough in time that they weren't cached.