Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Beta User WDL to use bulk ingest [VS-982] #8379

Closed
wants to merge 36 commits into from
Closed

Conversation

rsasch
Copy link

@rsasch rsasch commented Jun 22, 2023

latest run of Beta User WDL with bulk ingest and no data table column names provided: https://app.terra.bio/#workspaces/gvs-dev/RSA%20-%20GVS%20Quickstart%20V2%20/job_history/79337ea3-593f-4896-b056-9f47bfd7c7c7

@rsasch rsasch marked this pull request as draft June 22, 2023 21:37
@codecov
Copy link

codecov bot commented Jun 22, 2023

Codecov Report

❗ No coverage uploaded for pull request base (ah_var_store@90aa0fd). Click here to learn what that means.
The diff coverage is n/a.

Additional details and impacted files
@@               Coverage Diff                @@
##             ah_var_store     #8379   +/-   ##
================================================
  Coverage                ?   76.504%           
  Complexity              ?     21795           
================================================
  Files                   ?      1392           
  Lines                   ?     83134           
  Branches                ?     13184           
================================================
  Hits                    ?     63601           
  Misses                  ?     14369           
  Partials                ?      5164           

@rsasch rsasch marked this pull request as ready for review June 26, 2023 19:01
scripts/variantstore/wdl/GvsBulkIngestGenomes.wdl Outdated Show resolved Hide resolved
.dockstore.yml Show resolved Hide resolved
scripts/variantstore/wdl/GvsExtractAvroFilesForHail.wdl Outdated Show resolved Hide resolved
scripts/variantstore/wdl/GvsQuickstartHailIntegration.wdl Outdated Show resolved Hide resolved
scripts/variantstore/wdl/GvsQuickstartIntegration.wdl Outdated Show resolved Hide resolved
scripts/variantstore/wdl/GvsUtils.wdl Outdated Show resolved Hide resolved
@rsasch rsasch marked this pull request as draft June 28, 2023 13:33
@gatk-bot
Copy link

Github actions tests reported job failures from actions build 5404926238
Failures in the following jobs:

Test Type JDK Job ID Logs
conda 8 5404926238.3 logs

@gatk-bot
Copy link

Github actions tests reported job failures from actions build 5405169587
Failures in the following jobs:

Test Type JDK Job ID Logs
conda 8 5405169587.3 logs

@gatk-bot
Copy link

Github actions tests reported job failures from actions build 5405222155
Failures in the following jobs:

Test Type JDK Job ID Logs
conda 8 5405222155.3 logs

@gatk-bot
Copy link

Github actions tests reported job failures from actions build 5405579628
Failures in the following jobs:

Test Type JDK Job ID Logs
conda 8 5405579628.3 logs

@gatk-bot
Copy link

gatk-bot commented Jun 28, 2023

Github actions tests reported job failures from actions build 5405771210
Failures in the following jobs:

Test Type JDK Job ID Logs
conda 8 5405771210.3 logs
conda 8 5405771210.3 logs
conda 8 5405771210.3 logs

@rsasch rsasch requested a review from mcovarr June 28, 2023 21:52
@rsasch rsasch marked this pull request as ready for review June 28, 2023 21:52
@rsasch rsasch changed the title Update Beta User WDL to use bulk ingest and update integration WDL to use Beta User WDL [VS-982] Update Beta User WDL to use bulk ingest [VS-982] Jun 29, 2023
@@ -243,13 +243,13 @@ def get_column_values(columnSamples, numSamples, user_defined_vcf, user_defined_
# Check the contents of the columns: the duck algorithm. If its contents LOOK like vcfs and indexes, go from there
for col in path_ends_in_vcf_gz:
# ...and has an analogue that looks like an index file?
index_column = f"{column}_index"
index_column = f"{col}_index"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uh what was this doing before these changes?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

failing!

@mcovarr
Copy link
Collaborator

mcovarr commented Jul 10, 2023

Heads up my one-FOFN-to-rule-them-all PR ended up turning into a PR that does (or at least tries to) do all the bulk ingest things. Happy to discuss in mobbing

@rsasch
Copy link
Author

rsasch commented Jul 10, 2023

closing in favor of #8404 (and possible other future PRs)

@rsasch rsasch closed this Jul 10, 2023
@rsasch rsasch deleted the rsa_vs_982 branch October 24, 2023 17:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants