Add ability for indexed sequence file to be uploaded and used in an assembly #432

garrettjstevens · 2024-08-30T23:03:10Z

Fixes #405, fixes #406, fixes #407, fixes #424

garrettjstevens

One thing I didn't think about when we were discussing this before is that any Change can't have nodejs-specific code in it (i.e. no imports from node:something). This is because this code has to run both in the browser and on the server. I think the nodejs-specific code that got introduced is what is causing the tests to fail.

What we'll need to do instead is that in packages/apollo-collaboration-server/src/files/files.service.ts we'll add a getFileHandle method that returns a file handle from a file document, very similar to how getFileStream in that same service file returns a file stream from a file document. Then the implementation of LocalFileGzip will be in collaboration server (e.g. in packages/apollo-collaboration-server/src/files/filesUtil.ts) instead of the shared code.

Then the code in the Change will look something like this:

const { filesService } = backend
const { getFileHandle } = filesService
new IndexedFasta({
  fasta: getFileHandle(fastaDoc),
  fai: getFileHandle(faiDoc),
})

(Don't forget to add the definition for getFileHandle in packages/apollo-common/src/Operation.ts)

garrettjstevens · 2024-09-03T23:13:17Z

packages/apollo-cli/src/commands/assembly/add-file.ts

This should be combined with the add-fasta command, they should not be separate commands.

Also, add-fasta and add-gff are unclear as to their function based on the command name. So what we need to do is:

Rename add-gff to add-from-gff

Rename add-fasta to add-from-fasta

Combine add-file into add-from-fasta

Another thing to consider, maybe the argument to add-from-fasta should be positional and not a flag, since it's required in all cases. What do you think of this for example commands:

apollo assembly add-from-fasta ./my-genome.fa apollo assembly add-from-fasta ./my-genome.fa.gz apollo assembly add-from-fasta 507f1f77bcf86cd799439011 apollo assembly add-from-fasta ./my-genome.fa.gz --not-editable apollo assembly add-from-fasta ./my-genome.fa.gz --not-editable --fai ./my-genome.fa.gz.fai --gzi ./my-genome.fa.gz.gzi apollo assembly add-from-fasta 507f1f77bcf86cd799439011 --not-editable --fai 507f1f77bcf86cd799486197 --gzi 507f1f77bcf86cd785963292

garrettjstevens · 2024-09-03T23:19:38Z

packages/apollo-cli/src/commands/file/upload.ts

+      description:
+        'File type or "autodetect" for automatic detection.\n\
+        NB: There is no check for whether the file complies to this type',
+      options: ['text/x-fasta', 'text/x-gff3', 'autodetect'],


We need to add new options here, since we want to be able to tell the type later. Types I can think of are:

text/x-fasta

text/x-gff3

application/x-bgzip-fasta

text/x-fai

application/x-gzi

garrettjstevens · 2024-09-03T23:23:53Z

packages/apollo-cli/src/fileCommand.ts

+        Authorization: `Bearer ${accessToken}`,
+        'Content-Type': type,
+        'Content-Length': String(size),
+        'Content-Encoding': contentEncoding,


Leave out Content-Encoding altogether unless it's gzip, maybe something like:

const headers = new Headers({ Authorization: `Bearer ${accessToken}`, 'Content-Type': type, 'Content-Length': String(size), }) if (isGzip) { headers.append('Content-Encoding', 'gzip') }

garrettjstevens · 2024-09-03T23:24:33Z

packages/apollo-cli/src/fileCommand.ts

+    })
+
+    let contentEncoding = ''
+    if (file.endsWith('.gz')) {


Pass a param to uploadFile instead of checking the file name.

garrettjstevens · 2024-09-03T23:25:10Z

packages/apollo-collaboration-server/src/files/filesUtil.ts

-  const gz = createGzip()
-  await pipeline(stream, gz, fileWriteStream)
-
+  if (originalname.endsWith('.gz')) {


Check the ~~Content-Encoding~~ Content-Type instead of the filename.

garrettjstevens · 2024-09-03T23:25:51Z

packages/apollo-shared/src/Changes/AddAssemblyFromFileIdChange.ts

This should be folded in to the existing AddAssemblyFromFileChange instead of being a new change.

garrettjstevens · 2024-09-03T23:32:24Z

packages/apollo-cli/src/commands/assembly/add-fasta.ts

    }),
    force: Flags.boolean({
      char: 'f',
      description: 'Delete existing assembly, if it exists',
    }),
+    'no-db': Flags.boolean({


no-db should be something like not-editable, since that what is relevant to the user (the database is an internal detail they don't need to worry about)

garrettjstevens · 2024-09-03T23:33:43Z

packages/apollo-cli/src/commands/assembly/add-fasta.ts

+      description: wrapLines(
+        "Do not load the fasta sequence into the Apollo database. \
+      This option assumes the fasta file is bgzip'd with `bgzip` and indexed with `samtools faidx`.\
+      Indexes should be named <my.fasta.gz>.gzi and <my.fasta.gz>.fai",


Auto-detection is good, but there should also be a way to explicitly pass the fai and gzi files.

Fix #424

But most likely there will be changes to clunky code

garrettjstevens · 2024-09-18T23:15:14Z

@dariober the issue with the GZI filehandle we were seeing has been fixed

All tests except those for chekcs passing

* CLI can upload and add assembly from gzip files (#405 and #407) * CLI can add assembly in non-editable mode with `--no-db` flag, i.e. without loading sequence to mongodb (#406). Only bgzip'd & indexed fasta file are supported, not plain fa/fai files. See tests in test.py: `testFeatureChecksIndexed`, `testFileUploadGzip`, `testAddAssemblyWithoutLoadingInMongo`

Fix #424

But most likely there will be changes to clunky code

All tests except those for chekcs passing

garrettjstevens assigned dariober Aug 30, 2024

garrettjstevens commented Sep 3, 2024

View reviewed changes

dariober added a commit that referenced this pull request Sep 17, 2024

Tmp commit addressing #432

c988574

Fix #424

dariober added a commit that referenced this pull request Sep 18, 2024

This should address comments in #432

b01ad2a

But most likely there will be changes to clunky code

dariober added a commit that referenced this pull request Sep 19, 2024

Fix fileId vs fileIds and other issues for #432

883c7d3

All tests except those for chekcs passing

dariober and others added 24 commits September 20, 2024 19:51

Start implementing file upload|download|get

7ad394b

Add delete command and various small edits

ef4ec71

Add FileCommand handling file operations

07b0a09

Handle default profile via undefined

4ead5e0

Download file as stream

7112709

Fix linter

7b3255e

Formatting

e8a1919

Add description for file CLI topic

16aebbb

Deduplicate sequence fetching code

ae152ca

Reference File schema from Assembly schema

57d553e

Rename commands

fc363d9

Remove -i option since file is positional

63f2615

Rename option --no-db and some other fixes

b149a7f

Add additional type and make input arg required

48d2115

Tmp commit

f4b6887

fix app start issue

9467881

fix lint

dc6dd33

Tmp commit addressing #432

6865ff8

Fix #424

Tmp commit

e1c8777

This should address comments in #432

25bdefc

But most likely there will be changes to clunky code

Temp commit to fix buffer

860575a

Fix filehandle type

60ce5b2

Use files service to get filehandles

12c83f1

dariober added 2 commits September 20, 2024 19:55

Fix fileId vs fileIds and other issues for #432

3526bb8

All tests except those for chekcs passing

Allow fasta input to be gzip

1302ce4

garrettjstevens force-pushed the cli-upload-only-issue405-gs branch from 765ceab to 1302ce4 Compare September 20, 2024 19:55

Fixes after rebase

9d149dc

garrettjstevens force-pushed the cli-upload-only-issue405-gs branch from 620ac25 to 9d149dc Compare September 20, 2024 20:05

Move LocalFileGzip to collab server package

db455a3

garrettjstevens force-pushed the cli-upload-only-issue405-gs branch from 7e49495 to db455a3 Compare September 20, 2024 20:55

Match selector to other updated tests

14a2ada

garrettjstevens merged commit 89ccf49 into main Sep 21, 2024
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ability for indexed sequence file to be uploaded and used in an assembly #432

Add ability for indexed sequence file to be uploaded and used in an assembly #432

garrettjstevens commented Aug 30, 2024 •

edited

Loading

garrettjstevens left a comment

garrettjstevens Sep 3, 2024

garrettjstevens Sep 3, 2024

garrettjstevens Sep 3, 2024

garrettjstevens Sep 3, 2024

garrettjstevens Sep 3, 2024 •

edited

Loading

garrettjstevens Sep 3, 2024

garrettjstevens Sep 3, 2024

garrettjstevens Sep 3, 2024

garrettjstevens commented Sep 18, 2024

Add ability for indexed sequence file to be uploaded and used in an assembly #432

Add ability for indexed sequence file to be uploaded and used in an assembly #432

Conversation

garrettjstevens commented Aug 30, 2024 • edited Loading

garrettjstevens left a comment

Choose a reason for hiding this comment

garrettjstevens Sep 3, 2024

Choose a reason for hiding this comment

garrettjstevens Sep 3, 2024

Choose a reason for hiding this comment

garrettjstevens Sep 3, 2024

Choose a reason for hiding this comment

garrettjstevens Sep 3, 2024

Choose a reason for hiding this comment

garrettjstevens Sep 3, 2024 • edited Loading

Choose a reason for hiding this comment

garrettjstevens Sep 3, 2024

Choose a reason for hiding this comment

garrettjstevens Sep 3, 2024

Choose a reason for hiding this comment

garrettjstevens Sep 3, 2024

Choose a reason for hiding this comment

garrettjstevens commented Sep 18, 2024

garrettjstevens commented Aug 30, 2024 •

edited

Loading

garrettjstevens Sep 3, 2024 •

edited

Loading