Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate Library Contents API to FastAPI #18838

Open
wants to merge 71 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
bd70faf
Add tests for library contents API
arash77 Sep 12, 2024
d1ac6f0
Deleting and updating not possible for folders!
arash77 Sep 13, 2024
fa8ac49
Merge branch 'galaxyproject:dev' into FastAPI_library_contents
arash77 Sep 16, 2024
507e7c3
Refactor library content routes and controllers
arash77 Sep 16, 2024
c04ae31
Merge branch 'galaxyproject:dev' into FastAPI_library_contents
arash77 Sep 16, 2024
4621db9
Refactor library content routes and controllers
arash77 Sep 17, 2024
321951c
Merge branch 'galaxyproject:dev' into FastAPI_library_contents
arash77 Sep 17, 2024
1450ab2
Refactor library content routes and controllers
arash77 Sep 18, 2024
c5023cd
Refactor library content routes and controllers
arash77 Sep 19, 2024
0f26838
Refactor library content routes and controllers
arash77 Sep 20, 2024
dfe82e3
Refactor library content routes and controllers
arash77 Sep 24, 2024
7f039d9
Merge branch 'galaxyproject:dev' into FastAPI_library_contents
arash77 Sep 24, 2024
52ff504
Refactor library content routes and controllers
arash77 Sep 24, 2024
331bee7
Remove old library_contents API route configurations
arash77 Sep 24, 2024
e10fd62
make the review process easy
arash77 Sep 24, 2024
7a57443
Refactor library content routes and controllers
arash77 Sep 26, 2024
5591a8a
remove unused imports
arash77 Sep 26, 2024
c55e214
Merge branch 'galaxyproject:dev' into FastAPI_library_contents
arash77 Sep 26, 2024
73c6ddc
move _create_response function to services
arash77 Sep 26, 2024
f3e3f6d
Fix url_builder
arash77 Sep 26, 2024
eb36c65
Add `json=true` to library_content api tests
arash77 Sep 26, 2024
663d99c
update OpenAPI schema
arash77 Sep 26, 2024
d944728
Update OpenAPI schema
arash77 Sep 26, 2024
e60e2bc
Reformating populators.py
arash77 Sep 26, 2024
5f323a5
Fix test_libraries erros
arash77 Sep 26, 2024
912ba9b
add LibraryFolderDatabaseIdField Model for library_id
arash77 Sep 27, 2024
5bf8e41
Fix schema for uploaded_by being None in show library contents endpoi…
arash77 Sep 27, 2024
49b0796
Add function for supporting files in library_contents
arash77 Oct 1, 2024
b2ff40f
remove unused enumerate and files field into LibraryContentsFileCreat…
arash77 Oct 1, 2024
3e1cdb0
Refactor for better readability
arash77 Oct 1, 2024
52b94a1
send data as json when there is no files on creating new_library_data…
arash77 Oct 1, 2024
e53b4f9
update files to upload_files in create payload
arash77 Oct 1, 2024
d458015
remove unused import
arash77 Oct 1, 2024
2a5790d
send none files library_contents create in population with json
arash77 Oct 1, 2024
20bd8fb
Apply suggestions: s/title/description in schema, don't do try/except…
arash77 Oct 1, 2024
2a3d90e
remove unused import
arash77 Oct 1, 2024
d4e7f5d
Add tests for library contents API
arash77 Sep 12, 2024
a4118e4
Deleting and updating not possible for folders!
arash77 Sep 13, 2024
e5beafe
Refactor library content routes and controllers
arash77 Sep 16, 2024
adb0b2d
Refactor library content routes and controllers
arash77 Sep 17, 2024
3a4ee6c
Refactor library content routes and controllers
arash77 Sep 18, 2024
f8e2f24
Refactor library content routes and controllers
arash77 Sep 19, 2024
156ede6
Refactor library content routes and controllers
arash77 Sep 20, 2024
b80f249
Refactor library content routes and controllers
arash77 Sep 24, 2024
4dee8ed
Refactor library content routes and controllers
arash77 Sep 24, 2024
b90179d
Remove old library_contents API route configurations
arash77 Sep 24, 2024
0c6837d
make the review process easy
arash77 Sep 24, 2024
fdcc850
Refactor library content routes and controllers
arash77 Sep 26, 2024
f423125
remove unused imports
arash77 Sep 26, 2024
3d41893
move _create_response function to services
arash77 Sep 26, 2024
4cf20ed
Fix url_builder
arash77 Sep 26, 2024
4873336
Add `json=true` to library_content api tests
arash77 Sep 26, 2024
3995e6e
update OpenAPI schema
arash77 Sep 26, 2024
f3670d9
Update OpenAPI schema
arash77 Sep 26, 2024
75c6489
Reformating populators.py
arash77 Sep 26, 2024
a9151e9
Fix test_libraries erros
arash77 Sep 26, 2024
07db06a
add LibraryFolderDatabaseIdField Model for library_id
arash77 Sep 27, 2024
c0e589b
Fix schema for uploaded_by being None in show library contents endpoi…
arash77 Sep 27, 2024
9e47626
Add function for supporting files in library_contents
arash77 Oct 1, 2024
3e22733
remove unused enumerate and files field into LibraryContentsFileCreat…
arash77 Oct 1, 2024
6d59a46
Refactor for better readability
arash77 Oct 1, 2024
d43e6b8
send data as json when there is no files on creating new_library_data…
arash77 Oct 1, 2024
4d30692
update files to upload_files in create payload
arash77 Oct 1, 2024
9d0bd72
remove unused import
arash77 Oct 1, 2024
f4cffb3
send none files library_contents create in population with json
arash77 Oct 1, 2024
0a86966
Apply suggestions: s/title/description in schema, don't do try/except…
arash77 Oct 1, 2024
d7c3b81
remove unused import
arash77 Oct 1, 2024
89a056e
update schema
arash77 Oct 2, 2024
8d417fb
Merge branch 'LC_test' into FastAPI_library_contents
arash77 Oct 2, 2024
b03ce07
schema linting
arash77 Oct 2, 2024
110e467
Using `DecodedDatabaseIdField` instead of `Union[DecodedDatabaseIdFie…
arash77 Oct 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
838 changes: 829 additions & 9 deletions client/src/api/schema/schema.ts

Large diffs are not rendered by default.

198 changes: 112 additions & 86 deletions lib/galaxy/actions/library.py

Large diffs are not rendered by default.

327 changes: 327 additions & 0 deletions lib/galaxy/schema/library_contents.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,327 @@
import json
from enum import Enum
from typing import (
Any,
Dict,
List,
Optional,
Union,
)

from pydantic import (
ConfigDict,
Field,
RootModel,
)
from pydantic.functional_validators import field_validator
from typing_extensions import (
Annotated,
Literal,
)

from galaxy.schema.fields import (
DecodedDatabaseIdField,
EncodedDatabaseIdField,
EncodedLibraryFolderDatabaseIdField,
LibraryFolderDatabaseIdField,
ModelClassField,
)
from galaxy.schema.schema import (
Model,
TagCollection,
)


class UploadOption(str, Enum):
upload_file = "upload_file"
upload_paths = "upload_paths"
upload_directory = "upload_directory"


class CreateType(str, Enum):
file = "file"
folder = "folder"
collection = "collection"


class LinkDataOnly(str, Enum):
copy_files = "copy_files"
link_to_files = "link_to_files"


class LibraryContentsCreatePayload(Model):
create_type: CreateType = Field(
...,
description="the type of item to create",
)
upload_option: UploadOption = Field(
UploadOption.upload_file,
description="the method to use for uploading files",
)
folder_id: LibraryFolderDatabaseIdField = Field(
...,
description="the encoded id of the parent folder of the new item",
)
tag_using_filenames: bool = Field(
False,
description="create tags on datasets using the file's original name",
)
tags: List[str] = Field(
[],
description="create the given list of tags on datasets",
)
from_hda_id: Optional[DecodedDatabaseIdField] = Field(
None,
description="(only if create_type is 'file') the encoded id of an accessible HDA to copy into the library",
)
from_hdca_id: Optional[DecodedDatabaseIdField] = Field(
None,
description="(only if create_type is 'file') the encoded id of an accessible HDCA to copy into the library",
)
ldda_message: str = Field(
"",
description="the new message attribute of the LDDA created",
)
extended_metadata: Optional[Dict[str, Any]] = Field(
None,
description="sub-dictionary containing any extended metadata to associate with the item",
)

@field_validator("tags", mode="before", check_fields=False)
@classmethod
def tags_string_to_json(cls, v):
if isinstance(v, str):
return json.loads(v)
return v


class LibraryContentsFileCreatePayload(LibraryContentsCreatePayload):
dbkey: Union[str, list] = Field(
"?",
title="database key",
)
roles: str = Field(
"",
title="user selected roles",
)
file_type: Optional[str] = Field(
None,
title="file type",
)
server_dir: str = Field(
"",
description="(only if upload_option is 'upload_directory') relative path of the "
"subdirectory of Galaxy ``library_import_dir`` (if admin) or "
"``user_library_import_dir`` (if non-admin) to upload. "
"All and only the files (i.e. no subdirectories) contained "
"in the specified directory will be uploaded.",
)
filesystem_paths: str = Field(
"",
description="(only if upload_option is 'upload_paths' and the user is an admin) "
"file paths on the Galaxy server to upload to the library, one file per line",
)
link_data_only: LinkDataOnly = Field(
LinkDataOnly.copy_files,
description="(only when upload_option is 'upload_directory' or 'upload_paths')."
"Setting to 'link_to_files' symlinks instead of copying the files",
)
uuid: Optional[str] = Field(
None,
title="UUID of the dataset to upload",
)
upload_files: List[Dict[str, Any]] = Field(
[],
title="list of the uploaded files",
)

# uploaded file fields
model_config = ConfigDict(extra="allow")


class LibraryContentsFolderCreatePayload(LibraryContentsCreatePayload):
name: str = Field(
"",
title="name of the folder to create",
)
description: str = Field(
"",
title="description of the folder to create",
)


class LibraryContentsCollectionCreatePayload(LibraryContentsCreatePayload):
collection_type: str = Field(
...,
title="the type of collection to create",
)
element_identifiers: List[Dict[str, Any]] = Field(
...,
title="list of dictionaries containing the element identifiers for the collection",
)
name: Optional[str] = Field(
None,
title="the name of the collection",
)
hide_source_items: bool = Field(
False,
description="if True, hide the source items in the collection",
)
copy_elements: bool = Field(
False,
description="if True, copy the elements into the collection",
)


class LibraryContentsUpdatePayload(Model):
converted_dataset_id: Optional[DecodedDatabaseIdField] = Field(
None,
title="the decoded id of the dataset",
)


class LibraryContentsDeletePayload(Model):
purge: bool = Field(
False,
description="if True, purge the library dataset",
)


class LibraryContentsIndexResponse(Model):
type: str
name: str
url: str


class LibraryContentsIndexFolderResponse(LibraryContentsIndexResponse):
id: EncodedLibraryFolderDatabaseIdField


class LibraryContentsIndexDatasetResponse(LibraryContentsIndexResponse):
id: EncodedDatabaseIdField


class LibraryContentsIndexListResponse(RootModel):
root: List[Union[LibraryContentsIndexFolderResponse, LibraryContentsIndexDatasetResponse]]


class LibraryContentsShowResponse(Model):
name: str
genome_build: Optional[str]
update_time: str
parent_library_id: EncodedDatabaseIdField


class LibraryContentsShowFolderResponse(LibraryContentsShowResponse):
model_class: Annotated[Literal["LibraryFolder"], ModelClassField(Literal["LibraryFolder"])]
id: EncodedLibraryFolderDatabaseIdField
parent_id: Optional[EncodedLibraryFolderDatabaseIdField]
description: str
item_count: int
deleted: bool
library_path: List[str]


class LibraryContentsShowDatasetResponse(LibraryContentsShowResponse):
model_class: Annotated[Literal["LibraryDataset"], ModelClassField(Literal["LibraryDataset"])]
id: EncodedDatabaseIdField
ldda_id: EncodedDatabaseIdField
folder_id: EncodedLibraryFolderDatabaseIdField
state: str
file_name: str
created_from_basename: Optional[str]
uploaded_by: Optional[str]
message: Optional[str]
date_uploaded: str
file_size: int
file_ext: str
data_type: str
misc_info: Optional[str]
misc_blurb: Optional[str]
peek: Optional[str]
uuid: str
tags: TagCollection

# metadata fields
model_config = ConfigDict(extra="allow")


class LibraryContentsCreateResponse(Model):
name: str
url: str


class LibraryContentsCreateFolderResponse(LibraryContentsCreateResponse):
id: EncodedLibraryFolderDatabaseIdField


class LibraryContentsCreateFileResponse(LibraryContentsCreateResponse):
id: EncodedDatabaseIdField


class LibraryContentsCreateFolderListResponse(RootModel):
root: List[LibraryContentsCreateFolderResponse]


class LibraryContentsCreateFileListResponse(RootModel):
root: List[LibraryContentsCreateFileResponse]


class LibraryContentsCreateDatasetResponse(Model):
# id, library_dataset_id, parent_library_id should change to EncodedDatabaseIdField latter
# because they are encoded ids in _copy_hda_to_library_folder and _copy_hdca_to_library_folder
# functions that are shared by LibraryFolderContentsService too
id: str
hda_ldda: str
model_class: Annotated[
Literal["LibraryDatasetDatasetAssociation"], ModelClassField(Literal["LibraryDatasetDatasetAssociation"])
]
name: str
deleted: bool
visible: bool
state: str
library_dataset_id: str
file_size: int
file_name: str
update_time: str
file_ext: str
data_type: str
genome_build: str
misc_info: Optional[str]
misc_blurb: Optional[str]
created_from_basename: Optional[str]
uuid: str
parent_library_id: str

# metadata fields
model_config = ConfigDict(extra="allow")


class LibraryContentsCreateDatasetCollectionResponse(RootModel):
root: List[LibraryContentsCreateDatasetResponse]


class LibraryContentsDeleteResponse(Model):
id: EncodedDatabaseIdField
deleted: bool


class LibraryContentsPurgedResponse(LibraryContentsDeleteResponse):
purged: bool


AnyLibraryContentsShowResponse = Union[
LibraryContentsShowFolderResponse,
LibraryContentsShowDatasetResponse,
]

AnyLibraryContentsCreatePayload = Union[
LibraryContentsFolderCreatePayload, LibraryContentsFileCreatePayload, LibraryContentsCollectionCreatePayload
]

AnyLibraryContentsCreateResponse = Union[
LibraryContentsCreateFolderListResponse,
LibraryContentsCreateFileListResponse,
LibraryContentsCreateDatasetCollectionResponse,
LibraryContentsCreateDatasetResponse,
]
Loading
Loading