v2.4.0

Highlights

• Added ability to authenticate from a SYNAPSE_AUTH_TOKEN environment variable set with a valid personal access token.

# e.g. set environment variable prior to invoking a Synapse command or running a program that uses synapseclient
SYNAPSE_AUTH_TOKEN='<my_personal_access_token>' synapse <subcommand options>

The environment variable will take priority over credentials in the user’s .synapseConfig file or any credentials saved in a prior login using the remember me option.

See here for more details on usage.

• Added ability to silence all console output.

# from the command line, use the --silent option with any synapse subcommand, here it will suppress the download progress indicator
synapse --silent get <synid>

# from code using synapseclient, pass the silent option to the Synapse constructor
import synapseclient

syn = synapseclient.Synapse(silent=True)
syn.login()
syn.get(<synid>)

• Improved robustness during downloads with unstable connections. Specifically the client will automatically recover when encoutering some types of network errors that previously would have caused a download to start over as indicated by a reset progress bar.

Bug Fixes

• [SYNPY-198] - get: Unmet access requirement should not raise error if entity not downloadable
• [SYNPY-959] - FileEntity 'path' property has wrong separator in Windows
• [SYNPY-1113] - Confusing error when putting the positional FILE at the end of the synapse store command with an optional n-arg
• [SYNPY-1128] - failures downloading 14G vcf file
• [SYNPY-1130] - Migration tool trying to move URL-linked data
• [SYNPY-1134] - 500 error during part copy to AWS presigned url
• [SYNPY-1135] - Exceeding part limit during AD Migration
• [SYNPY-1136] - Connection aborted to AWS part copy to presigned url during AD Migration
• [SYNPY-1141] - synapse get command line nargs usage/error
• [SYNPY-1150] - Boolean-like string columns being reformatted (TRUE/FALSE to True/False)
• [SYNPY-1158] - race condition in test_caching.py#test_threaded_access
• [SYNPY-1159] - logging in with an email address and an authToken gives spurious error
• [SYNPY-1161] - ChunkEncodingError encountered from external collaborator during a synapseclient download

Improvements

• [SYNPY-638] - add after date to cache purge
• [SYNPY-929] - silent parameter for all functions which default to writing to stdout
• [SYNPY-1068] - Should show some progress indicator during upload md5 calculation
• [SYNPY-1125] - Allow login with environment variables

Stories

• [SYNPY-1138] - When using boto3 client to upload a file, also include ACL to give bucket owner full access

Tasks

• [SYNPY-948] - command line client set-annotations does not return proper error code when there's a problem
• [SYNPY-1024] - remove reference to deprecated 'status' field from Evaluation
• [SYNPY-1115] - Setting provenance in store ignored for tables
• [SYNPY-1143] - indicate in CLI doc's that select statement requires double quotes

v2.3.1

Highlights

• Entities can be annotated with boolean datatypes, e.g.:

  file = synapseclient.File('/path/to/file', parentId='syn123', synapse_is_great=True)
  syn.store(file)

• synapseclient is additionally packaged as a Python wheel.

Bug Fixes

• [SYNPY-829] - syn.store always updates annotations
• [SYNPY-1033] - If versionComment is left blank, previous version comment populates

Improvements

• [SYNPY-1120] - Build wheel distributions
• [SYNPY-1129] - support boolean annotations in Python client

v2.3.0

Highlights

• The index_files_for_migration and migrate_indexed_files functions are added to synapseutils to help migrate files in Synapse projects and folders between AWS S3 buckets in the same region. More details on using these utilities can be found here.

• This version supports login programatically and from the command line using personal access tokens that can be obtained from your synapse.org Settings. Additional documentation on login and be found here.

  # programmatic
  syn = synapseclient.login(authToken=<token>)

  # command line
  synapse login -p <token>

• The location where downloaded entities are cached can be customized to a location other than the user’s home directory. This is useful in environments where writing to a home directory is not appropriate (e.g. an AWS lambda).

  syn = synapseclient.Synapse(cache_root_dir=<directory path>)

• A helper method on the Synapse object has been added to enable obtaining the Synapse certification quiz status of a user.

  passed = syn.is_certified(<username or user_id>)

• This version has been tested with Python 3.9.

Bug Fixes

• [SYNPY-1039] - tableQuery asDataFrame() results with TYPE_LIST columns should be lists and not literal strings
• [SYNPY-1109] - unparseable synapse cacheMap raises JSONDecodeError
• [SYNPY-1110] - Cleanup on Windows console login
• [SYNPY-1112] - Concurrent migration of entities sharing the same file handle can result in an error
• [SYNPY-1114] - Mitigate new Rust compiler dependency on Linux via transitive cryptography dependency
• [SYNPY-1118] - Migration tool erroring when it shouldn’t

New Features

• [SYNPY-1058] - Accept oauth access token for authentication to use Synapse REST services
• [SYNPY-1103] - Multipart copy integration
• [SYNPY-1111] - Add function to get user certification status

Improvements

• [SYNPY-885] - Public interface to customize CACHE_ROOT_DIR
• [SYNPY-1102] - syncToSynapse adds empty annotation values
• [SYNPY-1104] - Python 3.9 support
• [SYNPY-1119] - Add source storage location option to storage migrate functions

v2.2.2

Highlights

• This version addresses an issue with downloads being retried unsuccessfully after encountering certain types of errors.
• A create_snapshot_version function is added for making table and view snapshots.

Bugs

• [SYNPY-1096] - Fix link to Synapse on PyPI
• [SYNPY-1097] - downloaded files are reset when disk space exhausted

New Features

• [SYNPY-1041] - Snapshot feature and programmatic clients

Improvements

• [SYNPY-1063] - Consolidate builds to GitHub Actions
• [SYNPY-1099] - Replace usage of deprecated PUT /entity/{id}/version endpoint

v2.2.0

Highlights

• Files that are part of syncFromSynapse and syncToSynapse operations (synapse get -r and synapse sync in the command line client, respectively) are transferred in in parallel threads rather than serially, substantially improving the performance of these operations.
• Table metadata from synapse get -q is automatically downloaded to a users working directory instead of to the Synapse cache (a hidden folder).
• Users can now pass their API key to synapse login in place of a password.

Bugs

• [SYNPY-1082] - Downloading entity linked to URL fails: module ‘urllib.parse’ has no attribute ‘urlretrieve’

Improvements

• [SYNPY-1072] - Improve throughput of multiple small file transfers
• [SYNPY-1073] - Parellelize upload syncs
• [SYNPY-1074] - Parallelize download syncs
• [SYNPY-1084] - Allow anonymous usage for public APIs like GET /teamMembers/{id}
• [SYNPY-1088] - Manifest is in cache with synapse get -q
• [SYNPY-1090] - Command line client does not support apikey

Tasks

• [SYNPY-1080] - Remove Versionable from SchemaBase
• [SYNPY-1085] - Move to pytest testing framework
• [SYNPY-1087] - Improve synapseclient installation instructions

v2.1.1

Highlights

• This version includes a performance improvement for syncFromSynapse downloads of deep folder hierarchies to local filesystem locations outside of the Synapse cache.

• Support is added for SubmissionViews that can be used to query and edit a set of submissions through table services.

  from synapseclient import SubmissionViewSchema
  project = syn.get("syn123")
  evaluation_id = '9876543'
  view = syn.store(SubmissionViewSchema(name='My Submission View', parent=project, scopes=[evaluation_id]))
  view_table = syn.tableQuery(f"select * from {view.id}")
  

Bug

• [SYNPY-1075] - Error in Python test (submission annotations
• [SYNPY-1076] - Upgrade/fix Pandas dependency

Improvement

• [SYNPY-1070] - Add support for submission views
• [SYNPY-1078] - Improve syncFromSynapse performance for large folder structures synced to external paths

v2.1.0

Highlights:

• A max_threads property of the Synapse object has been added to customize the number of concurrent threads that will be used during file transfers.

  import synapseclient
  syn = synapseclient.login()
  syn.max_threads = 20
  

  If not customized the default value is (CPU count + 4). Adjusting this value
  higher may speed up file transfers if the local system resources can take advantage of the higher setting.
  Currently this value applies only to files whose underlying storage is AWS S3.

  Alternately, a value can be stored in the synapseConfig configuration file that will automatically apply
  as the default if a value is not explicitly set.

  [transfer]
  max_threads=16
  

• This release includes support for directly accessing S3 storage locations using AWS Security Token Service
  credentials. This allows use of external AWS clients and libraries with Synapse storage, and can be used to
  accelerate file transfers under certain conditions. To create an STS enabled folder and set-up direct access to S3 storage, see here.

• The getAnnotations and setAnnotations methods of the Synapse object have been deprecated in
  favor of newer get_annotations and set_annotations methods, respectively. The newer versions
  are parameterized with a typed Annotations dictionary rather than a plain Python dictionary to prevent
  existing annotations from being accidentally overwritten. The expected usage for setting annotations is to first retrieve the existing Annotations for an entity before saving changes by passing back a modified value.

  annos = syn.get_annotations('syn123')
  
  # set key 'foo' to have value of 'bar' and 'baz'
  annos['foo'] = ['bar', 'baz']
  # single values will automatically be wrapped in a list once stored
  annos['qwerty'] = 'asdf'
  
  annos = syn.set_annotations(annos)
  

  The deprecated annotations methods may be removed in a future release.

A full list of issues addressed in this release are below.

Bug

• [SYNPY-913] - Travis Build badge for develop branch is pointing to pull request
• [SYNPY-960] - AppVeyor build badge appears to be failed while the builds are passed
• [SYNPY-1036] - different users storing same file to same folder results in 403
• [SYNPY-1056] - syn.getSubmissions fails due to new Annotation class in v2.1.0-rc

Improvement

• [SYNPY-1036] - Make upload speeds comparable to those of the AWS S3 CLI
• [SYNPY-1049] - Expose STS-related APIs

Task

• [SYNPY-1059] - Use collections.abc instead of collections

v2.0.0

Highlights

• Multi-threaded download of files from Synapse can be enabled by setting syn.multi_threaded to True on a synapseclient.Synapse object. This will become the default implementation in the future, but to ensure stability for the first release of this feature, it must be intentionally enabled.

  import synapseclient
  syn = synapseclient.login()
  syn.multi_threaded = True
  # syn123 now will be downloaded via the multi-threaded implementation
  syn.get("syn123")
  

  Currently, multi-threaded download only works with files stored in AWS S3, where most files on Synapse reside. This also includes custom storage locations that point to an AWS S3 bucket. Files not stored in S3 will fall back to single-threaded download even if syn.multi_threaded==True.

• synapseutils.copy() now has limitations on what can be copied

  • A user must have download permissions on the entity they want to copy.
  • Users cannot copy any entities that have access requirements.

• contentTypes and fileNames are optional parameters in synapseutils.copyFileHandles()

• Synapse Docker Repository(synapseclient.DockerRepository) objects can now be submitted to Synapse evaluation queues using the entity argument in synapseclient.Synapse.submit(). An optional argument docker_tag="latest" has also been added to synapseclient.Synapse.submit()" to designate which tagged Docker image to submit.

A full list of issues addressed in this release are below.

Bug

• [SYNPY-271] - cache.remove fails to return the file handles we removed
• [SYNPY-1032] - Support new columnTypes defined in backend

Task

• [SYNPY-999] - Remove unsafe copy functions from client
• [SYNPY-1027] - Copy function should copy things when users are part of a Team
  that has DOWNLOAD access

Improvement

• [SYNPY-389] - submission of Docker repository
• [SYNPY-537] - synapseutils.copyFileHandles requires fields that does not require
  at rest
• [SYNPY-680] - synapseutils.changeFileMetaData() needs description in
  documentation
• [SYNPY-682] - improve download speeds to be comparable to AWS
• [SYNPY-807] - Drop support for Python 2
• [SYNPY-907] - Replace `from <module> import ...` with `import <module>`
• [SYNPY-962] - remove 'password' as an option in default synapse config file
• [SYNPY-972] - Link on Synapse Python Client Documentation points back at itself

v1.9.4

Bug

• [SYNPY-881] -
  Synu.copy fails when copying a file with READ permissions
• [SYNPY-888] -
  Docker repositories cannot be copied
• [SYNPY-927] -
  trying to create a project with name that already exists hangs
• [SYNPY-1005] -
  cli docs missing sub-commands
• [SYNPY-1018] -
  Synu.copy shouldn't copy any files with access restrictions

New Feature

• [SYNPY-851] -
  invite user or list of users to a team

Improvement

• [SYNPY-608] -
  Add how to contribute md to github project
• [SYNPY-735] -
  command line for building a table
• [SYNPY-864] -
  docstring for the command line client doesn't have complete list of
  sub-commands available
• [SYNPY-926] -
  allow forceVersion false for command line client
• [SYNPY-1013] -
  Documentation of "store" command for Synapse command line client
• [SYNPY-1021] -
  change email contact for code of conduct

v1.9.2

In version 1.9.2, we improved Views' usability by exposing set_entity_types() function to change the entity types that will show up in a View:

    import synapseclient
    from synapseclient.table import EntityViewType

    syn = synapseclient.login()
    view = syn.get("syn12345")
    view.set_entity_types([EntityViewType.FILE, EntityViewType.FOLDER])
    view = syn.store(view)

Features

• [SYNPY-919] - Expose a way to update entity types in a view using EntityViewType

Bug Fixes

• [SYNPY-855] - Single thread uploading fails in Lambda python3.6 environment
• [SYNPY-910] - Store Wiki shows deprecation warning
• [SYNPY-920] - Project View turned into File View after using syndccutils template

Tasks

• [SYNPY-790] - Pin to a fixed version of the request package
• [SYNPY-866] - Update Synapse logo in Python docs :)

Improvements

• [SYNPY-783] - typos in comments and in stdout
• [SYNPY-916] - Wonky display on parameters
• [SYNPY-917] - Add instructions on how to login with API key
• [SYNPY-909] - Missing columnTypes in Column docstring