DICOM need - could DRS objects be folders?

[ianfore]

Not necessarily a full consideration of the above question, but one focused on the NCI Imaging Data Commons (IDC) DICOM need raised in #389.

One thing that hangs over this the immutability of DRS objects. So any consideration of this must assume any folder referred to by a DRS id would be of fixed content, but IDC is well advanced in handling that already.

What if the DRS object was the folder (for the series)?
Compute in place, or wholesale copy within the cloud provider should be straightforward, without the long list issue, or the other DRS implementation issues raised by @fedorov in #389.

Folders as DRS objects won't work with signed http:// URLs; you can't get signed URLs for folders.
But what if DRS were just providing s3:// or gs:// URIs for the folders?
As far as I can see, far from being illegal for DRS, it was within the intent.

It would be a problem for some scenarios, because the IAM specific to the given cloud would have to be managed.
But with the open storage in IDC that wouldn't be an issue.

Of course the client would have to be working with the specific cloud protocols, but the SB platform has to do that anyway.

[michael-conway]

I treat the 'bundle' as the iRODS Collection (folder) for on-prem. The collection is the natural place for attaching metadata, applying policies like immutability of folder contents and generating internal access tickets for signed URLs so it seems fairly in line with the IDC? I guess the question is whether there are additional API endpoints or FASP client operations involved? I could see some straight-forward things like getting the contents of a bundle as an archive (e.g. tar or zip)? That would be one case where the folder could act as an object. How would you do a copy? Would there be an endpoint or operation that says copy DRS bundle A to DRS endpoint B? I had wondered myself whether there would be a future mode of data movement in this manner. In iRODS you can create a writeable ticket so it would be conceivable to create an access URL at a destination DRS endpoint. That may be totally outside of the scope but I wondered how you would orchestrate data movement via the DRS spec, and is that a necessary case, especially when data is moving across cloud providers or involving on-prem and cloud data? Hopefully not too 'random'...

…

On Tue, Sep 20, 2022 at 7:50 AM Ian Fore ***@***.***> wrote: Not necessarily a full consideration of the above question, but one focused on the NCI Imaging Data Commons (IDC) DICOM need raised in #389 <#389>. One thing that hangs over this the immutability of DRS objects. So any consideration of this must assume any folder referred to by a DRS id would be of fixed content, but IDC is well advanced in handling that already. What if the DRS object was the folder (for the series)? Compute in place, or wholesale copy within the cloud provider should be straightforward, without the long list issue, or the other DRS implementation issues raised by @fedorov <https://github.com/fedorov> in #389 <#389>. Folders as DRS objects won't work with signed http:// URLs; you can't get signed URLs for folders. But what if DRS were just providing s3:// or gs:// URIs for the folders? As far as I can see, far from being illegal for DRS, it was within the intent. It would be a problem for some scenarios, because the IAM specific to the given cloud would have to be managed. But with the open storage in IDC that wouldn't be an issue. Of course the client would have to be working with the specific cloud protocols, but the SB platform has to do that anyway. — Reply to this email directly, view it on GitHub <#391>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIL4LIBGGIZAHDDI6C5FJLV7GQIFANCNFSM6AAAAAAQQ7XWBA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>