Input file streaming

[panta-123]

Dear All,
We at Belle II are trying out file streaming for jobs. I would like to understand the how the other community is implementing this?
The analysis software at Belle II can take root or http protocol's SURL as input.

1. How the division of input files are made for each jobs between download or streaming of input files?
2. After doing 1, how the input data policy is handled for file streaming ? (turn off InputDataDownload ? )
3. and so on.

Any sort of input (techinical or general) on file streaming for jobs will be helpful

[fstagni]

@chaen can provide you more details on the LHCb side, but basically each job type has a certain way of accessing the inputs. For almost all types of production jobs we download (the only exception being "analysis productions". Users always stream ("InputDataByProtocol").

[panta-123]

@fstagni , How do we make job to switch from DownloadInputData to InputDataByProtocol according to jobType?
Also is there a way to switch back to DownloadInputData if the protocol we want for streaming(like xrootD or https) is not available for the SE hosting input file.

[chaen]

Hi @panta-123

Instead of just throwing answers at you, I took the opportunity to write some doc, which I think should already answer quite a bit of your questions
Can you please check ? #6340

Feel free to let me know what is not clear, and what is not answered. I'll then improve the docs.

[panta-123]

Hi @chaen , than you preparing the doc. I will go through it. Will let you know if any extra info needed.
I will leave this discussion open until i do some test.

[panta-123]

Hi @chaen ,
Three question:

1. If InputDataPolicy is defined in JDL, this is taken whatever the CS defination is, right ? If yes, then to use CS defination, InputDataPolicy shouldn't be defined on the JDL.

2. If site is specified in CS (InputDataPolicy/Site), then all the jobs in that site uses that module?
   for eg: In following site LCG.CERN.cern , all jobs running at the site uses InputDataByProtocol

   InputDataPolicy
   {
    # A job running at CERN would stream the data
    LCG.CERN.cern = DIRAC.WorkloadManagementSystem.Client.InputDataByProtocol
    }
   

3. If want set protocol list as shown below. Then does the job uses DownloadInputData for other kind of protocol.

Protocols
     {
       # This list is used if the we are getting a file from a
       # StorageElement local to the site we are running on
       Local = file, xroot, root
       }

[chaen]

Hi @panta-123

1. If it is not in the JDL, it uses what you have in the CS
2. Yes, if nothing is specified in the JDL
3. No. If you use InputDataByProtocol, this list will be used. If you use DownloadInputData, that list is not used at all

[chaen]

Documentation added in #6340