Manage Tapes in DIRAC

[zhangxiaomei]

We try to add Tape SEs for each site and plan to migrate data between Disk SE and Tape SE with FTS in the same way as normal transfers between two Disk SEs. Is it the right way to do it? Does we need the commands "dirac-stager-*" in our case? It is the first time for us to manage Tapes in DIRAC and I hope get experience from other experiments.

[marianne013]

@zhangxiaomei
This is what we have set up for RAL-LCG2 tape for t2k.org. The idea is that, if they need the data, they stage to a neighbouring disk (also provided by RAL-LCG2. We haven't looked into the dirac-stager-* commands yet (I wasn't even aware that they existed). Right now to stage a file we use the sequence listed below for an example file.

Here's the file we are trying to stage, present on TAPE only:

lx04:2021_Dec_13_1345_lz_EL7 > dirac-dms-lfn-replicas /t2k.org/nd280/test/antareseostest.txt
LFN                                    StorageElement    URL
============================================================
/t2k.org/nd280/test/antareseostest.txt RAL-LCG2-T2K-tape root://[email protected]:1094//eos/antares/prod/t2k.org/nd280/test/antareseostest.txt

now we send the 'stage to neighbouring disk' request:

lx04:2021_Dec_13_1345_lz_EL7 > dirac-dms-replicate-and-register-request stagetest3 /t2k.org/nd280/test/antareseostest.txt RAL-LCG2-ECHO-disk
Request 'stagetest3' has been put to ReqDB for execution.
RequestID(s): 648435
You can monitor requests' status using command: 'dirac-rms-request <requestName/ID>'

it shows up in the system (note the time - it is now 9:21):

lx04:2021_Dec_13_1345_lz_EL7 > dirac-rms-request 648435
Request name='stagetest3' ID=648435 Status='Waiting'
Created 2022-07-04 09:21:43, Updated 2022-07-04 09:21:43, NotBefore 2022-07-04 09:21:43
Owner: '/C=UK/O=eScience/OU=Imperial/L=Physics/CN=daniela bauer', Group: t2k.org_production
  [0] Operation Type='ReplicateAndRegister' ID=622444 Order=0 Status='Waiting'
      TargetSE: RAL-LCG2-ECHO-disk - Created 2022-07-04 09:21:43, Updated 2022-07-04 09:21:43
    [01] ID=570886 LFN='/t2k.org/nd280/test/antareseostest.txt' Status='Waiting' Checksum='13b94938'

and now we wait ..... 10 min later I check again and woohoo, it's now in Scheduled state:

lx04:2021_Dec_13_1345_lz_EL7 > dirac-rms-request 648435
Request name='stagetest3' ID=648435 Status='Scheduled'
Created 2022-07-04 09:21:43, Updated 2022-07-04 09:32:16, NotBefore 2022-07-04 09:32:17
Owner: '/C=UK/O=eScience/OU=Imperial/L=Physics/CN=daniela bauer', Group: t2k.org_production
  [0] Operation Type='ReplicateAndRegister' ID=622444 Order=0 Status='Scheduled'
      TargetSE: RAL-LCG2-ECHO-disk - Created 2022-07-04 09:21:43, Updated 2022-07-04 09:32:16
    [01] ID=570886 LFN='/t2k.org/nd280/test/antareseostest.txt' Status='Scheduled' Checksum='13b94938'

I go an have a cup of tea and try again. I now have an FTS job associated with it:

lx04:2021_Dec_13_1345_lz_EL7 > dirac-rms-request 648435
Request name='stagetest3' ID=648435 Status='Scheduled'
Created 2022-07-04 09:21:43, Updated 2022-07-04 09:32:16, NotBefore 2022-07-04 09:32:17
Owner: '/C=UK/O=eScience/OU=Imperial/L=Physics/CN=daniela bauer', Group: t2k.org_production
  [0] Operation Type='ReplicateAndRegister' ID=622444 Order=0 Status='Scheduled'
      TargetSE: RAL-LCG2-ECHO-disk - Created 2022-07-04 09:21:43, Updated 2022-07-04 09:32:16
    [01] ID=570886 LFN='/t2k.org/nd280/test/antareseostest.txt' Status='Scheduled' Checksum='13b94938'


FTS3 jobs associated: 
https://fts00.grid.hep.ph.ic.ac.uk:8449/fts3/ftsmon/#/job/cc3f16a4-fb7c-11ec-920c-525400e091e6 (Staging)

Getting on with my work for a little while, checking again. Now the FTS job is finished, but DIRAC hasn't quite processed the request yet:

lx04:2021_Dec_13_1345_lz_EL7 > dirac-rms-request 648435
Request name='stagetest3' ID=648435 Status='Scheduled'
Created 2022-07-04 09:21:43, Updated 2022-07-04 09:32:16, NotBefore 2022-07-04 09:32:17
Owner: '/C=UK/O=eScience/OU=Imperial/L=Physics/CN=daniela bauer', Group: t2k.org_production
  [0] Operation Type='ReplicateAndRegister' ID=622444 Order=0 Status='Scheduled'
      TargetSE: RAL-LCG2-ECHO-disk - Created 2022-07-04 09:21:43, Updated 2022-07-04 09:32:16
    [01] ID=570886 LFN='/t2k.org/nd280/test/antareseostest.txt' Status='Scheduled' Checksum='13b94938'


FTS3 jobs associated: 
https://fts00.grid.hep.ph.ic.ac.uk:8449/fts3/ftsmon/#/job/cc3f16a4-fb7c-11ec-920c-525400e091e6 (Finished)

At 10:08 DIRAC is putting on the finishing touches: [1] is Done, [0] isn't:

lx04:2021_Dec_13_1345_lz_EL7 > dirac-rms-request 648435
Request name='stagetest3' ID=648435 Status='Waiting'
Created 2022-07-04 09:21:43, Updated 2022-07-04 10:08:47, NotBefore 2022-07-04 10:08:47
Owner: '/C=UK/O=eScience/OU=Imperial/L=Physics/CN=daniela bauer', Group: t2k.org_production
  [0] Operation Type='RegisterReplica' ID=623348 Order=0 Status='Waiting'
      TargetSE: RAL-LCG2-ECHO-disk - Created 2022-07-04 10:08:47, Updated 2022-07-04 10:08:47
    [01] ID=615886 LFN='/t2k.org/nd280/test/antareseostest.txt' Status='Waiting' Checksum='13b94938'
  [1] Operation Type='ReplicateAndRegister' ID=622444 Order=1 Status='Done'  
      TargetSE: RAL-LCG2-ECHO-disk - Created 2022-07-04 09:21:43, Updated 2022-07-04 10:08:47
    [01] ID=570886 LFN='/t2k.org/nd280/test/antareseostest.txt' Status='Done' Checksum='13b94938'


FTS3 jobs associated: 
https://fts00.grid.hep.ph.ic.ac.uk:8449/fts3/ftsmon/#/job/cc3f16a4-fb7c-11ec-920c-525400e091e6 (Finished)

At 10:24 i try again and it looks like it's time to break out the champagne:

lx04:2021_Dec_13_1345_lz_EL7 > dirac-rms-request 648435
Request name='stagetest3' ID=648435 Status='Done'
Created 2022-07-04 09:21:43, Updated 2022-07-04 10:24:27, NotBefore 2022-07-04 10:08:47
Owner: '/C=UK/O=eScience/OU=Imperial/L=Physics/CN=daniela bauer', Group: t2k.org_production
  [0] Operation Type='RegisterReplica' ID=623348 Order=0 Status='Done'
      TargetSE: RAL-LCG2-ECHO-disk - Created 2022-07-04 10:08:47, Updated 2022-07-04 10:24:27
    [01] ID=615886 LFN='/t2k.org/nd280/test/antareseostest.txt' Status='Done' Checksum='13b94938'
  [1] Operation Type='ReplicateAndRegister' ID=622444 Order=1 Status='Done'
      TargetSE: RAL-LCG2-ECHO-disk - Created 2022-07-04 09:21:43, Updated 2022-07-04 10:08:47
    [01] ID=570886 LFN='/t2k.org/nd280/test/antareseostest.txt' Status='Done' Checksum='13b94938'


FTS3 jobs associated: 
https://fts00.grid.hep.ph.ic.ac.uk:8449/fts3/ftsmon/#/job/cc3f16a4-fb7c-11ec-920c-525400e091e6 (Finished)

and now I have the files where I want them to be:

x04:2021_Dec_13_1345_lz_EL7 > dirac-dms-lfn-replicas /t2k.org/nd280/test/antareseostest.txt
LFN                                    StorageElement     URL
=============================================================
/t2k.org/nd280/test/antareseostest.txt RAL-LCG2-T2K-tape  root://[email protected]:1094//eos/antares/prod/t2k.org/nd280/test/antareseostest.txt
                                       RAL-LCG2-ECHO-disk root://xrootd.echo.stfc.ac.uk:1094/gen:t2k/t2k.org/nd280/test/antareseostest.txt

To remove the file from disk once I don't need it any longer I use:

dirac-dms-remove-replicas /t2k.org/nd280/test/antareseostest.txt RAL-LCG2-ECHO-disk

I will try to get an actual t2k person to comment.

[zhangxiaomei]

Thank you for detailed information, Daniela. From your example, it looks like we can treat disk SE and tape SE in the same way. In that case, I can also use the origin components: transformation system + fts to do bulk stag-in.