How to split job over a list of input files with Dirac

[Albert-IC]

How to split job over a list of input files with Dirac

[egede]

I am confused about that you talk about local access to root files and xrootd at the same time. If you plan to read the files via streaming, then there is no need for a local copy, but you need to have the permissions to read the files over the WAN. If you have the files locally where the DIRAC job runs, there is no need for xrootd.

Before you start on any of this, I should remind you that copying large files through the input sandbox (which seems to be what you propose here) is really not recommended and could cause trouble both in terms of very long submission times and problems for the overall DIRAC WMS. It would be much better to have the files as DiracFile object stored on the grid.

That being said, I think you can get this to work by using a GangaDatasetSplitter. If the files are available as LocalFile object on the machine whre you submit from, that should be straight forward. If they are only available via xrootd on the machine you submit from, I think you could get this to work by carefully configuring the MassStorageFile object, but it would quite likely be very inefficient (as you in effect would copy the files to /tmp on your local machine and then upload them to a temporary Dirac storage area, and finally download them to the machine running the Dirac job).

[Albert-IC]

Thanks for the reply, so it seems that just using the xrootd paths is not the correct solution. I will try and use GangaDatasetSplitter again for this, I opened #2220 for this reason.

Whilst trying GangaDatasetSplitter on a list of DiracFile with the Dirac backend in the 8.7.0 release I have started to get the following error (ERROR CredentialsError: Cannot get proxy which matches requirements DiracProxy(group='<vo>_user', encodeDefaultProxyFileName=True)). I have checked and I do not get this error when downgrading to 8.6.0.

[egede]

Please open a separate issue for the CredentialsError. An absolute minimal example that illustrate the problem is much appreciated.

[Albert-IC]

Thanks, I will open a separate issue for the Dirac problems